A 'Detour Copying' of 'Introduction to Anomaly Detection with Python' ~ Chapter 4: 'Anomaly Detection Based on Distance'
Chapter 4: 'Anomaly Detection Based on Distance'
Authors of the book: Dr. Kaoru Fueda, Dr. Takeshi Ezaki, and Dr. Jongchan Lee
This article covers a 'detour copying' of Chapter 4, 'Introduction to Anomaly Detection with Python' from the text 'Anomaly Detection Based on Distance'.Detour Copying'.
Since the text does not include the Python code for Chapter 4, the detour copying is at maximum freedom! Now, let's open the text and set off on a journey of anomaly detection🚀
This series is a documentary of 'detour copying' where I experimentally write Python code for themes that caught my interest but were not introduced in the book 'Introduction to Anomaly Detection with Python' (published by Scientific Information Publishing, referred to as 'the text'), or themes where I want to try methods other than those in the text, while referring to the anomaly detection theories, mathematical formulas, and Python programs in the book.detour copying is a documentary where I experimentally write Python code for themes that caught my interest but were not introduced in the text, or themes where I want to try methods other than those in the text.
Introduction
Introduction to the text 'Introduction to Anomaly Detection with Python'
The text is an introductory book on anomaly detection released in April 2023.
It contains both mathematical derivations and Python implementations.
The source code in Jupyter Notebook format and the data in csv format can be downloaded from the URL provided in the book as an exclusive benefit for purchasers.
Citation Notation
This article cites text and code from the book listed in the source, and modifies the posted text and code as appropriate.
[Source]
'Introduction to Anomaly Detection with Python - From Basics to Practice -' First Edition, Authors: Kaoru Fueda / Takeshi Ezaki / Jongchan Lee, Ohmsha
The illustrations in this article are borrowed from 'Cute Free Material Collection Irasutoya'.
Thank you!
Chapter 4: Anomaly Detection Based on Distance
I will write the Python code in Jupyter Notebook format (extension .ipynb).
This chapter of the text is mainly specialized in acquiring basic knowledge of anomaly detection based on distance. Specifically, it presents calculation formulas and results for 'Mahalanobis distance,' 'Euclidean distance,' 'Minkowski distance,' 'Manhattan distance,' 'Jacob similarity,' 'Jaccard similarity,' 'Cosine similarity,' 'distance from the nearest neighbor,' 'average distance from the k-neighbors,' and 'median distance to the k-nearest neighbors.' And there is no Python code provided!
This article will tackle the following two points of detour copying. It feels like facing a blank canvas!
① Creating Python trial code for similarity
I will calculate the following measures from the various scales listed in Section 4-2 of the text, 'Similarity (Distance),' using Python. Since there are no numerical examples in the text, it is unclear whether the Python code is appropriate... ・Mahalanobis distance ・Jacob similarity ・Jaccard similarity ・Cosine similarity
② Detecting anomalies in one-dimensional data based on distance
I will try the four distances listed in Section 4-3 of the text, 'Approaches to Anomaly Detection Based on Distance.' I will cite the numerical examples from the text. ・Distance to all data points ・Distance from the nearest neighbor ・Average distance from the k-neighbors ・Median distance to the k-nearest neighbors
Import
### インポート
# 数値・確率計算
import numpy as np
# 距離・類似度
from scipy.spatial import distance
from sklearn.metrics import jaccard_score
from sklearn.metrics.pairwise import cosine_similarity
# 描画
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['font.family'] = 'Meiryo'If you are using Google Colab, please replace the 'plotting' section as follows.
!pip install japanize_matplotlib
import matplotlib.pyplot as plt
import japanize_matplotlib
import seaborn as sns
4-2 Similarity (Distance)
Now, let's calculate as many similarities and distances as possible!!!
■ Mahalanobis Distance
I will quote the Mahalanobis distance formula from the text.
It calculates the distance between two data point vectors $$\boldsymbol{p}$$ and $$\boldsymbol{q}$$.
$$
d(\boldsymbol{p}, \boldsymbol{q}) = \sqrt{(\boldsymbol{p}-\boldsymbol{q})^{\top} \boldsymbol{S}^{-1} (\boldsymbol{p}-\boldsymbol{q})}
$$
$$\boldsymbol{S}$$ is the variance-covariance matrix, and $$\boldsymbol{S}^{-1}$$ is the inverse of the variance-covariance matrix.
I will create two-dimensional normal distribution random data and calculate the Mahalanobis distance.
Now, let's create the data.
### 100行2列のデータを作成
# 乱数生成器の設定
rng = np.random.default_rng(seed=777)
# 平均、分散共分散行列の設定
mean_true = [3, 5] # 平均
sigma1 = 1.5 # 標準偏差1
sigma2 = 1 # 標準偏差2
rho = 0.8 # 相関係数
cov_true = np.array([[sigma1**2, sigma1*sigma2*rho],
[sigma1*sigma2*rho, sigma2**2]])
print('--- 作成パラメータ ---')
print('平均:')
print(mean_true)
print('標準偏差:')
print([sigma1, sigma2])
print('分散共分散行列:')
print(cov_true)
# データの作成（２変量正規分布の乱数）
data = rng.multivariate_normal(mean=mean_true, cov=cov_true, size=200)
print('\n--- 作成データの概要 ---')
print('data.shape: ', data.shape)
# データの平均・標準偏差・分散共分散行列を算出
mean_data = data.mean(axis=0)
std_data = data.std(ddof=0, axis=0)
cov_data = np.cov(data.T, ddof=0)
print('平均:')
print(mean_data)
print('標準偏差:')
print(std_data)
print('分散共分散行列:')
print(cov_data)[Execution Result]
Data with a shape of (200, 2) has been created.
Let's visualize the data with a scatter plot.
I will calculate the distance between the data points at the two indices specified in "Setting the two data points to calculate the distance".
### データの可視化
# 距離を算出する２つのデータ点の設定
p_idx = 50
q_idx = 1
# 全データの散布図の描画
sns.scatterplot(x=data[:, 0], y=data[:, 1], s=80, color='lightyellow',
ec='tab:orange')
# データ点p(index=0)の描画
sns.scatterplot(x=[data[p_idx, 0]], y=[data[p_idx, 1]], s=80, color='tab:red',
label='データ点 $p$')
# データ点q(index=1)の描画
sns.scatterplot(x=[data[q_idx, 0]], y=[data[q_idx, 1]], s=80, color='royalblue',
label='データ点 $q$')
# 修飾
plt.title(f'相関係数: {np.corrcoef(data.T)[0, 1]:.3f}')
plt.xlabel('$x_1$', fontsize=14)
plt.ylabel('$x_2$', fontsize=14)
plt.grid(lw=0.5)[Execution Result]
This is data with a strong positive correlation.
I will calculate the distance between the red and blue data.
Now then, I will calculate the Mahalanobis distance.
I will calculate it using the formula mentioned above and a scipy function.
Note that I could not find any reference examples for the calculation, so I cannot guarantee the accuracy of the calculation logic...
### マハラノビス距離の算出
# マハラノビス距離を算出する２つのデータ点を取得
p = data[p_idx, :]
q = data[q_idx, :]
print('データ点 p:', p)
print('データ点 q:', q)
# データの分散共分散行列の逆行列を算出
cov_inv_data = np.linalg.pinv(cov_data)
# マハラノビス距離の算出：公式利用
mahal_calc = np.sqrt((p - q) @ cov_inv_data @ (p - q).reshape(-1, 1))[0]
print('\n--- マハラノビス距離 ---')
print('数式利用 : ', mahal_calc)
# マハラノビス距離の算出：scipy利用
mahal_scipy = distance.mahalanobis(u=p, v=q, VI=cov_inv_data)
print('scipy利用: ', mahal_scipy)
[Execution Result]
[Useful Information👂️]
This is the relationship between the Mahalanobis distance and "Hotelling's T-squared distribution," which is the main subject from Chapter 5 onwards.
The "$$a(x^{\prime})$$" used for the anomaly score in Hotelling's T-squared distribution is the square of the Mahalanobis distance.
If you remove the square root of the Mahalanobis distance, it becomes the anomaly score $$a(x^{\prime})$$.
However, the Mahalanobis distance formula used in this case is slightly different from the formula mentioned above.
$$
d(\boldsymbol{x}) = \sqrt{(\boldsymbol{x}-\boldsymbol{\mu})^{\top} \boldsymbol{S}^{-1} (\boldsymbol{x}-\boldsymbol{\mu})}
$$
It seems to show the distance between a certain data point $$\boldsymbol{x}$$ (vector) and the mean $$\boldsymbol{\mu}$$ (vector) of the data group.
My impression is that when you search for "Mahalanobis distance," most websites display this formula.
◆ ◆ ◆ ◆ ◆
■ Jaccard Similarity
I will quote the Jaccard similarity formula from the text.
It seems to calculate the distance based on the number of data points common to two datasets $${A,B}$$.
$$
J(A,B) = \cfrac{|A \cap B|}{|A \cup B|} = \cfrac{|A \cap B|}{|A|+|B|-|A \cap B|}
$$
The symbols for the union $$\cup$$ and intersection $$\cap$$ of sets have appeared!
Let's calculate the Jaccard similarity.
When it comes to sets in Python, there is the set type "set".
I will use "set" to represent the datasets $${A,B}$$.
In the final line "len(A & B) / len(A | B)," I am calculating the intersection $$\cap$$ with "&" and the union $$\cup$$ with "|".
## Jacobの類似性の計算例
# 集合A, Bの設定
A = set([0, 1, 4])
B = set([1, 3, 4, 5, 6, 7, 8])
print('--- 作成データの概要 ---')
print(f'集合A: {A}\n集合B: {B}')
# 計算
print('\n--- 計算過程 ---')
print(f'A ∩ B 要素数: {len(A & B)}, 要素: {A & B}')
print(f'A ∪ B 要素数: {len(A | B)}, 要素: {A | B}, ')
print('\n--- 計算結果 ---')
print(f'Jacobの類似性: {len(A & B) / len(A | B)}')[Execution Result]
Jacob's similarity is $${0.25}$$.
◆ ◆ ◆ ◆ ◆
■ Jaccard Similarity
I will quote the formula for Jaccard distance from the text.
It seems that all variables of the two data point vectors $${boldsymbol{p}}$$ and $${boldsymbol{q}}$$ are binary, and the distance is calculated based on the number of common variables.
$$
J(\boldsymbol{p},\boldsymbol{q}) = \cfrac{m_{11}}{m_{01} + m_{10} + m_{11}}
$$
These are the contents of the variables on the right side.
・$${m_{01}}$$: Number of cases where $${p_i=0}$$ and $${q_i=1}$$
・$${m_{10}}$$: Number of cases where $${p_i=1}$$ and $${q_i=0}$$
・$${m_{11}}$$: Number of cases where $${p_i=1}$$ and $${q_i=1}$$
I will write the Python code using the numerical examples from the text.
### テキストの数値例
# p,qの設定 テキストの値を引用
p = np.array([1, 1, 0, 0, 1, 0])
q = np.array([1, 0, 1, 0, 0, 1])
print('--- データの概要 ---')
print('データ点 p:', p)
print('データ点 q:', q)
# m01, m10, m11の算出
m01 = sum((p == 0) & (q == 1))
m10 = sum((p == 1) & (q == 0))
m11 = sum((p == 1) & (q == 1))
# Jaccard類似性の算出
J_pq = m11 / sum([m01, m10, m11])
# 結果表示
print('\n--- 計算過程と結果 ---')
print(f'm01={m01}, m10={m10}, m11={m11}, m01+m10+m11={m01+m10+m11}')
print(f'Jaccard類似性={J_pq}')[Execution Result]
The Jaccard similarity is $${0.2}$$.
[scikit-learn Corner📯]
I will calculate the Jaccard distance of the example data using scikit-learn's jaccard_score().
Since I am using an evaluation metric library, the arguments for the two data points are y_true and y_pred.
# ジャッカードスコアの計算
j_score = jaccard_score(y_true=p, y_pred=q, average='binary')
print('ジャッカードスコア:', j_score)[Execution Result]
It was calculated quickly.
◆ ◆ ◆ ◆ ◆
■ Cosine Similarity
I will quote the formula for cosine similarity from the text.
It calculates the distance between two data point vectors $${boldsymbol{p}}$$ and $${boldsymbol{q}}$$.
$$
\cos(\boldsymbol{p},\boldsymbol{q}) = \cfrac{\sum^d_{i=1}p_i q_i}{\sqrt{\sum^d_{i=1}p_i^2 \sum^d_{i=1}q_i^2}}
$$
Using the Jaccard distance data, I will calculate the cosine similarity in three ways.
① Formula from the text
sum(p * q) / np.sqrt(sum(p**2) * sum(q**2))[Execution Result]
The cosine similarity is $${0.3 \cdots}$$.
② Matrix calculation
I will calculate the cosine similarity using the following formula.
$$
\cos(\boldsymbol{p},\boldsymbol{q})=\cfrac{\boldsymbol{p}\boldsymbol{q}}{\|\boldsymbol{p}\|\|\boldsymbol{q}\|}
$$
# 行列計算
p @ q / (np.linalg.norm(p) * np.linalg.norm(p))[Execution Result]
It is slightly different (sweat)
This calculation formula felt better.
# 行列計算
p @ q / np.sqrt((p @ p) * (q @ q))[Execution Result]
3. Using scikit-learn's cosine_similarity()
Convert the shapes of p and q to 2D (1, 6) using reshape and pass them as arguments.
# scikit-learnで余弦類似度（コサイン類似度）を計算
cosine_similarity(X=p.reshape(1, -1), Y=q.reshape(1, -1))[0][0][Execution Result]
The last value is... (ugh)
That was a fun detour with the trial code!✨
4-3 Approaches to Anomaly Detection Based on Distance
I will take a detour and transcribe the examples from 4-3-1 to 4-3-5 in the text!
I will calculate the anomaly score using four types of distances for 1D data and perform anomaly detection.
Then, I will plot the data points identified as anomalies on a 1D number line.
Example 4-3-1: Judging the data point furthest from all other data points as an anomaly
The data point with the largest sum of distances to other data points is judged as an anomaly.
Citing the data from the text, I will calculate the anomaly score for each data point and detect the data point with the maximum anomaly score as an anomaly.
The anomaly score is calculated in the code using '[sum(abs(D - i)) for i in D]'.
### 例4-3-1
# データの登録 ※テキストのデータを引用
D = np.array([1, 2, 3, 8, 20, 21])
print('データセット','\t:', D)
# 異常度αの算出（データ点間のマンハッタン距離の合計値）
alpha = np.array([sum(abs(D - i)) for i in D])
print('異常度α', '\t:', alpha)
# 異常度が最大の要素の表示
max_idx = alpha.argmax()
print('異常値が最大のデータ点:', D[max_idx], ', 異常度:', alpha[max_idx])[Execution Result]
Data point 21 was judged as an anomaly.
Let's visualize it.
This corresponds to Figure 4-1 in the text.
### 図4-1
## 描画用の設定
# 最大値の点の色・マーカーを変えるためのhueの設定
hue = np.zeros(len(D))
hue[max_idx] = 1
## 描画処理
# 描画領域の指定
plt.figure(figsize=(7, 0.4))
# 数直線（矢印線）の描画
plt.quiver(0, 0, 24, 0, angles='xy', scale_units='xy', scale=1, width=0.005)
# 数直線上にデータ点を描画
sns.scatterplot(x=D, y=np.zeros(len(D)), hue=hue, palette=['tab:blue', 'tab:red'],
style=hue, markers=['o', 's'], s=100, legend=False)
# 修飾：x軸の範囲と目盛り
plt.xlim(-1, 24)
plt.xticks(range(0, 23))
# 修飾：枠線の消去
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['left'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
# 修飾：y軸の軸目盛りと軸ラベルの消去
plt.tick_params(left=False, labelleft=False);[Execution Result]
The red square point is data point 21, which was judged as an anomaly.
◆ ◆ ◆ ◆ ◆
Example 4-3-2: Judging the data point with the largest distance to its nearest neighbor as an anomaly
The data point with the largest distance to its nearest neighbor is judged as an anomaly. Citing the data from the text, I will calculate the anomaly score for each data point and detect the data point with the maximum anomaly score as an anomaly.
The anomaly score is calculated in the code using '[min(abs(D[D != i] - i)) for i in D]'.
### 例4-3-2
# データの登録 ※テキストのデータを引用
D = np.array([1, 2, 3, 8, 20, 21])
print('データセット','\t:', D)
# 異常度αの算出（最近傍からの距離）
alpha = np.array([min(abs(D[D != i] - i)) for i in D])
print('異常度α', '\t:', alpha)
# 異常度が最大の要素の表示
max_idx = alpha.argmax()
print('異常値が最大のデータ点:', D[max_idx], ', 異常度:', alpha[max_idx])[Execution Result]
Data point 8 was judged as an anomaly.
Let's visualize it.
### 例4-3-2 描画
## 描画用の設定
# 最大値の点の色・マーカーを変えるためのhueの設定
hue = np.zeros(len(D))
hue[max_idx] = 1
## 描画処理
# 描画領域の指定
plt.figure(figsize=(7, 0.4))
# 数直線（矢印線）の描画
plt.quiver(0, 0, 24, 0, angles='xy', scale_units='xy', scale=1, width=0.005)
# 数直線上にデータ点を描画
sns.scatterplot(x=D, y=np.zeros(len(D)), hue=hue, palette=['tab:blue', 'tab:red'],
style=hue, markers=['o', 's'], s=100, legend=False)
# 修飾：x軸の範囲と目盛り
plt.xlim(-1, 24)
plt.xticks(range(0, 23))
# 修飾：枠線の消去
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['left'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
# 修飾：y軸の軸目盛りと軸ラベルの消去
plt.tick_params(left=False, labelleft=False);[Execution Result]
The red square point is data point 8, which was judged as an anomaly.
Indeed, the distance to its neighbor is the longest!
◆ ◆ ◆ ◆ ◆
Example 4-3-3: Judging the data point with the largest distance to its nearest neighbor as an anomaly, Part 2
I will execute the same anomaly score judgment method as in Example 4-3-2, but with a different dataset.
### 例4-3-3
# データの登録 ※テキストのデータを引用
D = np.array([1, 3, 5, 7, 100, 101, 200, 202, 205, 208, 210, 212, 214])
print('データセット','\t:', D)
# 異常度αの算出（最近傍からの距離）
alpha = np.array([min(abs(D[D != i] - i)) for i in D])
print('異常度α', '\t:', alpha)
# 異常度が最大の要素の表示
max_idx = alpha.argmax()
print('異常値が最大のデータ点:', D[max_idx], ', 異常度:', alpha[max_idx])[Execution Result]
Data point 205 was judged as an anomaly.
Let's visualize it.
This corresponds to Figure 4-2 in the text.
### 図4-2 ★テキストに無いコード
## 描画用の設定
# 最大値の点の色・マーカーを変えるためのhueの設定
hue = np.zeros(len(D))
hue[max_idx] = 1
## 描画処理
# 描画領域の指定
plt.figure(figsize=(7, 0.4))
# 数直線（矢印線）の描画
plt.quiver(-20, 0, 260, 0, angles='xy', scale_units='xy', scale=1, width=0.002)
# 数直線上にデータ点を描画
sns.scatterplot(x=D, y=np.zeros(len(D)), hue=hue, palette=['tab:blue', 'tab:red'],
style=hue, markers=['o', 's'], s=20, legend=False)
# 修飾：x軸の範囲と目盛り
plt.xlim(-20, 260)
plt.xticks(range(0, 240, 20))
# 修飾：枠線の消去
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['left'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
# 修飾：y軸の軸目盛りと軸ラベルの消去
plt.tick_params(left=False, labelleft=False);[Execution Result]
The red square point is data point $${205}$$, which was determined to be an anomaly.
◆ ◆ ◆ ◆ ◆
Example 4-3-4: Determining anomalies based on the largest total distance from the $${k}$$-nearest neighbors
We define $${k}$$ nearest neighbors and determine the data point with the largest total distance from these $${k}$$ nearest neighbors to be an anomaly.
Using the data from the text, we calculate the anomaly score for each data point when $${k=3}$$ and detect the data point with the highest anomaly score as an anomaly.
The anomaly score is calculated in the code using "[np.sort(abs(D[D != i] - i))[:k].sum() for i in D]".
### 例4-3-4
# パラメータk
k = 3
# データの登録 ※テキストのデータを引用
D = np.array([1, 3, 5, 7, 100, 101, 200, 202, 205, 208, 210, 212, 214])
print('データセット','\t:', D)
# 異常度αの算出（k近傍からの距離の合計値）
alpha = np.array([np.sort(abs(D[D != i] - i))[:k].sum() for i in D])
print('異常度α', '\t:', alpha)
# 異常度が最大の要素の表示
max_idx = alpha.argmax()
print('異常値が最大のデータ点:', D[max_idx], ', 異常度:', alpha[max_idx])[Execution Result]
Data point $${101}$$ was determined to be an anomaly.
Let's visualize it.
This corresponds to Figure 4-3 in the text.
### 図4-3
## 描画用の設定
# 最大値の点の色・マーカーを変えるためのhueの設定
hue = np.zeros(len(D))
hue[max_idx] = 1
## 描画処理
# 描画領域の指定
plt.figure(figsize=(7, 0.4))
# 数直線（矢印線）の描画
plt.quiver(-20, 0, 260, 0, angles='xy', scale_units='xy', scale=1, width=0.002)
# 数直線上にデータ点を描画
sns.scatterplot(x=D, y=np.zeros(len(D)), hue=hue, palette=['tab:blue', 'tab:red'],
style=hue, markers=['o', 's'], s=20, legend=False)
# 修飾：x軸の範囲と目盛り
plt.xlim(-20, 260)
plt.xticks(range(0, 240, 20))
# 修飾：枠線の消去
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['left'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
# 修飾：y軸の軸目盛りと軸ラベルの消去
plt.tick_params(left=False, labelleft=False);[Execution Result]
The red square point is data point $${101}$$, which was determined to be an anomaly.
Let's try the case where $${k=4}$$.
### 例4-3-4
# パラメータk
k = 4
# データの登録 ※テキストのデータを引用
D = np.array([1, 3, 5, 7, 100, 101, 200, 202, 205, 208, 210, 212, 214])
print('データセット','\t:', D)
# 異常度αの算出（k近傍からの距離の合計値）
alpha = np.array([np.sort(abs(D[D != i] - i))[:k].sum() for i in D])
print('異常度α', '\t:', alpha)
# 異常度が最大の要素の表示
max_idx = alpha.argmax()
print('異常値が最大のデータ点:', D[max_idx], ', 異常度:', alpha[max_idx])[Execution Result]
Data point $${101}$$ was also determined to be an anomaly when $${k=4}$$.
Just to be sure, let's plot it.
### 例4-3-4 k=4 の描画
## 描画用の設定
# 最大値の点の色・マーカーを変えるためのhueの設定
hue = np.zeros(len(D))
hue[max_idx] = 1
## 描画処理
# 描画領域の指定
plt.figure(figsize=(7, 0.4))
# 数直線（矢印線）の描画
plt.quiver(-20, 0, 260, 0, angles='xy', scale_units='xy', scale=1, width=0.002)
# 数直線上にデータ点を描画
sns.scatterplot(x=D, y=np.zeros(len(D)), hue=hue, palette=['tab:blue', 'tab:red'],
style=hue, markers=['o', 's'], s=20, legend=False)
# 修飾：x軸の範囲と目盛り
plt.xlim(-20, 260)
plt.xticks(range(0, 240, 20))
# 修飾：枠線の消去
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['left'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
# 修飾：y軸の軸目盛りと軸ラベルの消去
plt.tick_params(left=False, labelleft=False);[Execution Result]
◆ ◆ ◆ ◆ ◆
Example 4-3-5: Determining anomalies based on the largest median distance from the $${k}$$-nearest neighbors
We define $${k}$$ nearest neighbors and determine the data point with the largest median distance from these $${k}$$ nearest neighbors to be an anomaly.
Using the data from the text, we calculate the anomaly score for each data point when $${k=3}$$ and detect the data point with the highest anomaly score as an anomaly.
The anomaly score is calculated in the code using "[np.median(np.sort(abs(D[D != i] - i))[:k]) for i in D]".
### 例4-3-5
# パラメータk
k = 3
# データの登録 ※テキストのデータを引用
D = np.array([1, 3, 5, 7, 100, 101, 200, 202, 205, 208, 210, 212, 214])
print('データセット','\t:', D)
# 異常度αの算出（k近傍からの距離の合計値）
alpha = np.array([np.median(np.sort(abs(D[D != i] - i))[:k]) for i in D])
print('異常度α', '\t:', alpha)
# 異常度が最大の要素の表示
max_idx = alpha.argmax()
print('異常値が最大のデータ点:', D[max_idx], ', 異常度:', alpha[max_idx])[Execution Result]
Data point $${101}$$ was determined to be an anomaly.
Let's visualize it.
### 例4-3-5 の描画
## 描画用の設定
# 最大値の点の色・マーカーを変えるためのhueの設定
hue = np.zeros(len(D))
hue[max_idx] = 1
## 描画処理
# 描画領域の指定
plt.figure(figsize=(7, 0.4))
# 数直線（矢印線）の描画
plt.quiver(-20, 0, 260, 0, angles='xy', scale_units='xy', scale=1, width=0.002)
# 数直線上にデータ点を描画
sns.scatterplot(x=D, y=np.zeros(len(D)), hue=hue, palette=['tab:blue', 'tab:red'],
style=hue, markers=['o', 's'], s=20, legend=False)
# 修飾：x軸の範囲と目盛り
plt.xlim(-20, 260)
plt.xticks(range(0, 240, 20))
# 修飾：枠線の消去
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['left'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
# 修飾：y軸の軸目盛りと軸ラベルの消去
plt.tick_params(left=False, labelleft=False);[Execution Result]
The red square point is data point $${101}$$, which was determined to be an anomaly.
Plotting the number line was fun!
This concludes the detour into Chapter 4.
Series Articles
Next Article
Previous Article
Table of Contents
Blog Introduction
I am writing seven series of articles on note.
Please feel free to take a look!
1. Relaxed Statistics
This is a blog that roughly explores probability and statistics using the Statistical Test Grade 2 problem collection as a guide.
Feel free to read it as if it were casual conversation. Please do take a look.
It corresponds to the Statistical Test Grade 2 Official Problem Collection CBT version.
Sample code for Python and EXCEL is also available.
2. Experiment! Fun Bayesian Modeling 1 & 2 with PyMC Ver. 5
I will draw and analyze the Bayesian models used in the psychology research from the books "Fun Bayesian Modeling" and "Fun Bayesian Modeling 2" using PyMC Ver. 5.
Starting with these books, many Bayesian models are written in R language + Stan.
I will strive to explore the possibilities of PyMC and make it easy to practice Bayesian modeling.
Since these are familiar and easy-to-visualize themes, please try running them with PyMC and let's enjoy it together!
3. Experiment! Bayesian Modeling from Iwanami Data Science 1 with PyMC Ver. 5
I will draw and analyze the Bayesian models by four Bayesians from the book "Experiment! Iwanami Data Science Vol. 1" using PyMC Ver. 5.
This book is a great resource for learning the basics of Bayesian programming.
I feel like I've become friends with Bayesian methods by happily running PyMC models.
Everyone, please try running them with PyMC, and let's play and learn together!
4. Fun Code Copying: Bayesian, Python, etc.
I will blog about the results of my "book code copying activities" for Bayesian, Python, and others.
I am mainly working on translations into Python.
I hope this serves as sample code for fellow learners who are also copying code 🍀
5. Introduction to Time Series Analysis for Psychology with R and Stan, using Python and PyMC Ver. 5
I will practice the time series analysis from the book "Introduction to Time Series Analysis for Psychology with R and Stan" using Python and PyMC Ver. 5.
This book is packed with time series analysis themes!
I realized the depth of time series analysis.
I will enjoy learning time series analysis with my favorite language, Python.
6. Writing about Data Science-like Things
I write columns on statistics, data analysis, AI, machine learning, and Python on an irregular basis.
There are many articles related to statistics and data science books.
Series on "Statistics," "Python," "Mathematics and Python," and "R" have been created.
7. Python Machine Learning Programming Practice Journal
I wrote articles about my various thoughts when studying the book "Python Machine Learning Programming: PyTorch & scikit-learn Edition."
This book is a textbook for scikit-learn and PyTorch.
Please feel free to give it a try.
Thank you very much for reading until the end.
いいなと思ったら応援しよう！
応援ありがとうございます。これからもがんばって記事を作成します！