A 'Detour Copying' of 'Introduction to Anomaly Detection with Python' - Chapter 2: 'Data Science for Anomaly Detection'

Chapter 2: 'Data Science for Anomaly Detection'

Book authors: Dr. Kaoru Fueda, Dr. Takeshi Ezaki, Dr. Jongchan Lee

This article covers a so-called 'detour copying' of Chapter 2, 'Data Science for Anomaly Detection,' from the textbook 'Introduction to Anomaly Detection with Python.' This time, I took a detour into '

data visualization' and 'variable selection for regression models.' Now, let's open the textbook and set off on our journey into anomaly detection 🚀

Introduction

Introducing the textbook 'Introduction to Anomaly Detection with Python'

This series is a 'detour copying' documentary where I experimentally write Python code for themes that I was curious about but were not covered by programs in the textbook, or themes where I wanted to try methods other than those in the textbook, while referring to the anomaly detection theories, mathematical formulas, and Python programs in the book 'Introduction to Anomaly Detection with Python' (published by Scientific Information Publishing, hereafter referred to as the 'textbook').This series is a 'detour copying' documentary where I experimentally write Python code for themes that I was curious about but were not covered by programs in the textbook, or themes where I wanted to try methods other than those in the textbook, while referring to the anomaly detection theories, mathematical formulas, and Python programs in the book 'Introduction to Anomaly Detection with Python' (published by Scientific Information Publishing, hereafter referred to as the 'textbook'). is the goal.

The textbook is an introductory book on anomaly detection released in April 2023.
It contains both mathematical derivations and Python implementations.
The source code in Jupyter Notebook format and the data in csv format can be downloaded from the URL provided in the book as an exclusive benefit for purchasers.

■ Overview of the textbook
The textbook is composed of the following three parts, gradually approaching anomaly detection.

There are three anomaly detection themes that delve deeply into the issues using Python implementations.
These are mainly Chapters 5 through 7.

Additionally, an implementation of Isolation Forest is provided as an appendix.

■ Impressions after reading through
This is a personal opinion, but I really enjoyed being able to perform concrete practical anomaly detection using Python from Chapter 5 onwards in the textbook! I was able to improve my skills in Hotelling's T^2 method, One-Class SVM, and time-series data anomaly detection.
On the other hand, I wasn't sure if Chapters 1 through 4 were strongly or weakly related to anomaly detection.
Also, there were cases where Python code was not provided for calculation examples, which sometimes left me feeling frustrated as
someone who wants to get closer to formulas and models through code.

That is why I leaned into detour copying!

■ What is detour copying?
Detour copying is an activity where I write sample Python code based on my personal interests, picking up themes that arise while working through the textbook, such as 'What happens if I implement this formula in Python even though no program was introduced?' or 'What if I try running this using a different library than the one in the textbook?'

It's just a hobby turned into a blog. Sorry about that.

■ Things I want to cherish
Detour copying is one way to enjoy the textbook until the end.
There is one other important trick.
That is to read through it with a 'laid-back attitude'.
When you detect this or that from the textbook, chant the spell 'Oh well' and take a deep breath.
That's the way to go ♬

Now, let's open the textbook and set off on our journey into anomaly detection 🚀

Citation notation

This article cites text and code published in the book listed in the sources, and the published text and code have been modified as appropriate.
[Source]
'Introduction to Anomaly Detection with Python: From Basics to Practice', First Edition, Authors: Kaoru Fueda / Takeshi Ezaki / Jongchan Lee, Ohmsha

The illustrations in this article are borrowed from "Cute Free Material Collection Irasutoya."
Thank you very much!

Chapter 2: Data Science for Anomaly Detection

I will write the Python code in Jupyter Notebook format (extension .ipynb).

This chapter of the text mainly covers "Visualization," "Regression Analysis," "Principal Component Analysis," and "Bayes' Theorem."

In this article, I will tackle the following three detour copy-coding tasks.

Imports

### インポート

# 数値・確率計算
import pandas as pd
import numpy as np
import scipy.stats as stats

# 確率モデル
import statsmodels.api as sm

# 機械学習
from sklearn.linear_model import LinearRegression

# 描画
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['font.family'] = 'Meiryo'

If you are using Google Colab, please replace the "plotting" section as follows.

!pip install japanize_matplotlib
import matplotlib.pyplot as plt
import japanize_matplotlib
import seaborn as sns

2.1 Visualization of Obtained Data

I will load the "Davis height and weight data" used in the text directly from the website and implement the text's visualizations using seaborn.

Loading Data
I will load "Davis.csv" directly from the Item "Davis" at the following URL.
It seems to be a famous dataset in the anomaly detection community.

[Citation Site]

### データの読み込み
URL = 'https://vincentarelbundock.github.io/Rdatasets/csv/carData/Davis.csv'
data1 = pd.read_csv(URL, usecols=[1, 2, 3, 4, 5])
print('data1.shape: ', data1.shape)
display(data1.head())

[Execution Results]
This is data on the weight [kg] and height [cm] of 200 people.
The variables are gender, measured weight, measured height, reported weight, and reported height.

Histogram
I will draw a histogram of the weight variable, corresponding to Figure 2-1 in the text, using seaborn.

### 図2-1 １つの変量のバラツキを確認するヒストグラム
sns.histplot(data=data1, x='weight', ec='white')
plt.title('体重[kg]のヒストグラム')
plt.grid(lw=0.5);

[Execution Results]
weight: There is data that looks like an outlier near 160 for measured weight!

Bar Chart
I will draw a bar chart of the sex variable (gender), corresponding to Figure 2-2 in the text, using seaborn.

### 図2-2 各項目におけるデータの数を比較する棒グラフ
sns.countplot(data=data1, x='sex', hue='sex', palette=['lightblue', 'lightpink'])
plt.title('男性M・女性Fの数')
plt.grid(lw=0.5);

[Execution Results]
Out of 200 data points, there are just under 90 males and just over 110 females.
I get the impression that there is slightly more female data.

Pie Chart
Let's represent this bar chart as a pie chart.
I didn't know how to draw a pie chart in seaborn, so I used matplotlib.

### 円チャート
plt.pie(data1['sex'].value_counts().sort_index(ascending=False),
        counterclock=False, startangle=90, labels=['男性M', '女性F'],
        colors=['lightblue', 'lightpink'], autopct='%1.1f%%');

[Execution Results]
The gender ratio is now clear.

Scatter Plot
I will draw a scatter plot of the weight and height variables, corresponding to Figure 2-3 in the text, using seaborn.

### 図2-3 ２つの連続する変量の関係性を可視化する散布図
sns.scatterplot(data=data1, x='weight', y='height', s=50, alpha=0.7)
plt.title('体重と身長の散布図')
plt.grid(lw=0.5);

[Execution Results]
The data point near 160 on the horizontal axis is significantly far from the other data groups.
It's starting to look like anomaly detection!
Are the points around 120 on the horizontal axis and 180 on the vertical axis also outliers???

Scatter Plot Matrix
I will draw a scatter plot corresponding to Figure 2-4 in the text using the same seaborn library as the text.
I modified it to set gender as the hue so that the distribution for each gender can be understood.

### 図2-4 複数の連続する変量の関係性を可視化する散布図行列
sns.pairplot(data=data1, hue='sex', diag_kind='hist', height=2, 
             palette=['lightblue', 'lightpink'], diag_kws={'ec': 'white'});

[Execution Results]
That outlier appears to be data for a female.

The visualization zone ends here.

2-2 Mathematical Formulation of Obtained Data: Regression Models

I will directly load the "Crime Rate Data in 50 US States" used in the text from the website, and work on calculating regression coefficients using methods other than scikit-learn, as well as implementing the Stepwise method (backward elimination).

Loading Data
I will directly load "crime.txt" from the crime data at the following URL.

[Source Website]

### データの読み込み
URL = r'https://hastie.su.domains/StatLearnSparsity_files/DATA/crime.txt'
data2 = pd.read_csv(URL, sep='\t',
                    usecols=[0, 2, 3, 4, 5, 6],
                    names=['Y', 'X1', 'X2', 'X3', 'X4', 'X5'])
print('data2.shape: ', data2.shape)
display(data2.head())

[Execution Results]
This is the data for 50 states.

The variables are roughly as follows:
Y: Crime rate
X1: Annual police budget
X2: Percentage of high school graduates (25 years and older)
X3: Percentage of population not attending high school (16-19 years old)
X4: Percentage of college students (18-24 years old)
X5: Percentage of four-year college graduates (25 years and older)

Simple Regression and Scatter Plot with Regression Line
I will draw a scatter plot of explanatory variable X1 and objective variable Y.
I will calculate the regression coefficients with scipy and draw it with seaborn.

### 図2-5 ２変数のデータの散布図（左図）と単回帰モデルを追加した図（右図）

## 回帰分析 by scipy.stats
# 傾きと切片の算出
slope, intercept = stats.linregress(x=data2['X1'], y=data2['Y'])[0:2]
print(f'切片: {intercept:.4f}, 傾き: {slope:.4f}')
# x軸の値の設定
xval = np.linspace(data2['X1'].min(), data2['X1'].max(), 1001)
# 回帰直線のy軸の値の算出
lmval = intercept + slope * xval

## 描画処理
# 描画領域の設定
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3))
# 左の散布図の描画
sns.scatterplot(data=data2, x='X1', y='Y', ax=ax1)
ax1.set(title='X1とYの散布図')
ax1.grid(lw=0.5)
# 右の散布図＋回帰直線の描画
sns.scatterplot(data=data2, x='X1', y='Y', ax=ax2)
ax2.plot(xval, lmval, color='tab:red', ls='--')
ax2.set(title='X1とYの散布図と回帰直線')
ax2.grid(lw=0.5)
plt.tight_layout();

[Execution Results]

Multiple Regression
I will calculate the regression coefficients for the multiple regression equation $${\boldsymbol{y} = \boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}}$$ on pages 18-19 of the text.
I will try calculating the regression coefficients using the following matrix calculation.

$$
\boldsymbol{\beta} = (\boldsymbol{X}^{\top} \boldsymbol{X})^{-1} \boldsymbol{X}^{\top} \boldsymbol{y}
$$

### 重回帰 行列で計算

## 指数表示を回避
np.set_printoptions(precision=6,suppress=True)

## データセットの作成
# 定数項のためのデータ作成
const = np.ones(len(data2)).reshape(-1, 1)
# 説明変数の作成（定数項を追加）
X = np.concatenate([const, data2.drop(columns=['Y']).values], axis=1)
# 目的変数の作成
y = data2['Y'].values

## 回帰係数の算出と表示
beta = np.linalg.inv(X.T @ X) @ X.T @ y
print(f'y = {beta[0]:.4f} {beta[1]:+.4f}x1 {beta[2]:+.4f}x2',
      f'{beta[3]:+.4f}x3 {beta[4]:+.4f}x4 {beta[5]:+.4f}x5')

[Execution Results]
I output it in the format of a regression equation. How does it compare to the results in the text?

Variable Selection by Stepwise Method
Continuing to use the "Crime Rate Data in 50 US States," I will implement the "Stepwise method (backward elimination) evaluated by AIC" described on pages 21-22 of the text using statsmodels.

First, I will define the regression analysis execution function.
I will use the Ordinary Least Squares (OLS) method from statsmodels.
statsmodels calculates the AIC for us.

### 回帰分析実行関数の定義

def exec_linear_regression_sm(X, y):
    # 回帰モデルのインスタンス生成・学習・予測
    X_const = sm.add_constant(X)
    lm_model = sm.OLS(endog=y, exog=X_const)
    lm_result = lm_model.fit()
    # 変数名の取得
    vars = ['const'] + X.columns.values.tolist()
    # AICの取得
    aic = lm_result.aic
    # 学習結果から切片と傾きの取得
    coef = lm_result.params.values.tolist()
    # 戻り値：coefのindex0は切片,index1以降が傾き
    return {'vars': vars, 'aic': aic, 'coef': coef}

Now, for the execution of the Stepwise method.
The code is very long.

### Stepwise法による変数選択の実行

## データセットの作成
# 説明変数の作成
X = data2.drop(columns=['Y'])
# 目的変数の作成
y = data2['Y'].values
# 選択変数の設定
stepwise_vars = X.columns

## 全ての説明変数による重回帰の実行
print(f'--- 説明変数の数: {len(stepwise_vars)} ---')
result = exec_linear_regression_sm(X, y)
print(result)
best_aic = result['aic']
best_vars = result['vars']
best_coefs = result['coef']

## Stepwise: 選択変数が１つになるまで変数を１つづつ減らして回帰分析を繰り返し実行
while len(stepwise_vars) > 1:
    # 除外変数を格納する変数の準備
    drop_col = None
    print(f'\n--- 説明変数の数: {len(stepwise_vars)-1} ---')
    # 選択変数から１変数を除外して回帰分析を実行
    for var in stepwise_vars:
        # 回帰分析する変数群から１変数を除外
        trial_vars = stepwise_vars[~(stepwise_vars == var)]
        X_trial = X[trial_vars]
        # 回帰分析の実行
        result = exec_linear_regression_sm(X_trial, y)
        print(result)
        # 除外変数の更新
        if drop_col == None:
            drop_col = var
            drop_aic = result['aic']
        elif result['aic'] < drop_aic:
            drop_col = var
            drop_aic = result['aic']
        # ベストなAICの場合、AIC・変数・回帰係数を保存
        if result['aic'] < best_aic:
            best_aic = result['aic']
            best_vars = result['vars']
            best_coefs = result['coef']
    # 最も悪いAICの変数を選択変数から除外する
    stepwise_vars = stepwise_vars[~(stepwise_vars == drop_col)]
    print(' ※除外変数: ', drop_col)

## ベストAICの変数を表示
print('\n--- best ---')
print('best AIC  : ', best_aic)
print('best vars : ', best_vars)
print('best coefs: ', best_coefs)

[Execution Results]
Part of it is cut off.

The best model with the minimum AIC was a multiple regression model with a constant, X1, and X2.
Expressed as a regression equation, it is $${y = 621.426 + 11.858 x_1 - 5.973 x_2}$$.

I wrote my own variable selection program for the first time.
I'm glad it managed to work!

I realized later that the implementation of the Stepwise method is very similar to the implementation for selecting the optimal values for p, d, and q in Chapter 6, 'Time Series ARIMA Models'.

That concludes the detour copying code for Chapter 2.

Series Articles

Next Article

Previous Article

Table of Contents

Blog Introduction

I am writing seven series of articles on note.
Please feel free to take a look!

1. Relaxed Statistics

This is a blog that roughly explores probability and statistics using the Statistics Certification Grade 2 workbook as a guide.
Feel free to read it as if it were casual conversation. Please do take a look.
It corresponds to the CBT-compatible version of the official Statistics Certification Grade 2 workbook.
Sample code for Python and EXCEL is also available.

2. Experiment! Fun Bayesian Modeling 1 & 2 with PyMC Ver. 5

I will draw and analyze the Bayesian models used in psychological research from the books 'Fun Bayesian Modeling' and 'Fun Bayesian Modeling 2' using PyMC Ver. 5.
Starting with these books, many Bayesian models are written in R + Stan.
I strive to explore the possibilities of PyMC and make Bayesian modeling easy to practice.
Since these are familiar and easy-to-visualize themes, please try running them with PyMC and let's enjoy it together!

3. Experiment! Bayesian Modeling from Iwanami Data Science Vol. 1 with PyMC Ver. 5

I will draw and analyze the Bayesian models by four Bayesians from the book 'Experiment! Iwanami Data Science Vol. 1' using PyMC Ver. 5.
This book is a great resource for learning the basics of Bayesian programming.
I feel like I've become friends with Bayesian methods by happily running PyMC models.
Everyone, please try running them with PyMC, and let's play and learn together!

4. Fun Code Copying: Bayesian, Python, etc.

I will blog about the results of my 'book code copying activities' for Bayesian, Python, and others.
I am mainly working on translating them into Python.
I hope this becomes useful sample code for fellow code-copying enthusiasts 🍀

5. Introduction to Time Series Analysis for Psychology with R and Stan, using Python and PyMC Ver. 5

I will practice the time series analysis from the book 'Introduction to Time Series Analysis for Psychology with R and Stan' using Python and PyMC Ver. 5.
This book is packed with time series analysis themes!
I realized the depth of time series analysis.
I will enjoy learning time series analysis with my favorite language, Python.

6. Writing about Data Science-like things

I write columns on statistics, data analysis, AI, machine learning, and Python on an irregular basis.
There are many articles related to statistics and data science books.
Series on "Statistics," "Python," "Mathematics and Python," and "R" have been created.

7. Python Machine Learning Programming Practice Notes

I wrote articles about my various thoughts when studying the book "Python Machine Learning Programming: PyTorch & scikit-learn Edition."
This book is a textbook for scikit-learn and PyTorch.
Please feel free to give it a try if you like.

Thank you very much for reading until the end.