Copying "Introduction to Multivariate Analysis for Beginners" in Python Vol. 26 - Chapter 5 "Discriminant Analysis for Beginners" (5) Various Classification Algorithms

Chapter 5 "Discriminant Analysis for Beginners"

Authors of the book: Dr. Sadao Ishimura, Dr. Koshiro Ishimura

This is a record of Python copying activities for Chapter 5 "Discriminant Analysis for Beginners" of the book "Introduction to Multivariate Analysis for Beginners".

This is a series where we learn the basics of multivariate analysis by copying code along with Python.

Discriminant analysis corresponds to a "classification" task in the context of machine learning.
In this article, I will borrow the example data from the book and visualize it using various classification algorithms from the machine learning library scikit-learn ! This is
all additional time not included in the book!

Let's get familiar with discriminant (classification) algorithms and broaden our range of data analysis skills.

We will proceed with ChatGPT-assisted learning!
Now, let's open the book and embark on a journey of multivariate analysis 🚀

Introduction

This blog series introduces the "joy of multivariate analysis" gained through copying the Python code from the book "Introduction to Multivariate Analysis for Beginners" (Tokyo Tosho, referred to as "the text").

Information on the book and citation notation are posted in the linked article.

Chapter 5: Discriminant Analysis for Beginners

This article is related to the discriminant analysis in Chapter 5, but it is not related to each specific section.

The data used in this article is cited directly from the data published in the text.
For data with a small number of entries, I register the data in the code, and for data with a large number of entries, I convert it to a CSV file and load it.

I will import the libraries used in this article.

## インポート

# 数値計算
import numpy as np
import pandas as pd

# 描画
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['font.family'] = 'Meiryo'  # または import japanize_matplotlib

What we will do in this article

We will train the example data from the text using 12 classification algorithms and draw decision boundaries.
We will simply admire the decision boundaries.

✅ What we will not do
・We will not touch on the specifics of each algorithm.
・We will not perform validation and will not address overfitting.

Preparation for analysis

We will set up the data.
We will cite the example data from Table 5.1.1 on page 179 of the text.

## マーカー測定結果 p.179 表5.1.1

data1 = pd.DataFrame(
    {'被験者No.': range(1, 16),
     'マーカーA': [3.4, 3.9, 2.2, 3.5, 4.1, 3.7, 2.8,
                  1.4, 2.4, 2.8, 1.7, 2.3, 1.9, 2.7, 1.3],
     'マーカーB': [2.9, 2.4, 3.8, 4.8, 3.2, 4.1, 4.2,
                  3.5, 2.6, 2.3, 2.6, 1.6, 2.1, 3.5, 1.9],
     '結果': np.hstack([np.ones(7), np.zeros(8)]).astype(int)})
data1

[Execution Result]
Result = 0 is Group 2 (prostatic hyperplasia), Result = 1 is Group 1 (prostate cancer).
The sample size is 7 for Group 1 and 8 for Group 2.

Split the data into explanatory variables and objective variables, and create prediction data (Mr. S's measurement results).

## データセットの作成

# 説明変数 X と目的変数 y の分離
X = data1[['マーカーA', 'マーカーB']].values
y = data1['結果'].values

# 予測データ shape(n, d)
X_new = np.array([[2.7, 3.1]])

[Execution Result] None

Define the helper function plot_decision_boundary for drawing decision boundaries.
Load the classifier clf created by each algorithm, and draw a scatter plot of the data and the boundary line (decision boundary).
Use scikit-learn's DecisionBoundaryDisplay().

## 決定境界描画ヘルパー関数の定義

# 追加インポート
from sklearn.inspection import DecisionBoundaryDisplay

def plot_decision_boundary(clf, title):
    ## 描画
    # 描画領域の設定
    fig, ax = plt.subplots(figsize=(6, 6))

    # 境界の領域の塗りつぶし描画
    DecisionBoundaryDisplay.from_estimator(
        clf,
        X,
        response_method='predict_proba',
        plot_method='pcolormesh',
        ax=ax,
        cmap='coolwarm_r',
        alpha=0.1,
    )

    # 境界線の描画
    DecisionBoundaryDisplay.from_estimator(
        clf,
        X,
        response_method='predict_proba',
        plot_method='contour',
        ax=ax,
        alpha=1.0,
        cmap='hsv',
        levels=[0.5],
    )

    # 実測値の散布図の描画
    y_pred = clf.predict(X)
    sns.scatterplot(data=data1, x='マーカーA', y='マーカーB', s=100,
                    hue=np.where(y==1, '実測 $G_1$', '実測 $G_2$'),
                    palette=['tab:blue', 'tomato'],
                    style=np.where(y_pred==1, '予測 $G_1$', '予測 $G_2$'),
                    zorder=10,
    )
    
    # Sさんの位置の描画
    plt.scatter(x=X_new[0, 0], y=X_new[0, 1], marker='*', s=200,
                color='tab:orange', zorder=10, label='S')
    
    # テキストの表示
    plt.text(x=1.7, y=5.2, s='グループ $G_1$', fontsize=16, color='blue')
    plt.text(x=1, y=1, s='グループ $G_2$', fontsize=16, color='tab:red')


    # 修飾
    plt.title(title)
    plt.gca().set_aspect('equal')
    plt.legend();

[Execution Result] None

12 Classification Algorithms

Import them while also listing the algorithms.
The criterion for selection is that they can be used with the "decision boundary drawing helper function".

## 追加インポート

# 線形判別分析 LDA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
# ロジスティック回帰
from sklearn.linear_model import LogisticRegression

# 二次判別分析 QDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
# ガウシアンナイーブベイズ
from sklearn.naive_bayes import GaussianNB
# ガウス過程
from sklearn.gaussian_process import GaussianProcessClassifier

# 決定木
from sklearn.tree import DecisionTreeClassifier
# ランダムフォレスト
from sklearn.ensemble import RandomForestClassifier
# 勾配ブースティング
from sklearn.ensemble import GradientBoostingClassifier
# ヒストグラム勾配ブースティング
from sklearn.ensemble import HistGradientBoostingClassifier

# k近傍法
from sklearn.neighbors import KNeighborsClassifier
# サポートベクターマシン
from sklearn.svm import SVC
# 多層パーセプトロン（ニューラルネットワーク）
from sklearn.neural_network import MLPClassifier

[Execution Result] None

Now, let's enter the world of classification!

Linear Algorithms

🟦 Linear Discriminant Analysis (LDA)
This algorithm is close to the "linear discriminant function" in the text.
I used it in the first article on discriminant analysis.
When generating the instance, I do not provide arguments and build the model with the algorithm's default values.

# LinearDiscriminantAnalysis
clf01 = LinearDiscriminantAnalysis() # 事前分布 priors=[0.8, 0.3]
clf01.fit(X, y)
plot_decision_boundary(clf01, 'Linear Discriminant Analysis')

[Execution Result]

The solid red line is the "boundary line" (decision boundary).
The intensity of the color represents the magnitude of the probability value for each group.

The left (lower) side of the boundary line is Group 2 (prostatic hyperplasia), and the right (upper) side is Group 1 (prostate cancer).

The two points marked with a blue X and a red circle are misclassifications.
You can reduce the number of misclassifications by appropriately setting the priors argument.

I will continue like this!

🟦 Logistic Regression
This is an algorithm that has a "linear predictor" similar to the regression equation of multiple regression analysis.
I do not provide arguments to adjust classification performance and build the model with the algorithm's default values.

# LogisticRegression
clf02 = LogisticRegression() # 正則化の強さ（の逆数）C=0.09
clf02.fit(X, y)
plot_decision_boundary(clf02, 'Logistic Regression')

[Execution Result]
The two points marked with a blue X and a red circle are misclassifications.
You can reduce the number of misclassifications by appropriately setting the C argument (the inverse of regularization strength).

Algorithms Related to Curves and Normal Distributions

🟦 Quadratic Discriminant Analysis (QDA)
It has a sibling relationship with Linear Discriminant Analysis (LDA), and the boundary line becomes a curve.
I do not provide arguments to adjust classification performance and build the model with the algorithm's default values.
The boundary line is similar to the "discrimination by Mahalanobis distance" in the text.

# QuadraticDiscriminantAnalysis
clf03 = QuadraticDiscriminantAnalysis() # 事前分布 priors=[0.4, 0.6]
clf03.fit(X, y)
plot_decision_boundary(clf03, 'Quadratic Discriminant Analysis')

[Execution Result]
Two points, a blue cross and a red circle, are misclassified.
You can reduce the number of misclassifications by appropriately setting the priors argument.

🟦 Gaussian Naive Bayes
This is the familiar Naive Bayes, often cited in the context of "Naive Bayes being used for spam filters."
We will use Gaussian Naive Bayes, which handles continuous explanatory variables.
We will build the model using the algorithm's default values without providing arguments to adjust classification performance.

# GaussianNB
clf04 = GaussianNB() # 事前分布 priors=[0.8, 0.2]
clf04.fit(X, y)
plot_decision_boundary(clf04, 'Gaussian Naive Bayes')

[Execution Result]
Two points, a blue cross and a red circle, are misclassified.
You can reduce the number of misclassifications by appropriately setting the priors argument.

🟦 Gaussian Process
This is a classification algorithm using Gaussian processes.
It seems that flexible fitting can be achieved by refining the kernel function, which is applied to the components of the variance-covariance matrix of the multivariate normal distribution.
In this case, we have set "1.0 * RBF(1.0)" as the kernel function.
Also, for models that use random numbers, we fix the random seed with random_state.

# GaussianProcessClassifier

# 追加インポート
from sklearn.gaussian_process.kernels import RBF
# カーネルの設定
kernel = 1.0 * RBF(1.0)
# モデルの構築
clf05 = GaussianProcessClassifier(kernel=kernel, random_state=123)
clf05.fit(X, y)
plot_decision_boundary(clf05, 'Gaussian Process Classifier')

[Execution Result]
One point, a red circle, is misclassified.

Decision Tree-based Algorithms

The boundaries of decision tree-based algorithms appear to be linear or staircase-like.

🟦 Decision Tree
This is a classification algorithm using decision trees.
We will visualize the decision tree's classification logic later.
We will build the model using the algorithm's default values without providing arguments to adjust classification performance.
Note that for models using random numbers, we fix the random seed with random_state.

# DecisionTreeClassifier
clf06 = DecisionTreeClassifier(random_state=0)
clf06.fit(X, y)
plot_decision_boundary(clf06, 'DecisionTreeClassifier')

[Execution Result]
The boundary is formed by vertical and horizontal straight lines.
There are no misclassifications.

Let's visualize the decision tree's classification logic in the form of a tree diagram.

# 樹形図の描画

# 追加インポート
from sklearn.tree import plot_tree

# 描画
plt.figure(figsize=(5, 5))
plot_tree(clf06, filled=True, fontsize=10, rounded=True,
          feature_names=['マーカーA', 'マーカーB'], class_names=['G2', 'G1']);

[Execution Result]

This decision tree model shows that it classifies as "Group 1" if "Marker A value is greater than 3.1" or "Marker B value is greater than 3.65", and as Group 2 otherwise.
When this condition is visualized, it results in linear, staircase-like boundaries.

🟦 Random Forest
This is an "ensemble" algorithm that determines the final classification by integrating parallel predictions from multiple small decision trees.
We will build the model using the algorithm's default values without providing arguments to adjust classification performance.
Note that for models using random numbers, we fix the random seed with random_state.

# RandomForestClassifier
clf07 = RandomForestClassifier(random_state=123)
clf07.fit(X, y)
plot_decision_boundary(clf07, 'RandomForest Classifier')

[Execution Result]
There are no misclassifications.
The complex boundary might be a warning of "overfitting," where the model fits the data too closely.

🟦 Gradient Boosting
This is a "boosting" algorithm that performs classification predictions by connecting multiple small decision trees in series.
The min_samples_leaf argument specifies the minimum number of data points included in the classification result of each decision tree.
Note that for models using random numbers, we fix the random seed with random_state.

# GradientBoostingClassifier
clf08 = GradientBoostingClassifier(min_samples_leaf=4, random_state=123)
clf08.fit(X, y)
plot_decision_boundary(clf08, 'Gradient Boosting Classifier')

[Execution Result]
There are no misclassifications.
The "dent" on the upper right is characteristic.

🟦 Histogram-based Gradient Boosting
This is a type of "boosting" algorithm that performs classification predictions by connecting multiple small decision trees in series, and like LightGBM, it is an algorithm that "histograms" the explanatory variables.
The min_samples_leaf argument specifies the minimum number of data points included in the classification result of each decision tree.
Note that for models using random numbers, we fix the random seed with random_state.

# HistGradientBoostingClassifier

clf09 = HistGradientBoostingClassifier(min_samples_leaf=4, random_state=123)
clf09.fit(X, y)
plot_decision_boundary(clf09, 'Histogram-based Gradient Boosting Classifier')

[Execution Result]
There are no misclassifications.
The "dent" on the upper right is characteristic. Pay attention to the "thinness" of the probability values.

Other Algorithms

These are algorithms capable of handling complex boundaries.

🟦 k-Nearest Neighbors
This is an algorithm that classifies data points into the group with the highest count among the k nearest neighbors.
The argument p is the power parameter for the Minkowski distance, and n_neighbors is the number of neighbors, k.

# KNeighborsClassifier
clf10 = KNeighborsClassifier(p=2, n_neighbors=1)
clf10.fit(X, y)
plot_decision_boundary(clf10, 'KNeighbors Classifier')

【Execution Results】
There are no misclassifications.
The convex part in the center gives an impression of overfitting.
It might be difficult to classify Group 2, which is included in the protrusion.

🟦 Support Vector Machine
This is a classification algorithm that is difficult to explain in a single word (my personal opinion).
The argument C is the reciprocal of the regularization strength.
Note that for models using random numbers, the random seed is fixed with random_state.

# SVC
clf11 = SVC(C=10, probability=True, random_state=5)
clf11.fit(X, y)
plot_decision_boundary(clf11, 'Support Vector Classification')

【Execution Results】
There are no misclassifications.
The convex part in the center gives an impression of overfitting.
It might be difficult to classify Group 2, which is included in the protrusion.

🟦 Multilayer Perceptron
This is a neural network classification algorithm.
The model is built using the algorithm's default values without providing arguments to adjust classification performance.
Note that for models using random numbers, the random seed is fixed with random_state.

# MLPClassifier
clf12 = MLPClassifier(max_iter=10000, random_state=5)
clf12.fit(X, y)
plot_decision_boundary(clf12, 'Multi-layer Perceptron classifier')

【Execution Results】
There are no misclassifications.
Although it seems linear, the slight convex part in the center gives an impression of overfitting.
It is trying hard (perhaps forced?) to fit the data (overfitting?).

Did you get closer to understanding the characteristics of the algorithms?
I think you can get even closer to the "feel" of the algorithms by carefully observing the characteristics of each model and checking how the boundaries change by varying the argument values!

This article marks the end of the discriminant analysis section.

ChatGPT will wrap up the end of the article.
While looking back at the various classification models.

📘 A word from ChatGPT:

That is all for this coding session.

Series Articles

Next Article

Previous Article

Table of Contents

Blog Introduction

I am writing seven series of articles on note.
Please come and take a look!

1. Relaxed Statistics

This is a blog that roughly digs into probability and statistics using the Statistics Grade 2 workbook as a guide.
Feel free to read it like casual conversation. Please come and take a look.
It corresponds to the Statistics Grade 2 Official Workbook CBT version.
Sample code for Python and EXCEL is also available.

2. Experiment! Fun Bayesian Modeling 1 & 2 with PyMC Ver. 5

I will draw and analyze the Bayesian models used in psychological research from the books 'Fun Bayesian Modeling' and 'Fun Bayesian Modeling 2' using PyMC Ver. 5.
Like this book, many Bayesian models are written in R + Stan.
I will strive to explore the possibilities of PyMC and make Bayesian modeling easy to practice.
Since these are familiar and easy-to-visualize themes, please try running them in PyMC and let's enjoy them together!

3. Experiment! Bayesian Modeling from Iwanami Data Science 1 with PyMC Ver. 5

I will draw and analyze the Bayesian models by four Bayesians from the book 'Experiment! Iwanami Data Science Vol. 1' using PyMC Ver. 5.
This book is a great resource for learning the basics of Bayesian programming.
I feel like I've become friends with Bayesian methods by happily running PyMC models.
Everyone, please try running them in PyMC, and let's play and learn together!

4. Fun Copying: Bayesian, Python, etc.

I will blog about the results of my 'book copying activities' for Bayesian, Python, and others.
I am mainly working on translations into Python.
I hope this serves as sample code for fellow learners who are also copying code. 🍀

5. Introduction to Time Series Analysis for Psychology with R and Stan, using Python and PyMC Ver. 5

I will practice the time series analysis from the book 'Introduction to Time Series Analysis for Psychology with R and Stan' using Python and PyMC Ver. 5.
This book is packed with themes on time series analysis!
I realized the depth of time series analysis.
I will enjoy learning time series analysis with my favorite language, Python.

6. Writing about Data Science-like things

I write columns on statistics, data analysis, AI, machine learning, and Python on an irregular basis.
There are many articles related to statistics and data science books.
Series on 'Statistics', 'Python', 'Math and Python', and 'R' have been created.

7. Practical Record of Python Machine Learning Programming

I wrote articles about my various thoughts when learning from the book 'Python Machine Learning Programming: PyTorch & scikit-learn Edition'.
This book is a textbook for scikit-learn and PyTorch.
Please feel free to try it out if you'd like.

Thank you very much for reading until the end.