chapter 5 支持向量机

OReilly.Hands-On Machine Learning with Scikit-Learn and TensorFlow读书笔记

chapter 5 支持向量机

Support Vector Machine (SVM) is capable of performing linear and nonlinear classification, regression, and even outlier detection.

SVM used to be the most popular ML model.

SVMs are particularly well suited for classification of complex but small- or medium-sized datasets.

5.1 Linear SVM Classification

Requirement: the dataset is linearly separable.
Principle: large margin classification. An SVM classifier can be thought of as fitting the widest possible margin between the classes.

The instances on the margin are called support vectors.

SVMs are sensitive to the feature scales. Better using $\text{StandardScaler}$ before training.

5.1.1 Soft Margin Classification

Hard margin classification: all instances are required to be off the margin and on the right side. First, it works only if the data is linearly separable. Second, it is quite sensitive to outlier.

Soft margin classification: to find a balance between keeping the interval as large as possible and limiting the margin violation, i.e., instances that end up in the middle of the interval or even on the wrong side.

Note that, an instance in the middle of the interval does not have to be wrongly classified. It may lies on the right side of the decision boundary.

In Scikit-Learn’s SVM classes, a $\text{C}$ hyperparameter is used to control this balance. A smaller $\text{C}$ indicates a wider interval but more margin violations. If your SVM model is overfitting, you can try regularizing it by reducing $\text{C}$.

import numpy as np
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC

iris=datasets.load_iris()
X=iris['data'][:,(2,3)]#petal length, petal width
y=(iris['target']==2).astype(np.float64) #iris-Virginica

svm_clf=Pipeline([
    ("scaler",StandardScaler()),
    ("linear_svc",LinearSVC(C=1,loss="hinge"))
])

svm_clf.fit(X,y)

svm_clf.predict([[5.5,1.7]]) #array([1.])

Alternatives:

• $\text{SVC(kernel="linear",C=1)}$

  slower, not recommended

• $\text{SGDClassifier(loss="hinge",alpha=1/(m*C))}$

  does not converge as fast as $\text{LinearSVC}$ class, but useful for huge datasets that do not fit in memory (out-of-core training), or to handle online classification tasks.

5.2 Nonlinear SVM Classification

For datasets that are not linearly separable, add new features such as polynomial features.

from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
X,y=make_moons()
polynomial_svm_clf=Pipeline([
    ('poly_features',PolynomialFeatures(degree=3)),
    ("scaler",StandardScaler()),
    ("svm_clf",LinearSVC(C=10,loss="hinge"))
])
polynomial_svm_clf.fit(X,y)

5.2.1 Polynomial Kernel

kernel trick: get the same results without adding polynomial features actually.

from sklearn.svm import SVC
poly_kernel_svm_clf=Pipeline([
    ("scaler",StandardScaler()),
    ("svm_clf",SVC(kernel="poly",degree=3,coef0=1,C=5))
])
poly_kernel_svm_clf.fit(X,y)

A hyperparameter $\text{coef0}$ can be used to control how much the model is influenced by high-degree polynomials versus low-degree polynomials.

A common way to find the right hyperparameter values is to use grid search ($\text{GridSearchCV}$ class).

5.2.2 Adding Similarity Features

Another way to introduce new features is to use a similarity function to compute instances similar to a particular landmark.

For example, use Gaussian Radial Basis Function (RBF) as similarity function, with $\gamma =0.3$.

A one-dimensional dataset with two landmarks $x_1=-2$ and $x_1=1$, where $x_1$ is the axis of the dataset.

Equation 5-1. Gaussian RBF
$$\varphi \gamma (\text{x},\ell )=\exp \left(-\gamma ||\text{x}-\ell |{|}^2\right)$$
It is a bell-shaped function varying from 0 (very far away from the landmark) to 1 (at the landmark).

For the instance $x_1=-1$, located at a distance of $1$ and $2$ to the two landmarks, the new features are $x_2=\exp (-0.3\times 1^2)\approx 0.74$ and $x_3=\exp (-0.3\times 2^2)\approx 0.30$.

How to select landmarks? A simple way is to use all the instances as landmarks. If the dataset contains $m$ instances with $n$ features, this results in a training set with $m$ instances and $m$ features if you drop the original features.

5.2.3 Gaussian RBF Kernel

Again, RBF kernel trick get same results as RBF similarity features without adding new features actually.

rbf_kernel_svm_clf=Pipeline([
    ("scaler",StandardScaler()),
    ("svm_clf",SVC(kernel="rbf",gamma=5,C=0.001))
])
rbf_kernel_svm_clf.fit(X,y)

We can see from Figure 5-9 that, increasing $\text{gamma}$ make the bell-shaped curve narrower, and as a result each instance’s range of influence is smaller: overfitting.

The usage of other kernels is rare.

As a rule of thumb, to select kernels for a specific task, you should always try the linear kernel first ($\text{LinearSVC}$ is much faster than $\text{SVC(kernel="linear")}$), especially if the training set is very large or if it has plenty of features. If the training set is not too large, try the Gaussian RBF kernel.

5.2.3 Computational Complexity

The $\text{LinearSVC}$ class is based on the liblinear library, which implements an optimized algorithm for linear SVMs. It does not support kernel trick, but scales linearly with the number of instances and the number of features: $O(m\times n)$.

The tolerance hyperparameter $ϵ$ ($\text{tol}$ in Scikit-Learn) controls the precision.

The $\text{SVC}$ class is based on the libsvm library, which implements SMO (sequential minimal Optimization) that supports the kernel trick. The training time complexity is usually between $O(m^2\times n)$ and $O(m^3\times n)$.

Table 5-1. Comparison of Scikit-Learn classes for SVM classification

5.3 SVM Regression

SVM Regression tries to fit as many instances as possible within the margin while limiting margin violations (i.e., instances off the street). The width of the street is controlled by a hyperparameter $ϵ$.

from sklearn.svm import LinearSVR
linear_svr_reg=Pipeline([
    ('scaler',StandardScaler()),
    ('svm_reg',LinearSVR(epsilon=1.5))
])
linear_svr_reg.fit(X,y)

from sklearn.svm import SVR
nonlinear_svr_reg=Pipeline([
    ('scaler',StandardScaler()),
    ('svm_reg',SVR(kernel="poly",degree=2,C=100,epsilon=0.1))
])
nonlinear_svr_reg.fit(X,y)

5.4 Under the Hood

5.4.1 Decision Function and Predictions

Equation 5-2. Linear SVM classifier prediction
$$\hat{y}=\{\begin{array}{c}0\text{ if }{\text{w}}^T\cdot \text{x}+b<0\\ 1\text{ if }{\text{w}}^T\cdot \text{x}+b\ge 0\end{array}$$
The decision function is of the form ${\text{w}}^T\cdot \text{x}+b=w_1{x}_1+\cdots +w_n{x}_n+b$.

$h={\text{w}}^T\cdot \text{x}+b$ of each and every $h$ constitute a $n$-dimension hyperplane, whereas decision boundary $0={\text{w}}^T\cdot \text{x}+b$ is the intersection of $h={\text{w}}^T\cdot \text{x}+b$ and $h=0$, which is a $(n-1)$-dimension hyperplane.

5.4.2 Training Objective

Hard margin SVM:

Objective: make the interval as large as possible.

Analysis: Through observation, the smaller the norm $||\text{w}||$ of the weight vector $\text{w}$, the larger the interval. Minimizing $||\text{w}||$ is equivalent to minimizing $\frac{1}{2}||\text{w}|{|}^2$, which is equal to $\frac{1}{2}{\text{w}}^T\text{w}$.

Constraints: if ${\text{w}}^T\cdot {\text{x}}^{(i)}+b\ge 1$, it should be classified as positive. Otherwise, it should be classified as negative. By introducing a variable $t^{(i)}$, which is $1$ when ${\text{w}}^T\cdot {\text{x}}^{(i)}+b\ge 1$ and $0$ otherwise, we can express this constraint as $t^{(i)}({\text{w}}^T\cdot {\text{x}}^{(i)}+b)\ge 1$.

Equation 5-3. Hard margin linear SVM classifier objective
KaTeX parse error: No such environment: align* at position 8: \begin{̲a̲l̲i̲g̲n̲*̲}̲ \mathop{\textr…
Soft margin SVM

Introduce a slack variable ${\zeta }^{(i)}\ge 0$ for each instance ${\text{x}}^{(i)}$ to measure how much ${\text{x}}^{(i)}$ is allowed to violate the margin and $\text{C}$ introduced in 5.1.1 to control the balance between making the interval as large as possible and

Equation 5-4. Soft margin linear SVM classifier objective
KaTeX parse error: No such environment: align* at position 8: \begin{̲a̲l̲i̲g̲n̲*̲}̲ \mathop{\textr…

5.4.3 Quadratic Programming

Equation 5-5. Quadratic Programming problem
KaTeX parse error: No such environment: align* at position 8: \begin{̲a̲l̲i̲g̲n̲*̲}̲ \mathop{\textr…
The expression $\text{A}\cdot \text{p}\le \text{b}$ actually defines $n_c$ constraints: ${\text{p}}^T\cdot {\text{a}}^{(i)}\le b^{(i)}$ for $i=1,2,\cdots ,n_c$, where ${\mathbf{a}}^{(i)}$ is the vector containing the elements of the $i$-th row of $\text{A}$ and $b^{(i)}$ is the $i$-th element of $\text{b}$.

Hard margin linear SVM classifier objective:

• $n_p=n+1$, where $n$ is the number of features (the +1 is for the bias term)
• $n_c=m$, where $m$ is the number of training instances
• $\text{H}$ is the $n_p\times n_p$ identity matrix, except with a zero in the top-left cell (to ignore the bias term)
• $\text{f}=0$, an $n_p$-dimensional vector full of 0s.
• $\mathbf{b}=1$, an $n_c$-dimensional vector full of 1s.
• ${\mathbf{a}}^{(i)}=–t^{(i)}{\dot{\mathbf{x}}}^{(i)}$, where ${\dot{\mathbf{x}}}^{(i)}$ is equal to ${\mathbf{x}}^{(i)}$ with an extra bias feature ${\dot{\mathbf{x}}}_0=1$.

So one way to train a hard margin linear SVM classifier is just to use an off-the-shelf QP solver by passing it the preceding parameters.

Using kernel trick makes the algorithm more efficient than directly using QP. To use the kernel trick, a different optimization problem is to be solved.

5.4.4 The Dual Problem

Given a constrained optimization problem, known as the primal problem, it is possible to express a different but closely related problem, called its dual problem. The solution to the dual problem gives a lower bound to the solution of primal problem, but under some conditions it can even have the same solutions as the primal problem. The conditions required for same solutions are as follows.

• the objective function is convex
• the inequality constraints are continuously differentiable and convex functions.

Equation 5-6. Dual form of the linear SVM objective
$$\begin{gathered} \underset{\alpha }{\mathrm{minimize}}\frac{1}{2}\sum _{i=1}^m\sum _{j=1}^m{\alpha }^{(i)}{\alpha }^{(j)}{t}^{(i)}{t}^{(j)}{\text{x}}^{(i{)}^T}{\text{x}}^{(j)}-\sum _{i=1}^m{\alpha }^{(i)} \\ \text{subject to }{\alpha }^{(i)}\ge 0\text{ for }i=1,2,\cdots ,m \end{gathered}$$
Once you find the vector $\hat{\alpha }$ that minimizes Equation 5-6 (using a QP solver), you can compute $\hat{\text{w}}$ and $\hat{b}$ that minimize the primal problem by using Equation 5-7.

Equation 5-7. From the dual solution to the primal solution
$$\begin{gathered} \hat{\text{w}}=\sum _{i=1}^m{\hat{\alpha }}^{(i)}{t}^{(i)}{\text{x}}^{(i)} \\ \hat{b}=\frac{1}{n_s}\underset{{\hat{\alpha }}^{(i)}>0}{\sum _{i=1}^m}\left(1-t^{(i)}\left(\hat{\text{w}}\cdot {\text{x}}^{(i)}\right)\right) \end{gathered}$$
The dual problem is faster to solve than the primal when the number of training instances is smaller than the number of features.

5.4.5 Kernelized SVM

Equation 5-8. Second-degree polynomial mapping
$$\varphi (\text{x})=\varphi \left(\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)\right)=\left(\begin{array}{c}{x}_1^2\\ \sqrt{2}{x}_1{x}_2\\ x_2^2\end{array}\right)$$
Equation 5-9. Kernel trick for a 2nd-degree polynomial mapping
$$\begin{gathered} \varphi (\text{a}{)}^T\varphi (\text{b})={\left(\begin{array}{c}{a}_1^2\\ \sqrt{2}{a}_1{a}_2\\ a_2^2\end{array}\right)}^T\cdot \left(\begin{array}{c}{b}_1^2\\ \sqrt{2}{b}_1{b}_2\\ b_2^2\end{array}\right)=a_1^2{b}_1^2+2a_1{b}_1{a}_2{b}_2+a_2^2{b}_2^2 \\ =(a_1{b}_1+a_2{b}_2{)}^2={\left({\left(\begin{array}{c}{a}_1\\ a_2\end{array}\right)}^T\cdot \left(\begin{array}{c}{b}_1\\ b_2\end{array}\right)\right)}^2={\left({\text{a}}^T\cdot \text{b}\right)}^2 \end{gathered}$$
Now here is the key insight: if the data is not linear separable, then you may transform it into high dimensional space. If you apply the transformation $\varphi$ to all training instances, then the dual problem (see Equation 5-6) will contain the dot product $\varphi (x^{(i)}{)}^T\cdot \varphi (x^{(j)})$. But if $\varphi$ is the 2nd-degree polynomial transformation defined in Equation 5-8, then you can replace this dot product of transformed vectors simply by $(x^{(i{)}^T}\cdot x^{(j)}{)}^2$.

So you don’t actually need to transform the training instances at all: just replace the dot product by its square in Equation 5-6. The result will be strictly the same as if you went through the trouble of actually transforming the training set then fitting a linear SVM algorithm, but this trick makes the whole process much more computationally efficient. This is the essence of the kernel trick.

The function $K(\mathbf{a},\text{b})=({\text{a}}^T\cdot \text{b}{)}^2$ is called a 2nd-degree polynomial kernel. In Machine
Learning, a kernel is a function capable of computing the dot product $\varphi (\text{a}{)}^T\cdot \varphi (\text{b})$ based only on the original vectors $\text{a}$ and $\text{b}$, without having to compute (or even to know about) the transformation $\varphi$. Equation 5-10 lists some of the most commonly used kernels.
Equation 5-10. Common kernels
$$\begin{array}{rl}\text{Linear:}& K(\text{a},\text{b})={\text{a}}^T\cdot \text{b}\\ \text{Polynomial:}& K(\text{a},\text{b})={\left(\gamma {\text{a}}^T\cdot \text{b}+r\right)}^d\\ \text{Gaussian RBF:}& K(\text{a},\text{b})=\exp \left(-\gamma ||\text{a}-\text{b}|{|}^2\right)\\ \text{Sigmoid:}& K(\text{a},\text{b})=\tanh \left(\gamma {\text{a}}^T\cdot \text{b}+r\right)\end{array}$$
Mercer’s Theorem: if a function $K(\text{a},\text{b})$ respects Mercer’s conditions ($K$ must be continuous, symmetric in its arguments so $K(\text{a},\text{b})=K(\text{b},\text{a})$, etc.), then there exists a function $\varphi$ that maps $\text{a}$ and $\text{b}$ into another space (possibly with much higher dimensions) such that $K(\text{a},\text{b})=\varphi (\text{a}{)}^T\cdot \varphi (\text{b})$. So you can use $K$ as a kernel since you know $\varphi$ exists, even if you don’t know what $\varphi$ is. In the case of the Gaussian RBF kernel, it can be shown that $\varphi$ actually maps each training instance to an infinite-dimensional space, so it’s a good thing you don’t need to actually perform the mapping!

Using kernel trick makes the parameters $\text{w}$ and $b$ hard to obtain, as it may have as many dimension as $\varphi (x^{(i)})$, which may be infinite. However, you can ignore the step of computing the model parameters and directly jump to the prediction step.

Equation 5-11. Making predictions with a kernelized SVM
$$\begin{gathered} h_{\hat{\text{w}},\hat{b}}\left(\varphi \left({\text{x}}^{(n)}\right)\right)={\hat{\text{w}}}^T\cdot \varphi \left({\text{x}}^{(n)}\right)+\hat{b}={\left(\sum _{i=1}^m{\hat{\alpha }}^{(i)}{t}^{(i)}\varphi \left({\text{x}}^{(i)}\right)\right)}^T\cdot \varphi \left({\text{x}}^{(n)}\right)+\hat{b} \\ =\sum _{i=1}^m{\hat{\alpha }}^{(i)}{t}^{(i)}\left(\varphi {\left({\text{x}}^{(i)}\right)}^T\cdot \varphi \left({\text{x}}^{(n)}\right)\right)+\hat{b} \\ =\underset{{\hat{\alpha }}^{(i)}>0}{\sum _{i=1}^m}{\hat{\alpha }}^{(i)}{t}^{(i)}K\left({\text{x}}^{(i)},{\text{x}}^{(n)}\right)+\hat{b} \end{gathered}$$
Where ${\text{x}}^{(n)}$ is a new instance.

Equation 5-12. Computing the bias term using the kernel trick
$$\begin{gathered} \hat{b}=\frac{1}{n_s}\underset{{\hat{\alpha }}^{(i)}>0}{\sum _{i=1}^m}\left(1-t^{(i)}{\hat{\text{w}}}^T\cdot \varphi \left({\text{x}}^{(i)}\right)\right) \\ =\frac{1}{n_s}\underset{{\hat{\alpha }}^{(i)}>0}{\sum _{i=1}^m}\left(1-t^{(i)}{\left(\sum _{j=1}^m{\hat{\alpha }}^{(j)}{t}^{(j)}\varphi \left({\text{x}}^{(j)}\right)\right)}^T\cdot \varphi \left({\text{x}}^{(i)}\right)\right) \\ =\frac{1}{n_s}\underset{{\hat{\alpha }}^{(i)}>0}{\sum _{i=1}^m}\left(1-t^{(i)}\underset{{\hat{\alpha }}^{(j)}>0}{\sum _{j=1}^m}{\hat{\alpha }}^{(j)}{t}^{(j)}K\left({\text{x}}^{(j)}\cdot {\text{x}}^{(i)}\right)\right) \end{gathered}$$

5.4.6 Online SVMs

For training online linear SVM classifiers, one way is to use Gradient Descent to minimized the cost function in Equation 5-13.

Equation 5-13. Linear SVM classifier cost function
$$J(\text{w},b)=\frac{1}{2}{\text{w}}^T\text{w}+C\sum _{i=1}^m\max \left(0,1-t^{(i)}\left({\text{w}}^T\cdot {\text{x}}^{(i)}+b\right)\right)$$
The first sum in the cost function will push the model to have a small weight vector w, leading to a larger margin. The second sum computes the total of all margin violations. An instance’s margin violation is equal to 0 if it is located off the street and on the correct side, or else it is proportional to the distance to the correct side of the street.

Hinge Loss: the function $\max (0,1-t)$ is called the hinge loss. It is equal to 0 when $t\ge 1$ and $1-t$ if $t<1$. Its derivative (slope) is equal to -1 if $t<1$ and 0 if $t>1$. It is not differentiable at $t=1$, but just like for Lasso Regression you can still use Gradient Descent using any sub-derivative at $t=0$ (i.e., any value between -1 and 0).

To sum up, SVM is one of the most popular ML model. It serves as linear and nonlinear classifier, and performs linear and nonlinear regression.

1. Linear SVM Classification

   preprocessing: $\text{StandardScaler}$

   Balance between keeping the interval as large as possible and limiting the margin violation is controlled by the $\text{C}$ hyperparameter in Scikit-Learn’s SVM classes. A smaller $\text{C}$ value leads to a wider interval but more margin violation.

   Three ways to perform linear SVM classification:

   • $\text{LinearSVC(loss="hinge",C=1)}$ fast
   • $\text{SVC(kernel="linear",C=1)}$ slow
   • $\text{SGDRegression(loss="hinge", alpha=1/(m*C))}$fast

2. Nonlinear SVM Classification

   Four ways to perform nonlinear SVM classification:

   • $\text{PolynomialFeatures(degree=3)}$

     $\text{LinearSVC(loss="hinge",C=10)}$

   • polynomial kernel trick: $\text{SVC(kernel="poly",degree=3,coef0=1,C=5)}$

     $\text{coef0}$ controls the influence of high degree versus low degree

   • adding similarity features by using similarity functions, such as Gaussian RBF

   • Gaussian RBF Kernel: $\text{SVC(kernel="rbf",gamma=5,,C=0.001)}$

3. SVM Regression

   • $\text{LinearSVR(epsilon=1.5)}$

     $\text{epsilon}$ controls the width of the interval

   • $\text{SVR(kernel="poly",degree=2,C=100,epsilon=0.1)}$

4. Under the Hood

   decision function: ${\text{w}}^T\cdot \text{x}+b$

   decision boundary: ${\text{w}}^T\cdot \text{x}+b=0$

   training objective and constraints:

   • hard margin classification

     $\underset{\text{w},b}{\mathrm{minimize}}$ $\frac{1}{2}{\text{w}}^T\cdot \text{w}$

     subject to $t^{(i)}\left({\text{w}}^T\cdot {\text{x}}^{(i)}+b\right)\ge 1$ for $i=1,2,\cdots ,m$

   • soft margin classification

     $\underset{\text{w},b}{\mathrm{minimize}}$ $\frac{1}{2}{\text{w}}^T\cdot \text{w}+C{\sum }_{i=1}^m{\zeta }^{(i)}$

     subject to $t^{(i)}\left({\text{w}}^T\cdot {\text{x}}^{(i)}+b\right)\ge 1-{\zeta }^{(i)}$ for ${\zeta }^{(i)}\ge 0,i=1,2,\cdots ,m$

   quadratic programming: solved by an off-the-shelf QP solver

   dual problem: under some conditions, has the same solution as the primal problem.

   kernelized SVM: kernel functions make the dot product in high dimension space easy to compute.

   online SVM:

   cost function

$$J(\text{w},b)=\frac{1}{2}{\text{w}}^T\text{w}+C\sum _{i=1}^m\max \left(0,1-t^{(i)}\left({\text{w}}^T\cdot {\text{x}}^{(i)}+b\right)\right)$$