梯度下降法公式推导及示例迭代

一个维度数据

只有$x$和$y$的数据，$x$是输入，$y$是标记。

例如：

• 房间的尺寸和价格。
• 西瓜的大小和价格。

目标函数（直线）

$$\begin{gathered} f(x)=kx+b \\ ={\theta }_{1}x+{\theta }_0 \\ ={\theta }_0+{\theta }_{1}x \end{gathered}$$

最小二乘

$$L=\frac{1}{2}[(y_1-f(x_1){)}^2+(y_2-f(x_2){)}^2+\cdots +(y_n-f(x_n){)}^2]=\frac{1}{2}\sum _{i=1}^n(y_i-f(x_i){)}^2$$

其中$y$是训练样本的标记，$f(x)$是要备选函数，$L$是损失函数。让损失函数最小，即，每个训练样本的标记到备选函数值的间距最小。

最优化问题（求最小）

$$g(x)=(x-1{)}^2=x^2-2x+1$$

$$\frac{dg(x)}{dx}=\frac{d}{dx}g(x)=g^{\prime }(x)=2x-2$$

$$x<1,g^{\prime }(0)=-2<0,g(x)减小$$

$$x=1,g^{\prime }(1)=0$$

$$x>1,g^{\prime }(2)=2>0,g(x)增大$$

梯度下降

$$x:=x-\eta \frac{d}{dx}g(x)=x-\eta g^{\prime }(x)=x-\eta (2x-2)$$

求最小$L$的${\theta }_0$和${\theta }_1$

$$f(x)={\theta }_0+{\theta }_{1}x$$

$$L=\frac{1}{2}\sum _{i=1}^n(y_i-f(x_i){)}^2$$

$${\theta }_0:={\theta }_0-\eta \frac{\partial L}{\partial {\theta }_0}$$

$${\theta }_1:={\theta }_1-\eta \frac{\partial L}{\partial {\theta }_1}$$

$$\frac{\partial L}{\partial {\theta }_0}=\frac{\partial L}{\partial f}\cdot \frac{\partial f}{\partial {\theta }_0}$$

$$\frac{\partial L}{\partial {\theta }_1}=\frac{\partial L}{\partial f}\cdot \frac{\partial f}{\partial {\theta }_1}$$

$$\begin{gathered} \frac{\partial L}{\partial f}=\frac{\partial }{\partial f}L \\ =\frac{\partial }{\partial f}(\frac{1}{2}\sum _{i=1}^n(y_i-f(x_i){)}^2) \\ =\frac{1}{2}\sum _{i=1}^n(\frac{\partial }{\partial f}(y_i-f(x_i){)}^2) \\ =\frac{1}{2}\sum _{i=1}^n(\frac{\partial }{\partial f}(y_i^2-2y_{i}f(x_i)+f(x_i{)}^2)) \\ =\frac{1}{2}\sum _{i=1}^n(\frac{\partial }{\partial f}(-2y_i+2f(x_i))) \\ =\sum _{i=1}^n(f(x_i)-y_i) \\ =\sum _{i=1}^n({\theta }_0+{\theta }_1{x}_i-y_i) \\ =({\theta }_0+{\theta }_1{x}_1-y_1)+({\theta }_0+{\theta }_1{x}_2-y_2)+\cdots +({\theta }_0+{\theta }_1{x}_n-y_n) \\ =n{\theta }_0+{\theta }_1(x_1+x_2+\cdots +x_n)-(y_1+y_2+\cdots +y_n) \\ =n{\theta }_0+{\theta }_1\sum _{i=1}^n{x}_i-\sum _{i=1}^n{y}_i \\ =n({\theta }_0+{\theta }_1\bar{x}-\bar{y}) \end{gathered}$$

$$\frac{\partial f}{\partial {\theta }_0}=\frac{\partial }{\partial {\theta }_0}f=\frac{\partial }{\partial {\theta }_0}({\theta }_0+{\theta }_{1}x)=1$$

$$\frac{\partial f}{\partial {\theta }_1}=\frac{\partial }{\partial {\theta }_1}f=\frac{\partial }{\partial {\theta }_1}({\theta }_0+{\theta }_{1}x)=x$$

$$\frac{\partial L}{\partial {\theta }_0}=\sum _{i=1}^n(f(x_i)-y_i)$$

$$\frac{\partial L}{\partial {\theta }_1}=\sum _{i=1}^n(f(x_i)-y_i)x_i$$

最终公式

$${\theta }_0:={\theta }_0-\eta \sum _{i=1}^n(f(x_i)-y_i)$$

$${\theta }_1:={\theta }_1-\eta \sum _{i=1}^n(f(x_i)-y_i)x_i$$

举例

房子尺寸和价格：

• 尺寸$x$：$[1,2,3]$（百平方米）
• 价格$y$：$[2,4,6]$（十万元）
• 初始$\theta$：$[0,0]$（线性回归中初始的$\theta$都设置为0即可）
• 学习率$\eta$：$0.1$

第1次迭代

$${\theta }_0:={\theta }_0-\eta \sum _{i=1}^n(f(x_i)-y_i)={\theta }_0-\eta \sum _{i=1}^n({\theta }_0+{\theta }_1{x}_i-y_i)=0-0.1\ast [(0+0\ast 1-2)+(0+0\ast 2-4)+(0+0\ast 3-6)]=0-0.1\ast -12=0+1.2=1.2$$

$${\theta }_1:={\theta }_1-\eta \sum _{i=1}^n(f(x_i)-y_i)x_i={\theta }_0-\eta \sum _{i=1}^n({\theta }_0+{\theta }_1{x}_i-y_i)x_i=0-0.1\ast [(0+0\ast 1-2)\ast 1+(0+0\ast 2-4)\ast 2+(0+0\ast 3-6)\ast 3]=0-0.1\ast -(2+8+18)=0-0.1\ast -28=2.8$$

• 新 $\theta$：$[1.2,2.8]$

第2次迭代

$${\theta }_0:={\theta }_0-\eta \sum _{i=1}^n(f(x_i)-y_i)={\theta }_0-\eta \sum _{i=1}^n({\theta }_0+{\theta }_1{x}_i-y_i)=1.2-0.1\ast [(1.2+2.8\ast 1-2)+(1.2+2.8\ast 2-4)+(1.2+2.8\ast 3-6)]=1.2-0.1\ast +8.4=0.36$$

$${\theta }_1:={\theta }_1-\eta \sum _{i=1}^n(f(x_i)-y_i)x_i={\theta }_0-\eta \sum _{i=1}^n({\theta }_0+{\theta }_1{x}_i-y_i)x_i=2.8-0.1\ast [(1.2+2.8\ast 1-2)\ast 1+(1.2+2.8\ast 2-4)\ast 2+(1.2+2.8\ast 3-6)\ast 3]=2.8-0.1\ast (2+5.6+10.8)=2.8-0.1\ast 18.4=0.96$$

新 $\theta$：$[0.36,0.96]$

第100次迭代

$\theta$：$[0.0187,1.9917]$

根据样本中的的$x$和$y$，可以直接算出${\theta }_0$为$0$，${\theta }_1$为$2$，即，$f(x)=2x$。此处使用梯度下降演示如何通过迭代的方式逼近理想$\theta$。

在线性回归中使用“正规方程法”能最快算出$\theta$的值，但梯度下降更为通用。在公式较为复杂，维度较多（10000以上）时，“正规方程法”通常无法进行计算，此时只能使用梯度下降方法。

关于初始$\theta$设置为0，是因为在线性回归中，一定能找到最小值，所以初始设置为多少无所谓。比较好的方式是小随机数初始化。

在神经网络中初始$\theta$设置为0是不可取的。

示例代码

def get(theta_0, theta_1):
    '''
    f(x) = theta_0 + theta_1x
    '''
    eta = 0.1
    return {
        "theta_0": theta_0 - eta * ((theta_0 + theta_1 * 1 - 2)     + (theta_0 + theta_1 * 2 - 4)     + (theta_0 + theta_1 * 3 - 6)), 
        "theta_1": theta_1 - eta * ((theta_0 + theta_1 * 1 - 2) * 1 + (theta_0 + theta_1 * 2 - 4) * 2 + (theta_0 + theta_1 * 3 - 6) * 3)
    }

if __name__ == "__main__":
    theta = {
        "theta_0": 0,
        "theta_1": 0,
    }
    print(theta)

    for i in range(100):
        theta = get(**theta)
        print(theta)

目标函数（曲线）

$$\begin{gathered} f(x)={\theta }_2{x}^2+{\theta }_{1}x+{\theta }_0 \\ ={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2 \end{gathered}$$

$${\theta }_0:={\theta }_0-\eta \sum _{i=1}^n(f(x_i)-y_i)$$

$${\theta }_1:={\theta }_1-\eta \sum _{i=1}^n(f(x_i)-y_i)x_i$$

$${\theta }_2:={\theta }_2-\eta \frac{\partial L}{\partial {\theta }_2}={\theta }_2-\eta \cdot \frac{\partial L}{\partial f}\cdot \frac{\partial f}{\partial {\theta }_2}$$

$$\frac{\partial f}{\partial {\theta }_2}=\frac{\partial }{\partial {\theta }_2}f=\frac{\partial }{\partial {\theta }_2}({\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2)=x^2$$

$$\frac{\partial L}{\partial {\theta }_2}=\sum _{i=1}^n(f(x_i)-y_i)x_i^2$$

最终公式

$${\theta }_2:={\theta }_2-\eta \sum _{i=1}^n(f(x_i)-y_i)x_i^2$$

多个维度数据

有多个$x_1,x_2,\dots ,x_n$和$y$的数据，$x_1,x_2,\dots ,x_n$是输入，$y$是标记。

例如：

• 房子的楼层，尺寸，朝向和价格。

目标函数（多维）

$$\begin{gathered} f(x_1,x_2,\dots ,x_n)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n \\ ={\theta }_0\cdot 1+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n \end{gathered}$$

公式太长，使用向量简化写法

$${\theta }^T=[{\theta }_0,{\theta }_1,{\theta }_2,\dots ,{\theta }_n]$$

$$x^T=[1,x_1,x_2,\dots ,x_n]$$

$$f(x)={\theta }^T\cdot x$$

最小二乘

$$L=\frac{1}{2}[(y_1-f(x_1){)}^2+(y_2-f(x_2){)}^2+\cdots +(y_n-f(x_n){)}^2]=\frac{1}{2}\sum _{i=1}^n(y_i-f(x_i){)}^2$$

梯度下降

$${\theta }_j:={\theta }_j-\eta \frac{\partial L}{\partial {\theta }_j}$$

$$\frac{\partial L}{\partial {\theta }_j}=\frac{\partial L}{\partial f}\cdot \frac{\partial f}{\partial {\theta }_j}$$

$$\begin{gathered} \frac{\partial L}{\partial f}=\frac{\partial }{\partial f}L \\ =\frac{\partial }{\partial f}(\frac{1}{2}\sum _{i=1}^n(y_i-f(x_i){)}^2) \\ =\frac{1}{2}\sum _{i=1}^n(\frac{\partial }{\partial f}(y_i-f(x_i){)}^2) \\ =\sum _{i=1}^n(f(x_i)-y_i) \end{gathered}$$

$$\begin{gathered} \frac{\partial f}{\partial {\theta }_j}=\frac{\partial }{\partial {\theta }_j}f=\frac{\partial }{\partial {\theta }_j}(f(x))=\frac{\partial }{\partial {\theta }_j}({\theta }^T\cdot x) \\ =\frac{\partial }{\partial {\theta }_j}({\theta }_0\cdot 1+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n) \\ =x_j \end{gathered}$$

$x_j$是输入向量的第$j$个元素。

$$\frac{\partial L}{\partial {\theta }_j}=\sum _{i=1}^n(f(x_i)-y_i)x_j$$

最终公式

$${\theta }_j:={\theta }_j-\eta \sum _{i=1}^n(f(x_i)-y_i)x_{ij}$$

这里$x_i$是输入向量，$x_{ij}$是输入向量的第$j$个元素，即，第$i$个训练样本的第$j$个输入。

随机梯度下降

前面的是批量梯度下降，使用所有训练样本进行迭代计算。训练样本较多的时候，计算量较大，但参数收敛稳定。

随机梯度下降每次随机选择一个样本计算梯度并更新参数，计算效率高、适合大规模数据，且能避免陷入局部最优解。缺点是参数收敛可能会震荡。

目标函数（直线）

$$f(x)={\theta }_0+{\theta }_{1}x$$

最小二乘

随机选取一个样本，迭代一次

$$L=\frac{1}{2}(y-f(x){)}^2$$

梯度下降

$${\theta }_0:={\theta }_0-\eta \frac{\partial L}{\partial {\theta }_0}$$

$${\theta }_1:={\theta }_1-\eta \frac{\partial L}{\partial {\theta }_1}$$

$$\frac{\partial L}{\partial {\theta }_0}=\frac{\partial L}{\partial f}\cdot \frac{\partial f}{\partial {\theta }_0}$$

$$\frac{\partial L}{\partial {\theta }_1}=\frac{\partial L}{\partial f}\cdot \frac{\partial f}{\partial {\theta }_1}$$

$$\begin{gathered} \frac{\partial L}{\partial f}=\frac{\partial }{\partial f}L \\ =\frac{\partial }{\partial f}\frac{1}{2}(y-f(x){)}^2 \\ =\frac{\partial }{\partial f}\frac{1}{2}(y^2-2yf(x)+f(x{)}^2) \\ =\frac{1}{2}(-2y+2f(x)) \\ =f(x)-y \end{gathered}$$

$$\frac{\partial f}{\partial {\theta }_0}=\frac{\partial }{\partial {\theta }_0}({\theta }_0+{\theta }_{1}x)=1$$

$$\frac{\partial f}{\partial {\theta }_1}=\frac{\partial }{\partial {\theta }_1}({\theta }_0+{\theta }_{1}x)=x$$

最终公式

$${\theta }_0:={\theta }_0-\eta ((f(x)-y)\cdot 1)={\theta }_0-\eta ({\theta }_0+{\theta }_{1}x-y)$$

$${\theta }_1:={\theta }_1-\eta ((f(x)-y)\cdot x)={\theta }_1-\eta ({\theta }_0+{\theta }_{1}x-y)x$$

举例

房子尺寸和价格：

• 尺寸$x$：$[1,2,3]$（百平方米）
• 价格$y$：$[2,4,6]$（十万元）
• 初始$\theta$：$[0,0]$（线性回归中初始的$\theta$都设置为0即可）
• 学习率$\eta$：$0.01$

第1次迭代（随机选择样本 $x=1,y=2$）

$${\theta }_0:=0-0.01\ast (0+0\ast 1-2)=0.02$$

$${\theta }_1:=0-0.01\ast (0+0\ast 1-2)\ast 1=0.02$$

新 $\theta$：$[0.02,0.02]$

第2次迭代（随机选择样本 $x=3,y=6$）
$${\theta }_0:=0.02-0.01\ast (0.02+0.02\ast 3-6)=0.0792$$

$${\theta }_1:=0.02-0.01\ast (0.02+0.02\ast 3-6)\ast 3=0.1976$$

新 $\theta$：$[0.0792,0.1976]$

示例代码

import random

def get(theta_0, theta_1, x, y):
    '''
    f(x) = theta_0 + theta_1x
    '''
    eta = 0.01
    return {
        "theta_0": theta_0 - eta * (theta_0 + theta_1 * x - y), 
        "theta_1": theta_1 - eta * (theta_0 + theta_1 * x - y) * x
    }

if __name__ == "__main__":

    data = [
        {"x": 1, "y": 2},
        {"x": 2, "y": 4},
        {"x": 3, "y": 6},
    ]

    theta = {
        "theta_0": 0,
        "theta_1": 0,
    }

    for i in range(2000):
        random_number = int(random.random() * 10) % len(data)
        d = data[random_number]
        print(d)
        print(theta)
        theta = get(**theta, **d)

小批量梯度下降

结合了批量梯度下降和随机梯度下降。

• 批量梯度下降：用全部样本，更新多次参数；
• 随机梯度下降：每次选一个样本，更新一次参数；
• 小批量梯度下降：每次选几个样本，更新多次参数；选20个样本，更新50次参数，再选20个样本，再更新50次参数…