python实现线性回归之梯度下降法，梯度下降详解

ゝ一纸荒年。 2022-01-23 15:45 458阅读 0赞

线性回归的有关概念已在笔者[相关文章][Link 1]中进行介绍。本篇内容将介绍梯度下降(BGD)相关内容。

### 1.梯度下降 ###

梯度下降常用于机器学习中求解符合最小损失函数的模型的参数值，梯度下降也是BP神经网络的核心，本文将介绍批量梯度下降法(BGD)。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM4MDc1NDI1_size_16_color_FFFFFF_t_70][]

如上图所示，梯度下降的过程便是沿梯度方向，按照一定的步伐求解极小(大)值。这里举一个简单的例子，假如你在一座山上，你怎样才能最安全最快速地下山，这里有两个条件，一是安全下山，二是快速下山。答案便是沿着较为陡峭(梯度)的地方，且容易落脚(步伐大小，即学习率)的地方一步一步地下山。

梯度，也就是“山”的斜度，在上图中，便是每个点处的斜率，即导数。下面给出梯度下降的数学公式：

![ΘΘ][gif.latex_25u0398_25u0398]![\\theta \_\{i+1\}=\\theta\_\{i\}+\\alpha \\frac\{\\partial J(\\theta )\}\{\\partial \\theta \}][theta _i_1_theta_i_alpha _frac_partial J_theta _partial _theta]

上式中，![\\theta \_\{i+1\}][theta _i_1]便是下一步，![\\theta \_\{i\}][theta _i]是当前所处的位置，![\\alpha][alpha]是步伐大小（学习率），![\\frac\{\\partial J(\\theta )\}\{\\partial \\theta \}][frac_partial J_theta _partial _theta]为梯度，也就是![J(\\theta )][J_theta]对![\\theta][theta]的偏导数。

另外，上式中的![J(\\theta )][J_theta]为损失函数，也就是描述结果准确程度的函数，在这里，我们定义损失函数为：

![J(\\theta )=\\frac\{1\}\{2m\}\\sum\_\{i=0\}^\{m\}(h\_\{\\theta \}(x^\{i\})-y^\{i\})^2][J_theta _frac_1_2m_sum_i_0_m_h_theta _x_i_-y_i_2]

理论上，当损失函数取最小值0时，便为最优的结果。对于上述![J(\\theta )][J_theta]，其实还有其他的损失函数也能很好地发挥作用，但是平方误差损失函数是解决回归问题最常用的手段。

在上式中，![h\_\{\\theta \}(x^\{i\})][h_theta _x_i]表示当前![\\theta][theta]与矩阵![X][]的点乘。如：

![\\theta =\\begin\{bmatrix\} 1 & 2\\\\ 6 & 9\\\\ 5 &7 \\end\{bmatrix\}][theta _begin_bmatrix_ 1 _ 2_ 6 _ 9_ 5 _7 _end_bmatrix]，![X=\\begin\{bmatrix\} 2 & 3& 8\\\\ 5& 2& 7 \\end\{bmatrix\}][X_begin_bmatrix_ 2 _ 3_ 8_ 5_ 2_ 7 _end_bmatrix]

则                                                                 ![h\_\{\\theta \}(x^\{i\})=\\begin\{bmatrix\}12 & 7& 22\\\\ 57&36 & 111\\\\ 45& 29& 89\\end\{bmatrix\}][h_theta _x_i_begin_bmatrix_12 _ 7_ 22_ 57_36 _ 111_ 45_ 29_ 89_end_bmatrix]

在上式中，![y^\{i\}][y_i]是我们的实际数据。

对![J(\\theta )][J_theta]求偏导，得：

![\\frac\{\\partial J(\\theta )\}\{\\partial \\theta \}=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}(h\_\{\\theta \}(x^\{i\})-y^\{i\})x^\{i\}][frac_partial J_theta _partial _theta _frac_1_m_sum_i_1_m_h_theta _x_i_-y_i_x_i]

即最终表达式为：

![\\theta \_\{i+1\}=\\theta\_\{i\}+\\alpha(\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}(h\_\{\\theta \}(x^\{i\})-y^\{i\})x^\{i\})][theta _i_1_theta_i_alpha_frac_1_m_sum_i_1_m_h_theta _x_i_-y_i_x_i]

按照上式，便可求出最终参数![\\theta][theta]。

关于学习率![\\alpha][alpha]的说明：如果学习率太小，会导致学习时间变长，消耗资源变大。如果学习率太大，可能会越过最低(高)点，误差变大，甚至导致函数无法收敛。

### 2.python实现 ###

**一元参数测试：**

import numpy as np
    
    X = mat([1, 2, 3]).reshape(3, 1)  # x为1,2,3
    Y = mat([5, 10, 15]).reshape(3, 1)  # y为5,10,15
    theta = 1 #初始为1
    alpha = 0.1 # 学习率为0.1
    for i in range(100):
        theta = theta + np.sum(alpha * (Y - dot(X, theta)) * X.reshape(1, 3))/3 # 公式实现
    print(theta)

由代码的初始数据可知，方程应为![y=5x][y_5x]，下面我们测试theta是否为5：

![201906041438098.png][]

测试结果正确。

**实战：**

同样，这里准备了最小二乘法的测试数据，待下面测试结束后，读者可检查结果是否与最小二乘法的结果一致。

数据示例截图如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM4MDc1NDI1_size_16_color_FFFFFF_t_70 1][]

上述数据为某一商品的销售量与售价、服务投资和其它投资的对应关系，即我们要求出：

![X=\\begin\{bmatrix\} 3.5 & 1.4 & 0.2\\\\ 3.0 & 1.4& 0.2\\\\ 3.2 & 1.3& 0.2\\\\ ...& ...& ... \\end\{bmatrix\}][X_begin_bmatrix_ 3.5 _ 1.4 _ 0.2_ 3.0 _ 1.4_ 0.2_ 3.2 _ 1.3_ 0.2_ ..._ ..._ ... _end_bmatrix],![Y=\\begin\{bmatrix\} 5.1\\\\ 4.9\\\\ 4.7\\\\ ... \\end\{bmatrix\}][Y_begin_bmatrix_ 5.1_ 4.9_ 4.7_ ... _end_bmatrix],![\\theta =\\begin\{bmatrix\} \\theta \_\{0\}\\\\ \\theta\_\{1\}\\\\ \\theta\_\{2\}\\\\ ... \\end\{bmatrix\}][theta _begin_bmatrix_ _theta _0_ _theta_1_ _theta_2_ ... _end_bmatrix]

![Y=X\\theta][Y_X_theta]中，矩阵![\\theta][theta]。

代码如下：

import numpy as np
    import pandas as pd
    from numpy import dot
    
    
    dataset = pd.read_csv('C:\\Users\\57105\\Desktop\\data.csv')  # 读入数据
    X = dataset.iloc[:, 2: 5]  # x为所有行，2到4列
    Y = dataset.iloc[:, 1]  # y为所有行，第1列
    theta = np.array([1.0, 1.0, 1.0]).reshape(3, 1)  # theta初始值
    alpha = 0.1  # 学习率
    temp = theta
    X0 = X.iloc[:, 0].values.reshape(60, 1)
    X1 = X.iloc[:, 1].values.reshape(60, 1)
    X2 = X.iloc[:, 2].values.reshape(60, 1)
    Y = Y.values.reshape(60, 1)
    for i in range(1000):  # 这里特别注意，在完成一次循环后，整体更新theta
        temp[0] = theta[0] + alpha * np.sum((Y - dot(X, theta)) * X0) / 60.0
        temp[1] = theta[1] + alpha * np.sum((Y - dot(X, theta)) * X1) / 60.0
        temp[2] = theta[2] + alpha * np.sum((Y - dot(X, theta)) * X2) / 60.0
        theta = temp
    print(theta)

测试结果：

![20190604144914306.png][]

即![\\theta =\\begin\{bmatrix\} 1.157\\\\ 0.780\\\\ -0.586 \\end\{bmatrix\}][theta _begin_bmatrix_ 1.157_ 0.780_ -0.586 _end_bmatrix]，结果与最小二乘法几乎一致，如想再精确，可调整学习率。即![y=1.157x\_\{1\}+0.78x\_\{2\}-0.586x\_\{3\}][y_1.157x_1_0.78x_2_-0.586x_3],该回归方程反应了商品销售量与售价、服务投资和其它投资的关系。

上述便为梯度下降的相关内容。

[Link 1]: https://blog.csdn.net/m0_38075425/article/details/90738415
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM4MDc1NDI1_size_16_color_FFFFFF_t_70]: /images/20220123/b03672abea46447bbe517ac7e5dbe1e6.png
[gif.latex_25u0398_25u0398]: https://private.codecogs.com/gif.latex?%25u0398%25u0398
[theta _i_1_theta_i_alpha _frac_partial J_theta _partial _theta]: https://private.codecogs.com/gif.latex?%5Ctheta%20_%7Bi&plus;1%7D%3D%5Ctheta_%7Bi%7D&plus;%5Calpha%20%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%7D%7B%5Cpartial%20%5Ctheta%20%7D
[theta _i_1]: https://private.codecogs.com/gif.latex?%5Ctheta%20_%7Bi&plus;1%7D
[theta _i]: https://private.codecogs.com/gif.latex?%5Ctheta%20_%7Bi%7D
[alpha]: https://private.codecogs.com/gif.latex?%5Calpha
[frac_partial J_theta _partial _theta]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%7D%7B%5Cpartial%20%5Ctheta%20%7D
[J_theta]: https://private.codecogs.com/gif.latex?J%28%5Ctheta%20%29
[theta]: https://private.codecogs.com/gif.latex?%5Ctheta
[J_theta _frac_1_2m_sum_i_0_m_h_theta _x_i_-y_i_2]: https://private.codecogs.com/gif.latex?J%28%5Ctheta%20%29%3D%5Cfrac%7B1%7D%7B2m%7D%5Csum_%7Bi%3D0%7D%5E%7Bm%7D%28h_%7B%5Ctheta%20%7D%28x%5E%7Bi%7D%29-y%5E%7Bi%7D%29%5E2
[h_theta _x_i]: https://private.codecogs.com/gif.latex?h_%7B%5Ctheta%20%7D%28x%5E%7Bi%7D%29
[X]: https://private.codecogs.com/gif.latex?X
[theta _begin_bmatrix_ 1 _ 2_ 6 _ 9_ 5 _7 _end_bmatrix]: https://private.codecogs.com/gif.latex?%5Ctheta%20%3D%5Cbegin%7Bbmatrix%7D%201%20%26%202%5C%5C%206%20%26%209%5C%5C%205%20%267%20%5Cend%7Bbmatrix%7D
[X_begin_bmatrix_ 2 _ 3_ 8_ 5_ 2_ 7 _end_bmatrix]: https://private.codecogs.com/gif.latex?X%3D%5Cbegin%7Bbmatrix%7D%202%20%26%203%26%208%5C%5C%205%26%202%26%207%20%5Cend%7Bbmatrix%7D
[h_theta _x_i_begin_bmatrix_12 _ 7_ 22_ 57_36 _ 111_ 45_ 29_ 89_end_bmatrix]: https://private.codecogs.com/gif.latex?h_%7B%5Ctheta%20%7D%28x%5E%7Bi%7D%29%3D%5Cbegin%7Bbmatrix%7D12%20%26%207%26%2022%5C%5C%2057%2636%20%26%20111%5C%5C%2045%26%2029%26%2089%5Cend%7Bbmatrix%7D
[y_i]: https://private.codecogs.com/gif.latex?y%5E%7Bi%7D
[frac_partial J_theta _partial _theta _frac_1_m_sum_i_1_m_h_theta _x_i_-y_i_x_i]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%7D%7B%5Cpartial%20%5Ctheta%20%7D%3D%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi%3D1%7D%5E%7Bm%7D%28h_%7B%5Ctheta%20%7D%28x%5E%7Bi%7D%29-y%5E%7Bi%7D%29x%5E%7Bi%7D
[theta _i_1_theta_i_alpha_frac_1_m_sum_i_1_m_h_theta _x_i_-y_i_x_i]: https://private.codecogs.com/gif.latex?%5Ctheta%20_%7Bi&plus;1%7D%3D%5Ctheta_%7Bi%7D&plus;%5Calpha%28%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi%3D1%7D%5E%7Bm%7D%28h_%7B%5Ctheta%20%7D%28x%5E%7Bi%7D%29-y%5E%7Bi%7D%29x%5E%7Bi%7D%29
[y_5x]: https://private.codecogs.com/gif.latex?y%3D5x
[201906041438098.png]: /images/20220123/23ae454ef7694b81a762cf888fed474e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM4MDc1NDI1_size_16_color_FFFFFF_t_70 1]: /images/20220123/5876ad46146647559174ad21512635ae.png
[X_begin_bmatrix_ 3.5 _ 1.4 _ 0.2_ 3.0 _ 1.4_ 0.2_ 3.2 _ 1.3_ 0.2_ ..._ ..._ ... _end_bmatrix]: https://private.codecogs.com/gif.latex?X%3D%5Cbegin%7Bbmatrix%7D%203.5%20%26%201.4%20%26%200.2%5C%5C%203.0%20%26%201.4%26%200.2%5C%5C%203.2%20%26%201.3%26%200.2%5C%5C%20...%26%20...%26%20...%20%5Cend%7Bbmatrix%7D
[Y_begin_bmatrix_ 5.1_ 4.9_ 4.7_ ... _end_bmatrix]: https://private.codecogs.com/gif.latex?Y%3D%5Cbegin%7Bbmatrix%7D%205.1%5C%5C%204.9%5C%5C%204.7%5C%5C%20...%20%5Cend%7Bbmatrix%7D
[theta _begin_bmatrix_ _theta _0_ _theta_1_ _theta_2_ ... _end_bmatrix]: https://private.codecogs.com/gif.latex?%5Ctheta%20%3D%5Cbegin%7Bbmatrix%7D%20%5Ctheta%20_%7B0%7D%5C%5C%20%5Ctheta_%7B1%7D%5C%5C%20%5Ctheta_%7B2%7D%5C%5C%20...%20%5Cend%7Bbmatrix%7D
[Y_X_theta]: https://private.codecogs.com/gif.latex?Y%3DX%5Ctheta
[20190604144914306.png]: /images/20220123/20efaa3ca6ed47aba32d06d2d22fe14b.png
[theta _begin_bmatrix_ 1.157_ 0.780_ -0.586 _end_bmatrix]: https://private.codecogs.com/gif.latex?%5Ctheta%20%3D%5Cbegin%7Bbmatrix%7D%201.157%5C%5C%200.780%5C%5C%20-0.586%20%5Cend%7Bbmatrix%7D
[y_1.157x_1_0.78x_2_-0.586x_3]: https://private.codecogs.com/gif.latex?y%3D1.157x_%7B1%7D&plus;0.78x_%7B2%7D-0.586x_%7B3%7D