机器学习实验（二）：梯度下降和牛顿法解逻辑回归

骑猪看日落 2022-04-14 03:58 231阅读 0赞

一：梯度下降法

1.  加载实验数据画出画出散点图，区分出能进入大学的学生和不能进入大学的学生。
2.  回顾逻辑回归，其假设函数为

*h**θ**x**=g**θ**T**x**=**1**1+**e**-**θ**T**x=p(y=1|x;θ)*![20181124081430499.png][]

在使用前先声明此函数，代码如下：

g=inline(‘1.0./(1.0+exp(-z)))

1.  当我们给定一个训练数据集时，*x**(i)i=1,2,…,m*![20181124081430496.png][] 我们定义似然函数为：

Jθ=*i=1**m**(**h**θ**(**x**(i)**))**y(i)**(1-**h**θ**(**x**(i)**))1-y(i)*![20181124081430495.png][]

为了简化计算，我们转变成下列的似然函数：

Lθ=*1**m**i=1**m**y**(i)*log*h**θ**x**i**+(1-**y**i**)*log⁡*(**h**θ**(**x**(i)))*![20181124081430505.png][]

并且L的梯度可以被定义为：

∇*θ**L=**1**m**x**i=1**m**(**h**θ**x**i**-**y**(i)**)**x(i)*![20181124081430392.png][]

1.  每次更新θ：![20181124081430605.png][]

θ←θ-α∇*θL*![20181124081430606.png][]

直到似然函数的差值变得很小，即：

*L**+*θ-Lθ<ε![20181124081430701.png][]

二：牛顿法

（1）牛顿法求解逻辑回归：θ![20181124081430696.png][] 的更新变为

θ←θ-*H**-1*∇*θL*![20181124081430702.png][]

逻辑回归中，海塞矩阵的求法为：

H=*1**m**i=1**m**h**θ**(**x**i**)(1-**h**θ**(**x**i**))**x**i**T**(**x**(i))*![20181124081430710.png][]

（2）其余的与上面的方法一样。

三。实验结果：

梯度下降：学习率为0.0024，成本函数的变化情况：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70][]

求出的决策边界：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70 1][]

牛顿法：成本函数的变化情况（跟之前相比加负号）。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70 2][]

牛顿法的划分结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70 3][]

四、相关代码：（MATLAB）

%梯度下降法：
    clear,clc;
    x=load('ex2x.dat');
    y=load('ex2y.dat');
    m=length(y);
    one=ones(m,1);
    x=[one,x];
    %x(:,3)=(x(:,3)-mean(x(:,3)))./std(x(:,3));
    %x(:,2)=(x(:,2)-mean(x(:,2)))./std(x(:,2));
    pos = find (y == 1); neg = find (y == 0);
    plot (x(pos , 2) , x(pos ,3) , '+' );
    hold on ;
    plot (x(neg , 2) , x(neg , 3) ,'ro ' );
    g = inline ( '1.0 ./ (1.0 + exp(-z ))'); 
    c0=0;c1=0;c2=0;
    c=[c0,c1,c2];
    step=0.0024 ; %ÉèÖÃ²½³¤,³õÊ¼Öµ,µü´ú¼ÇÂ¼Êý  
    L_val_current=0;
    L_val_previous=100;
      min=9999999999;
      j=1; temp=100;
     while abs(L_val_current-L_val_previous)>0.000001
        %for i=1:m
         % r1=r1+(g(c*x(i,:)'))-y(i,:);
          %r2=r2+(g(c*x(i,:)')-y(i,:))*x(i,2);
          %r3=r3+(g(c*x(i,:)')-y(i,:))*x(i,3);
        %end
        r1=(g(x*c')-y)'*x(:,1);
        r2=(g(x*c')-y)'*x(:,2);
        r3=(g(x*c')-y)'*x(:,3);
        c0=c0-step*r1/m;
        c1=c1-step*r2/m;
        c2=c2-step*r3/m;
        c=[c0,c1,c2];
        cc(j,:)=c;
        %for k=1:m
         %  L_val_current=L_val_current+(y(k,:)*log(g(x(k,:)*c'))+(1-y(k,:))*log(1-g(x(k,:)*c')))/m;
        %end
        L_val_current=(-(log(g(x*c')))'*y-(log(one-g(x*c')))'*(ones(m,1)-y))/m;
            L_val_previous=temp;
            f(j,:)=L_val_current;
            temp=L_val_current;
      %  j_values(j,:)=fx;
       % if(L_val_current<min)
        %    min=L_val_current;
         %   cmin=[c0,c1,c2];
          %  num=j;
      
        j=j+1; 
     end 
    %a=[-2.5:0.1:2.5];
    a=[10:1:70];
    plot(a,-c1/c2*a-c0/c2,'r-')
    xlabel('Exam 1 score');
    ylabel('Exam 2 score');
    legend('admitted','Not admitted','decision boundary');
    figure ;
    plot(1:j-1,f(:,1),'r-');
    
    
    %牛顿法：
    clear,clc;
    x=load('ex2x.dat');
    y=load('ex2y.dat');
    m=length(y);
    x=[ones(m,1),x];
    %x(:,3)=(x(:,3)-mean(x(:,3)))./std(x(:,3));
    %x(:,2)=(x(:,2)-mean(x(:,2)))./std(x(:,2));
    pos = find (y == 1); neg = find (y == 0);
    plot (x(pos , 2) , x(pos ,3) , '+' );
    hold on ;
    plot (x(neg , 2) , x(neg , 3) ,'ro ' );
     
    g = inline ( '1.0 ./ (1.0 + exp(-z ))'); 
    c=[0,0,0];
    step=0.009; %ÉèÖÃ²½³¤,³õÊ¼Öµ,µü´ú¼ÇÂ¼Êý  
    L_val_current=0;
    L_val_previous=100;
      min=9999999999;
      j=1; temp=100;
     while abs(L_val_current-L_val_previous)>0.000001
         L_val_current=0;
        r1=0;r2=0;r3=0;
        H=zeros(3,3);
        for i=1:m
          r1=r1+((g(c*x(i,:)'))-y(i,:));
          r2=r2+((g(c*x(i,:)')-y(i,:))*x(i,2));
          r3=r3+((g(c*x(i,:)')-y(i,:))*x(i,3));
          H=H+(g(c*x(i,:)')*(1-g(c*x(i,:)'))*x(i,:)'*x(i,:))/m;
        end
        c=c-[r1,r2,r3]/H/m;
        cc(j,:)=c;
        for k=1:m
            L_val_current=L_val_current+(y(k,:)*log(g(x(k,:)*c'))+(1-y(k,:))*log(1-g(x(k,:)*c')))/m;
        end
            L_val_previous=temp;
            f(j,:)=L_val_current;
            temp=L_val_current;
            
      %  j_values(j,:)=fx;
        if(L_val_current<min)
            min=L_val_current;
            cmin=[c(1),c(2),c(3)];
            num=j;
        end
        j=j+1; 
     end 
    a=[10:0.1:70];
    plot(a,-c(:,2)/c(:,3)*a-c(:,1)/c(:,3),'r-')
    xlabel('Exam 1 score');
    ylabel('Exam 2 score');
    legend('admitted','Not admitted','decision boundary');
    figure ;
    plot(1:j-1,f(:,1),'r-');

[20181124081430499.png]: /images/20220414/ef5f77fabfcb40faae3f05e18636dea7.png
[20181124081430496.png]: /images/20220414/518d4dd34030434280fd147e462a588a.png
[20181124081430495.png]: /images/20220414/49adf52b310543d9a1fb5110c79dee6a.png
[20181124081430505.png]: /images/20220414/4af1f1a1df0642eaba79a4d414d05e73.png
[20181124081430392.png]: /images/20220414/8daa11a6e90c496a8efd5b3efc39ecca.png
[20181124081430605.png]: /images/20220414/1e559ec97db641e3886ceabb5b59bd19.png
[20181124081430606.png]: /images/20220414/c972b4fcb44d4a1b9269e7ce7895607c.png
[20181124081430701.png]: /images/20220414/ac1be1760d774f9290d69d42e55e4d7c.png
[20181124081430696.png]: /images/20220414/dfbeb8551a3549aaac515128895c26df.png
[20181124081430702.png]: /images/20220414/84bab84e8785484facdbea9e13984d70.png
[20181124081430710.png]: /images/20220414/f36d6b39fc5344d1ae443cb030e85899.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70]: /images/20220414/8d340b4cad3446ae8839c25182413ae8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70 1]: /images/20220414/7ad67a7078a64fda866d840cad9d80df.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70 2]: /images/20220414/d233909e12ec4e95bfbe465ae3c98572.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjQ3NTQ0_size_16_color_FFFFFF_t_70 3]: /images/20220414/7d40d228625d4010b68aa621287b9428.png