机器学习编程作业（exe3）总结

喜欢ヅ旅行 2022-05-18 02:17 169阅读 0赞

# Andrew Ng coursera上的《机器学习》ex3 #

按照课程所给的ex3的文档要求，ex3要求完成以下几个计算过程的代码编写：

<table> 
 <thead> 
  <tr> 
   <th>exerciseName</th> 
   <th>description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>IrcostFunction.m</td> 
   <td>logistics regression cost function</td> 
  </tr> 
  <tr> 
   <td>oneVSall.m</td> 
   <td>train one-vs-all muti-class classifier</td> 
  </tr> 
  <tr> 
   <td>predictOneVsAll.m</td> 
   <td>prediction function</td> 
  </tr> 
  <tr> 
   <td>predict.m</td> 
   <td>neutral network prediction function</td> 
  </tr> 
 </tbody> 
</table>

%% Machine Learning Online Class - Exercise 3 | Part 1: One-vs-all
    
    %  Instructions
    %  ------------
    %
    %  This file contains code that helps you get started on the
    %  linear exercise. You will need to complete the following functions
    %  in this exericse:
    %
    %     lrCostFunction.m (logistic regression cost function)
    %     oneVsAll.m
    %     predictOneVsAll.m
    %     predict.m
    %
    %  For this exercise, you will not need to change any code in this file,
    %  or any other files other than those mentioned above.
    %
    
    %% Initialization
    clear ; close all; clc
    
    %% Setup the parameters you will use for this part of the exercise
    input_layer_size  = 400;  % 20x20 Input Images of Digits
    num_labels = 10;          % 10 labels, from 1 to 10
                              % (note that we have mapped "0" to label 10)
    
    %% =========== Part 1: Loading and Visualizing Data =============
    %  We start the exercise by first loading and visualizing the dataset.
    %  You will be working with a dataset that contains handwritten digits.
    %
    
    % Load Training Data
    fprintf('Loading and Visualizing Data ...\n')
    
    load('ex3data1.mat'); % training data stored in arrays X, y
    m = size(X, 1);
    
    % Randomly select 100 data points to display
    rand_indices = randperm(m);
    sel = X(rand_indices(1:100), :);
    
    displayData(sel);
    
    fprintf('Program paused. Press enter to continue.\n');
    pause;

![70][]

回顾了正则化逻辑回归模型：

%% ============ Part 2a: Vectorize Logistic Regression ============
    %  In this part of the exercise, you will reuse your logistic regression
    %  code from the last exercise. You task here is to make sure that your
    %  regularized logistic regression implementation is vectorized. After
    %  that, you will implement one-vs-all classification for the handwritten
    %  digit dataset.
    %
    
    % Test case for lrCostFunction
    fprintf('\nTesting lrCostFunction() with regularization');
    
    theta_t = [-2; -1; 1; 2];
    X_t = [ones(5,1) reshape(1:15,5,3)/10];
    y_t = ([1;0;1;0;1] >= 0.5);
    lambda_t = 3;
    [J grad] = lrCostFunction(theta_t, X_t, y_t, lambda_t);
    
    fprintf('\nCost: %f\n', J);
    fprintf('Expected cost: 2.534819\n');
    fprintf('Gradients:\n');
    fprintf(' %f \n', grad);
    fprintf('Expected gradients:\n');
    fprintf(' 0.146561\n -0.548558\n 0.724722\n 1.398003\n');
    
    fprintf('Program paused. Press enter to continue.\n');
    pause;

1. Vectorizing Logistic Regression

(1) Vectorizing the cost function

(2) Vectorizing the gradient

(3) Vectorizing the regularized cost function

(4) Vectorizing the regularized gradient

All above 4 formulas can be found in the previous blog: click [here][].

lrCostFunction.m

function [J, grad] = lrCostFunction(theta, X, y, lambda)
    %LRCOSTFUNCTION Compute cost and gradient for logistic regression with 
    %regularization
    %   J = LRCOSTFUNCTION(theta, X, y, lambda) computes the cost of using
    %   theta as the parameter for regularized logistic regression and the
    %   gradient of the cost w.r.t. to the parameters. 
    
    % Initialize some useful values
    m = length(y); % number of training examples
    
    % You need to return the following variables correctly 
    J = 0;
    grad = zeros(size(theta));
    
    % ====================== YOUR CODE HERE ======================
    % Instructions: Compute the cost of a particular choice of theta.
    %               You should set J to the cost.
    %               Compute the partial derivatives and set grad to the partial
    %               derivatives of the cost w.r.t. each parameter in theta
    %
    % Hint: The computation of the cost function and gradients can be
    %       efficiently vectorized. For example, consider the computation
    %
    %           sigmoid(X * theta)
    %
    %       Each row of the resulting matrix will contain the value of the
    %       prediction for that example. You can make use of this to vectorize
    %       the cost function and gradient computations. 
    %
    % Hint: When computing the gradient of the regularized cost function, 
    %       there're many possible vectorized solutions, but one solution
    %       looks like:
    %           grad = (unregularized gradient for logistic regression)
    %           temp = theta; 
    %           temp(1) = 0;   % because we don't add anything for j = 0  
    %           grad = grad + YOUR_CODE_HERE (using the temp variable)
    % 正则化的逻辑回归模型
    % log(hθ(x(i))
    hx = sigmoid(X * theta);
    % 正则化参数
    reg = lambda/(2*m)*sum(theta(2:size(theta),:).^2);
    % 正则化逻辑回归函数
    J = -1/m*(y'*log(hx)+(1-y)'*log(1-hx)) +reg;
    theta(1) = 0;
    % 正则化逻辑回归的梯度(求导)
    grad = 1/m*X'*(hx-y)+lambda/m*theta;
    % =============================================================
    
    grad = grad(:);
    
    end

2. One-vs-all Classification (Training)

Return all the classifier parameters in a matrix Θ (a K x N+1 matrix, K is the num\_labels and N is the num\_features ), where each row of Θ corresponds to the learned logistic regression parameters for one class. You can do this with a 'for'-loop from 1 to K, training each classifier independently.

oneVsAll.m

function [all_theta] = oneVsAll(X, y, num_labels, lambda)
    %ONEVSALL trains multiple logistic regression classifiers and returns all
    %the classifiers in a matrix all_theta, where the i-th row of all_theta 
    %corresponds to the classifier for label i
    %   [all_theta] = ONEVSALL(X, y, num_labels, lambda) trains num_labels
    %   logistic regression classifiers and returns each of these classifiers
    %   in a matrix all_theta, where the i-th row of all_theta corresponds 
    %   to the classifier for label i
    
    % Some useful variables
    m = size(X, 1);
    n = size(X, 2);
    
    % You need to return the following variables correctly 
    all_theta = zeros(num_labels, n + 1);
    
    % Add ones to the X data matrix
    X = [ones(m, 1) X];
    
    % ====================== YOUR CODE HERE ======================
    % Instructions: You should complete the following code to train num_labels
    %               logistic regression classifiers with regularization
    %               parameter lambda. 
    %
    % Hint: theta(:) will return a column vector.
    %
    % Hint: You can use y == c to obtain a vector of 1's and 0's that tell you
    %       whether the ground truth is true/false for this class.
    %
    % Note: For this assignment, we recommend using fmincg to optimize the cost
    %       function. It is okay to use a for-loop (for c = 1:num_labels) to
    %       loop over the different classes.
    %
    %       fmincg works similarly to fminunc, but is more efficient when we
    %       are dealing with large number of parameters.
    %
    % Example Code for fmincg:
    %
    %     % Set Initial theta
    %     initial_theta = zeros(n + 1, 1);
    %     
    %     % Set options for fminunc
    %     options = optimset('GradObj', 'on', 'MaxIter', 50);
    % 
    %     % Run fmincg to obtain the optimal theta
    %     % This function will return theta and the cost 
    %     [theta] = ...
    %         fmincg (@(t)(lrCostFunction(t, X, (y == c), lambda)), ...
    %                 initial_theta, options);
    %
    for c=1:num_labels,
      initial_theta = all_theta(c,:)';
      options = optimset('GradObj','on','MaxIter',50);
      theta = fmincg(@(t)(lrCostFunction(t,X,(y==c),lambda)),initial_theta,options);
      all_theta(c,:) = theta';
     end;
    % =========================================================================
    end

主程序：

%% ============ Part 2b: One-vs-All Training ============
    fprintf('\nTraining One-vs-All Logistic Regression...\n')
    
    lambda = 0.1;
    [all_theta] = oneVsAll(X, y, num_labels, lambda);
    
    fprintf('Program paused. Press enter to continue.\n');
    pause;

3. One-vs-all Classification (Prediction)

predictOneVsAll.m

X是一个5000\*400的矩阵，表示的是5000个训练集，每个训练集有20\*20的表示数字的灰度图。all\_theta是一个10\*401的矩阵，每一行表示每个类别的theta值，theta\*X’返回一个10\*5000的矩阵，表示5000个值中，分别为十个类别的概率。矩阵的每列的最大值所在行，表示为该行所代表数字的可能性最大。比如第一列的第三个数所求的数字最大，则第一列的求出的十个数字中，为3的概率最大。

function p = predictOneVsAll(all_theta, X)
    %PREDICT Predict the label for a trained one-vs-all classifier. The labels 
    %are in the range 1..K, where K = size(all_theta, 1). 
    %  p = PREDICTONEVSALL(all_theta, X) will return a vector of predictions
    %  for each example in the matrix X. Note that X contains the examples in
    %  rows. all_theta is a matrix where the i-th row is a trained logistic
    %  regression theta vector for the i-th class. You should set p to a vector
    %  of values from 1..K (e.g., p = [1; 3; 1; 2] predicts classes 1, 3, 1, 2
    %  for 4 examples) 
    
    m = size(X, 1);
    num_labels = size(all_theta, 1);
    
    % You need to return the following variables correctly 
    p = zeros(size(X, 1), 1);
    
    % Add ones to the X data matrix
    X = [ones(m, 1) X];
    
    % ====================== YOUR CODE HERE ======================
    % Instructions: Complete the following code to make predictions using
    %               your learned logistic regression parameters (one-vs-all).
    %               You should set p to a vector of predictions (from 1 to
    %               num_labels).
    %
    % Hint: This code can be done all vectorized using the max function.
    %       In particular, the max function can also return the index of the 
    %       max element, for more information see 'help max'. If your examples 
    %       are in rows, then, you can use max(A, [], 2) to obtain the max 
    %       for each row.
    %   X是一个5000*400的矩阵，表示的是5000个训练集，每个训练集有20*20的表示数字的灰度图。
    %   all_theta是一个10*401的矩阵，每一行表示每个类别的theta值，
    %   theta*X’返回一个10*5000的矩阵，表示5000个值中，分别为十个类别的概率。
    %   矩阵的每列的最大值所在行，表示为该行所代表数字的可能性最大。
    %   比如第一列的第三个数所求的数字最大，则第一列的求出的十个数字中，为3的概率最大。    
    [X,p] = max(sigmoid(all_theta * X'));  
    p = p';
    % =========================================================================
    end

%% ================ Part 3: Predict for One-Vs-All ================
    % 得到所有训练集 预测的结果
    pred = predictOneVsAll(all_theta, X);
    % 将所有训练集的预测结果 与实际分类结果 相比 看准确率多少
    fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100);

前半部分就是对上一章的复习 逻辑回归模型

2.Neural Networks

逻辑回归不能完成更加复杂的hypothesis，因为它只是一个线性的分类器。神经网络可以用于更加复杂的non-linear hypothesis.

input\_layer\_size = 400;

hidden\_layer\_size = 25;

完成前向反馈公式，并像上面一样，对最后求出的h进行概率最大值处理，最大值所在行数表示为行数对应的数字概率最大。

%% =========== Part 1: Loading and Visualizing Data =============
    %  We start the exercise by first loading and visualizing the dataset. 
    %  You will be working with a dataset that contains handwritten digits.
    %
    
    % Load Training Data
    fprintf('Loading and Visualizing Data ...\n')
    
    load('ex3data1.mat');
    m = size(X, 1);
    
    % Randomly select 100 data points to display
    sel = randperm(size(X, 1));
    sel = sel(1:100);
    
    displayData(X(sel, :));
    
    fprintf('Program paused. Press enter to continue.\n');
    pause;

![70 1][]

***predict.m***

function p = predict(Theta1, Theta2, X)
    %PREDICT Predict the label of an input given a trained neural network
    %   p = PREDICT(Theta1, Theta2, X) outputs the predicted label of X given the
    %   trained weights of a neural network (Theta1, Theta2)
    
    % Useful values
    m = size(X, 1);
    num_labels = size(Theta2, 1);
    
    % You need to return the following variables correctly 
    p = zeros(size(X, 1), 1);
    
    % ====================== YOUR CODE HERE ======================
    % Instructions: Complete the following code to make predictions using
    %               your learned neural network. You should set p to a 
    %               vector containing labels between 1 to num_labels.
    %
    % Hint: The max function might come in useful. In particular, the max
    %       function can also return the index of the max element, for more
    %       information see 'help max'. If your examples are in rows, then, you
    %       can use max(A, [], 2) to obtain the max for each row.
    %
    
    a1 = [ones(size(X,1),1) X];
    z2 = a1*Theta1';
    a2 = sigmoid(z2);
    a2 = [ones(size(a2,1),1) a2];
    z3 = Theta2*a2';
    [a3,p] = max(sigmoid(z3));
    p = p';% 小心这个不写 运行完此函数 会出现维度不一致错误
    % =========================================================================
    end

算法的实现思路：

![70 2][]

%% ================ Part 2: Loading Pameters ================
    % In this part of the exercise, we load some pre-initialized 
    % neural network parameters.
    
    fprintf('\nLoading Saved Neural Network Parameters ...\n')
    
    % Load the weights into variables Theta1 and Theta2
    load('ex3weights.mat');
    
    %% ================= Part 3: Implement Predict =================
    %  After training the neural network, we would like to use it to predict
    %  the labels. You will now implement the "predict" function to use the
    %  neural network to predict the labels of the training set. This lets
    %  you compute the training set accuracy.
    
    pred = predict(Theta1, Theta2, X);
    
    fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100);
    
    fprintf('Program paused. Press enter to continue.\n');
    pause;
    
    %%  To give you an idea of the network's output, you can also run
    %  through the examples one at the a time to see what it is predicting.
    
    %  Randomly permute examples
    % p = randperm(n)   返回一行包含从1到n的整数
    rp = randperm(m);
    
    for i = 1:m
        % Display 
        fprintf('\nDisplaying Example Image\n');
        displayData(X(rp(i), :));
    
        pred = predict(Theta1, Theta2, X(rp(i),:));
        fprintf('\nNeural Network Prediction: %d (digit %d)\n', pred, mod(pred, 10));
        
        % Pause with quit option
        s = input('Paused - press enter to continue, q to exit:','s');
        if s == 'q'
          break
        end
    end

![70 3][]

[70]: /images/20220518/eb9fa37d1f9f466f9381e80fcf36c468.png
[here]: http://www.cnblogs.com/CheeseZH/p/4600837.html
[70 1]: /images/20220518/10ee3046daab4855a51056b8290e98ac.png
[70 2]: /images/20220518/81c9969a6177421e95651e223d31d9c2.png
[70 3]: /images/20220518/f54e7fe2a053416a895fa4ee7dc12354.png