深度学习笔记(3.numpy实现深层神经网络二分类)

小咪咪 2023-08-17 15:48 129阅读 0赞

### 前言 ###

看完课程，根据记忆用numpy实现一遍深层神经网络，用上一篇实验数据验证实现算法是否正确。自己做的逻辑比较简单，实现了init\_parameters、forward\_propagation、back\_propagation、update\_parameters几个主要方法。hidden layer可选tanh和relu激活函数，输出层LR二分类。相比于Andrew Ng给出的jupyter代码实现课件，感觉自己实现的代码比较粗糙，只是实现了大块的功能，没有细分功能模块，代码复用性不高。

### 网络结构（课件截图） ###

激活函数σ=tanh(z)，tanh求导公式σ’(z)=1-(tanh)2

激活函数σ=relu(z)，relu求导公式分段σ’(z)=1, z>0  
σ’(z)=0, z<=0

output layer激活函数σ=sigmoid(z)，求导σ’(z) = σ(1-σ)

input layer层（l=0，x1和x2，节点数2）  
hidden layer（l=1，2，3，4，节点数分别为3，5，4，2）  
output layer（l=5，节点数1）  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70]

### 主要模块实现 ###

（1）正向传播  
L：网络层数（输入，隐藏，输出）  
l=0：input layer，输入数据  
l=1~L-2：hidden layer，参数部分W1…W(L-2)，b1…b(L-2)  
l=L-1：output layer，参数W(L-1), b(L-1)  
说明：hidden layer和output layer的激活函数不同，计算forward propagation时候分成2部分处理，如代码for内部是hidden layer，外部淡出计算output layer。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 1]  
（2）反向传播  
输出层和隐层激活函数不同，反向传播分开计算，如下  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 2]

### 测试输出结果 ###

网络初始化参数，和上一篇浅层神经网络一样，得到结果也一样。layer\_dims：网络结构。layer\_dims\[0\]是input layer数据特征数；layer\_dims\[-1\]是 output layer输出结果，二分类是0或1；其它是hidden layer，每层的节点数。  
![在这里插入图片描述][20190927151128114.png]  
结果（隐层激活函数\[各层节点数\]=准去率）  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 3]  
下面是感觉很迷茫的结果。不知道为啥，relu多隐藏层后，总是如下图这样，哎！！！  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 4]

### 代码如下 ###

和上一篇浅层神经网络很像，主要修改了init\_parameters、forward\_propagation、back\_propagation和update\_parameters，添加了relu激活函数

""" @Time : 2019/9/26 20:29 PM @Author : bjjoy2009 """
    import matplotlib.pyplot as plt
    import numpy as np
    
    
    def sigmoid(Z):
        A = 1/(1 + np.exp(-Z))
        return A
    
    
    def relu(Z):
        A = np.maximum(0, Z)
        return A
    
    
    def sigmoid_loss(y, al):
        result = -1/len(y) * (np.dot(y, np.log(al).T) + np.dot(1-y, np.log(1-al).T))
        return result
    
    
    def load_planar_dataset():
        np.random.seed(1)
        m = 400  # number of examples
        N = int(m/2)  # number of points per class
        D = 2  # dimensionality
        X = np.zeros((m, D))  # data matrix where each row is a single example
        Y = np.zeros((m, 1), dtype='uint8')  # labels vector (0 for red, 1 for blue)
        a = 4  # maximum ray of the flower
    
        for j in range(2):
            ix = range(N*j, N*(j+1))
            t = np.linspace(j*3.12, (j+1)*3.12, N) + np.random.randn(N)*0.2  # theta
            r = a*np.sin(4*t) + np.random.randn(N)*0.2  # radius
            X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
            Y[ix] = j
    
        X = X.T
        Y = Y.T
    
        return X, Y
    
    
    def plot_decision_boundary(model, X, y, title):
        # Set min and max values and give it some padding
        x_min, x_max = X[0, :].min() - 1, X[0, :].max() + 1
        y_min, y_max = X[1, :].min() - 1, X[1, :].max() + 1
        h = 0.01
        # Generate a grid of points with distance h between them
        xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
        # Predict the function value for the whole grid
        Z = model(np.c_[xx.ravel(), yy.ravel()])
        Z = Z.reshape(xx.shape)
        # Plot the contour and training examples
        plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
        plt.ylabel('x2')
        plt.xlabel('x1')
        plt.scatter(X[0, :], X[1, :], c=y.ravel(), cmap=plt.cm.Spectral)
        plt.title(title)
        plt.show()
    
    
    class DNN:
        def __init__(self, X, Y, layer_dims, max_iter=10000, alpha=0.05, print_loss=False, activation='relu'):
            """ :param X: 训练集 :param Y: labels :param layer_dims: 各个层节点数list，layer_dims[0]训练集特征数，layer_dims[L]=1输出层节点数 :param alpha: 梯度下降学习率 """
            self.X = X
            self.Y = Y
            self.layer_dims = layer_dims
            self.max_iter = max_iter
            self.alpha = alpha
            self.m = self.X.shape[1]
            self.L = len(layer_dims)
            self.print_loss = print_loss
            self.parameters = { }
            self.activation = activation
    
        def init_parameters(self):
            parameters = { }
            for l in range(1, self.L):
                Wl = np.random.random((self.layer_dims[l], self.layer_dims[l-1])) * 0.01
                bl = np.zeros((self.layer_dims[l], 1))
                parameters['W' + str(l)] = Wl
                parameters['b' + str(l)] = bl
            return parameters
    
        def forward_propagation(self, parameters, X):
            cache = { 'A0': self.X}
            A_pre = X
            for l in range(1, self.L-1):
                Wl = parameters['W' + str(l)]
                bl = parameters['b' + str(l)]
                Zl = np.dot(Wl, A_pre) + bl
                if self.activation == 'relu':
                    Al = relu(Zl)  # relu做激活函数
                elif self.activation == 'tanh':
                    Al = np.tanh(Zl)  # tanh做激活函数
                A_pre = Al
                cache['A' + str(l)] = Al
                cache['Z' + str(l)] = Zl
    
            Wl = parameters['W' + str(self.L-1)]
            bl = parameters['b' + str(self.L-1)]
            Zl = np.dot(Wl, A_pre) + bl
            Al = sigmoid(Zl)
            cache['A' + str(self.L-1)] = Al
            cache['Z' + str(self.L-1)] = Zl
    
            return cache, Al
    
        def back_propagation(self, parameters, cache):
            grads = { }
            # 输出层反向传播（L-1层）
            L = self.L - 1
            Al = cache['A' + str(L)]
            Zl = cache['Z' + str(L)]
            dZl = Al - self.Y
            dWl = 1/self.m * np.dot(dZl, cache['A' + str(L-1)].T)
            dbl = 1/self.m * np.sum(dZl, axis=1, keepdims=True)
            grads['dW' + str(L)] = dWl
            grads['db' + str(L)] = dbl
            # 隐藏层和输入层反向传播（0~L-2层）
            for l in reversed(range(1, L)):
                dAl = np.dot(parameters['W' + str(l+1)].T, dZl)
                if self.activation == 'relu':
                    dZl = np.array(dAl, copy=True)  # relu做激活使用
                    dZl[cache['Z' + str(l)] <= 0] = 0  # relu做激活使用
                elif self.activation == 'tanh':
                    dZl = np.multiply(dAl, (1-np.power(cache['A' + str(l)], 2)))  # tanh做激活使用
                dWl = 1/self.m * np.dot(dZl, cache['A' + str(l-1)].T)
                dbl = 1/self.m * np.sum(dZl, axis=1, keepdims=True)
                grads['dW' + str(l)] = dWl
                grads['db' + str(l)] = dbl
            return grads
    
        def update_parameters(self, parameters, grads):
            for l in range(1, self.L):
                parameters['W' + str(l)] = parameters['W' + str(l)] - self.alpha * grads['dW' + str(l)]
                parameters['b' + str(l)] = parameters['b' + str(l)] - self.alpha * grads['db' + str(l)]
            return parameters
    
        def fit(self):
            parameters = self.init_parameters()
            for i in range(self.max_iter):
                cache, al = self.forward_propagation(parameters, self.X)
                if self.print_loss and i % 1000 == 0:
                    loss = sigmoid_loss(self.Y, al)
                    print(i, loss)
                grads = self.back_propagation(parameters, cache)
                parameters = self.update_parameters(parameters, grads)
            self.parameters = parameters
    
        def predict(self, X):
            cache, a2 = self.forward_propagation(self.parameters, X)
            predicts = (a2 > 0.5)
            return predicts
    
    
    np.random.seed(1)
    X, Y = load_planar_dataset()
    nn = DNN(X=X, Y=Y, layer_dims=[2, 8, 8, 1], alpha=0.5, print_loss=False, activation='relu')
    nn.fit()
    predicts = nn.predict(X)
    accuracy = float((np.dot(Y, predicts.T) + np.dot(1-Y, 1-predicts.T))/float(Y.size)*100)
    print('Accuracy: %f ' % accuracy+ '%')
    plot_decision_boundary(lambda x: nn.predict(x.T), X, Y, 'relu[2881]='+ str(accuracy)+ '%')

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70]: /images/20230810/05900c7dd37f447393a09c987f0dea9e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 1]: /images/20230810/d9dfa0257ec849edbbc2af463bd1e39f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 2]: /images/20230810/cceb5b95224e47aebacae4c0500400e9.png
[20190927151128114.png]: /images/20230810/59a52f7e7ade4131b25974509485245e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 3]: /images/20230810/fe5ac3df087c46838fb9116aadbdc2bd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jqam95MjAwOQ_size_16_color_FFFFFF_t_70 4]: /images/20230810/fdf7a92fd04b4e7b89ab2b5df763fd54.png