深度學習——反向傳播算法推導

前言

之前寫過單層前饋神經網絡，但是其中的推導是針對sigmoid函數的，本篇博客使用矩陣向量求導方式進行反向傳播算法的推導

符號約定

符號	含義
$S_{in}^i$	第$i$層神經元的輸入，若一層有n個神經元，則$S_{in}^i$是一個$n*1$的向量
$S_{out}^i$	第$i$層神經元的輸出，若一層有n個神經元，則$S_{out}^i$是一個$n*1$的向量
$W^i$	第$i$層神經元對應的權重矩陣，若$i-1$層有$m$個神經元，第$i$層有$n$個神經元，則$W^i$爲$n*m$的矩陣
$B^i$	第$i$層的偏移矩陣，若一層有n個神經元，則$B^i$是一個$n*1$的向量
$cost$	損失函數值

若$x$表示$\begin{bmatrix} x_1\\ x_2\\ ....\\ x_n \end{bmatrix}$，第i層的激活函數向量$f^i(x)$表示爲$\begin{bmatrix} f(x_1)\\ f(x_2)\\ ....\\ f(x_n) \end{bmatrix}$，$f(x)$爲激活函數，$(f^i(x))'$表示爲$\begin{bmatrix} \frac{\partial{f(x_1)}}{{\partial x_1}}\\ \frac{\partial{f(x_2)}}{{\partial x_2}}\\ ....\\ \frac{\partial{f(x_n)}}{{\partial x_n}} \end{bmatrix}$

基於上述符號約定，對於第$i$層的神經元，我們有
$\begin{aligned} S_{out}^{i-1}=&f^i(S_{in}^{i-1})\\ S_{in}^i=&W^iS_{out}^{i-1}+B^i \end{aligned}$

標量對向量求導的鏈式法則

對於$n$層前饋神經網絡，我們有
$cost\leftarrow S_{in}^n\leftarrow S_{in}^{n-1}.....\leftarrow S_{in}^1$
左箭頭表示映射，對於前饋神經網絡，映射即爲
$\begin{aligned} S_{in}^{i+1}=W^{i+1}f^{i}(S_{in}^{i})+B^{i+1} \end{aligned}$
損失函數與最後一層的映射需要依據損失函數的類型決定（例如均方誤差、交叉熵），在上述映射關係的基礎上，標量對向量求導的鏈式法則定義爲
$\begin{aligned} \frac{\partial cost}{\partial S_{in}^i}&=(\frac{\partial S_{in}^n}{\partial S_{in}^{n-1}}*\frac{\partial S_{in}^{n-1}}{\partial S_{in}^{n-2}}*.......*\frac{\partial S_{in}^{i+1}}{\partial S_{in}^{i}})^T*\frac{\partial cost}{\partial S_{in}^n}\\ &=(\frac{\partial S_{in}^{i+1}}{\partial S_{in}^{i}})^T*.....*(\frac{\partial S_{in}^{n-1}}{\partial S_{in}^{n-2}})^T*(\frac{\partial S_{in}^n}{\partial S_{in}^{n-1}})^T*\frac{\partial cost}{\partial S_{in}^n}\\ &=(\frac{\partial S_{in}^{i+1}}{\partial S_{in}^{i}})^T*.....*(\frac{\partial S_{in}^{n-1}}{\partial S_{in}^{n-2}})^T*\frac{\partial cost}{\partial S_{in}^{n-1}}\\ &=.......\\ &=(\frac{\partial S_{in}^{i+1}}{\partial S_{in}^{i}})^T\frac{\partial cost}{\partial S_{in}^{i+1}} \end{aligned}$

常用向量對向量求導的公式

若$Y=AX+B$，$Y$、$X、B$爲向量，$A$爲矩陣，使用分子佈局，則有$\frac{\partial{Y}}{\partial{X}}=A$

反向傳播算法推導

假設有一個n層前饋神經網絡，則第$i$層的梯度爲
$\begin{aligned} \frac{\partial cost}{\partial S_{in}^i}&=(\frac{\partial S_{in}^{i+1}}{\partial S_{in}^{i}})^T*\frac{\partial cost}{\partial S_{in}^{i+1}}\\ &=(W^{i+1})^T*\frac{\partial cost}{\partial S_{in}^{i+1}} ☉ (f^{i}(S_{in}^i))' \end{aligned}\tag{式1}$
☉爲Hadamard乘積，用於矩陣或向量之間點對點的乘法運算，即相同位置的元素相乘，對於最後一步，具體的理解如下，假設第$i$層有n個神經元
$\begin{aligned} (W^{i+1})^T*\frac{\partial cost}{\partial S_{in}^{i+1}}☉ (f^{i}(S_{in}^i))'=&(\frac{\partial S_{in}^{i+1}}{\partial f(S_{in}^{i})})^T*\frac{\partial cost}{\partial S_{in}^{i+1}}☉ (f^{i}(S_{in}^i))'\\ =&\frac{\partial cost}{\partial f(S_{in}^{i})}☉ (f^{i}(S_{in}^i))'\\ =& \begin{bmatrix} \frac{\partial cost}{\partial f((S_{in}^{i})_1)}\\ \frac{\partial cost}{\partial f((S_{in}^{i})_2)}\\ ......\\ \frac{\partial cost}{\partial f((S_{in}^{i})_n)} \end{bmatrix}☉ (f^{i}(S_{in}^i))'\\ =& \begin{bmatrix} \frac{\partial cost}{\partial f((S_{in}^{i})_1)}\\ \frac{\partial cost}{\partial f((S_{in}^{i})_2)}\\ ......\\ \frac{\partial cost}{\partial f((S_{in}^{i})_n)} \end{bmatrix}☉ \begin{bmatrix} \frac{\partial {f((S_{in}^{i})_1)}}{\partial((S_{in}^{i})_1)}\\ \frac{\partial {f((S_{in}^{i})_2)}}{\partial((S_{in}^{i})_2)}\\ ......\\ \frac{\partial {f((S_{in}^{i})_n)}}{\partial((S_{in}^{i})_n)} \end{bmatrix}\\ =&\begin{bmatrix} \frac{\partial cost}{\partial f((S_{in}^{i})_1)}*\frac{\partial {f((S_{in}^{i})_1)}}{\partial((S_{in}^{i})_1)}\\ \frac{\partial cost}{\partial f((S_{in}^{i})_2)}*\frac{\partial {f((S_{in}^{i})_2)}}{\partial((S_{in}^{i})_2)}\\ ......\\ \frac{\partial cost}{\partial f((S_{in}^{i})_n)}*\frac{\partial {f((S_{in}^{i})_n)}}{\partial((S_{in}^{i})_n)} \end{bmatrix}\\ =&\begin{bmatrix} \frac{\partial cost}{\partial((S_{in}^{i})_1)}\\ \frac{\partial cost}{\partial((S_{in}^{i})_2)}\\ ......\\ \frac{\partial cost}{\partial((S_{in}^{i})_n)} \end{bmatrix}\\ =&\frac{\partial cost}{\partial S_{in}^i} \end{aligned}$
接下來就是權重更新的梯度，推出第$i$層的梯度後，對權重梯度與偏移的求導可以使用定義法求得到：
$\begin{aligned} \frac{\partial cost}{\partial W^i}&=\frac{\partial cost}{\partial S_{in}^i}*(S_{out}^{i-1})^T\tag{式2} \end{aligned}$
$\begin{aligned} \frac{\partial cost}{\partial B^i}&=\frac{\partial cost}{\partial S_{in}^i}\tag{式3} \end{aligned}$
$\frac{\partial cost}{\partial S_{in}^n}$需要依據矩陣求導的定義法自己求出，求出後，即可依據式1、2、3求出各參數的梯度，關於矩陣求導的定義法，可以查看快，快點我，我等不及了