Backpropagation Algorithm 的梯度

損失函數 J(θ)$\mathrm{J}\left(\theta \right)$

J(θ)=−1m∑i=1m∑k=1K[y(i)kln(hθ(X(i))k)+(1−y(i)k)ln(1−hθ(X(i))k)]$\mathrm{J}\left(\theta \right)=-{\displaystyle \frac{1}{m}}\sum _{i=1}^m\sum _{k=1}^K\left[y_k^{\left(i\right)}\ln \left(h_{\theta }{\left(X^{\left(i\right)}\right)}_k\right)+\left(1-y_k^{\left(i\right)}\right)\ln \left(1-h_{\theta }{\left(X^{\left(i\right)}\right)}_k\right)\right]$
              +λ2m∑l=1L−1∑i=1sl+1∑j=1sl(θ(l)i,j)2$+{\displaystyle \frac{\lambda }{2m}}\sum _{l=1}^{L-1}\sum _{i=1}^{s_{l+1}}\sum _{j=1}^{s_l}{\left({\theta }_{i,j}^{\left(l\right)}\right)}^2$

λ=0$\lambda =0$ 時的單樣本損失函數 cost(θ;X,Y)$\mathrm{cost}\left(\theta ;X,Y\right)$

λ=0$\lambda =0$ 時，單一樣本 X=⎛⎝⎜⎜x1⋮xs1⎞⎠⎟⎟,Y=⎛⎝⎜⎜y1⋮yK⎞⎠⎟⎟$X=\left(\begin{array}{c}{x}_1\\ ⋮\\ x_{s_1}\end{array}\right),Y=\left(\begin{array}{c}{y}_1\\ ⋮\\ y_K\end{array}\right)$ 的損失函數：
cost(θ;X,Y)=−∑k=1K[ykln(hθ(X)k)+(1−yk)ln(1−hθ(X)k)]$\mathrm{cost}\left(\theta ;X,Y\right)=-\sum _{k=1}^K\left[y_k\ln \left(h_{\theta }{\left(X\right)}_k\right)+\left(1-y_k\right)\ln \left(1-h_{\theta }{\left(X\right)}_k\right)\right]$

令 a(1)=X$a^{\left(1\right)}=X$
Z(l+1)=θ(l)a(l),1≤l≤L−1$Z^{\left(l+1\right)}={\theta }^{\left(l\right)}{a}^{\left(l\right)},1\le l\le L-1$
a(l)=g(Z(l)),2≤l≤L,$a^{\left(l\right)}=g\left(Z^{\left(l\right)}\right),2\le l\le L,$ 其中函數 g$g$ 是 Logistic 函數。
則 a(L)=hθ(X)$a^{\left(L\right)}=h_{\theta }\left(X\right)$
於是 cost(θ;X,Y)=−∑k=1K[yklna(L)k+(1−yk)ln(1−a(L)k)]$\mathrm{cost}\left(\theta ;X,Y\right)=-\sum _{k=1}^K\left[y_k\ln a_k^{\left(L\right)}+\left(1-y_k\right)\ln \left(1-a_k^{\left(L\right)}\right)\right]$
則 J(θ)=1m∑i=1mcost(θ;X(i),Y(i))+λ2m∑l=1L−1∑i=1sl+1∑j=1sl(θ(l)i,j)2$\mathrm{J}\left(\theta \right)={\displaystyle \frac{1}{m}}\sum _{i=1}^m\mathrm{cost}\left(\theta ;X^{\left(i\right)},Y^{\left(i\right)}\right)+{\displaystyle \frac{\lambda }{2m}}\sum _{l=1}^{L-1}\sum _{i=1}^{s_{l+1}}\sum _{j=1}^{s_l}{\left({\theta }_{i,j}^{\left(l\right)}\right)}^2$

cost(θ;X,Y)$\mathrm{cost}\left(\theta ;X,Y\right)$ 關於 Z(l)$Z^{\left(l\right)}$ 的梯度

令 δ(l)=∂∂Z(l)cost(θ;X,Y)=⎛⎝⎜⎜⎜⎜⎜⎜⎜⎜∂∂z(l)1cost(θ;X,Y)⋮∂∂z(l)slcost(θ;X,Y)⎞⎠⎟⎟⎟⎟⎟⎟⎟⎟,2≤l≤L,${\delta }^{\left(l\right)}={\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{Z}^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X,Y\right)=\left(\begin{array}{c}{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_1^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X,Y\right)\\ ⋮\\ {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_{s_l}^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X,Y\right)\end{array}\right),2\le l\le L,$
則 δ(l)={a(L)−Y,(θ(l))⊺δ(l+1) .∗ a(l) .∗ (1−a(l)),l=L,2≤l≤L−1,${\delta }^{\left(l\right)}=\{\begin{array}{ll}{a}^{\left(L\right)}-Y,& l=L,\\ {\left({\theta }^{\left(l\right)}\right)}^{⊺}{\delta }^{\left(l+1\right)}.\ast a^{\left(l\right)}.\ast \left(1-a^{\left(l\right)}\right),& 2\le l\le L-1,\end{array}$
其中運算符  .∗ $.\ast$ 爲 element-wise 的乘積，如 ⎛⎝⎜⎜x1⋮xn⎞⎠⎟⎟ .∗ ⎛⎝⎜⎜y1⋮yn⎞⎠⎟⎟=⎛⎝⎜⎜x1y1⋮xnyn⎞⎠⎟⎟$\left(\begin{array}{c}{x}_1\\ ⋮\\ x_n\end{array}\right).\ast \left(\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}\right)=\left(\begin{array}{c}{x}_1{y}_1\\ ⋮\\ x_n{y}_n\end{array}\right)$ 。

證明

命題等價於：
δ(l)j=⎧⎩⎨⎪⎪a(L)j−yj,[∑i=1sl+1θ(l)i,jδ(l+1)i]⋅δ(l)j(1−a(l)j),l=L,2≤l≤L−1,1≤j≤sl${\delta }_j^{\left(l\right)}=\{\begin{array}{ll}{a}_j^{\left(L\right)}-y_j,& l=L,\\ \left[\sum _{i=1}^{s_{l+1}}{\theta }_{i,j}^{\left(l\right)}{\delta }_i^{\left(l+1\right)}\right]\cdot {\delta }_j^{\left(l\right)}\left(1-a_j^{\left(l\right)}\right),& 2\le l\le L-1,\end{array}1\le j\le s_l$

由 {Z(l+1)=θ(l)a(l),a(l)=g(Z(l)),1≤l≤L−1,2≤l≤L,$\{\begin{array}{ll}{Z}^{\left(l+1\right)}={\theta }^{\left(l\right)}{a}^{\left(l\right)},& 1\le l\le L-1,\\ a^{\left(l\right)}=g\left(Z^{\left(l\right)}\right),& 2\le l\le L,\end{array}$ 得：
⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪∂z(l+1)i∂a(l)j=θ(l)i,j,1≤l≤L−1,da(l)jdz(l)j=g′(z(l)j)=a(l)j(1−a(l)j),2≤l≤L,$\{\begin{array}{l}{\displaystyle \frac{\mathrm{\partial }{z}_i^{\left(l+1\right)}}{\mathrm{\partial }{a}_j^{\left(l\right)}}}={\theta }_{i,j}^{\left(l\right)},1\le l\le L-1,\\ {\displaystyle \frac{\mathrm{d}{a}_j^{\left(l\right)}}{\mathrm{d}{z}_j^{\left(l\right)}}}=g^{\prime }\left(z_j^{\left(l\right)}\right)=a_j^{\left(l\right)}\left(1-a_j^{\left(l\right)}\right),2\le l\le L,\end{array}$
因此 ∂z(l+1)i∂z(l)j=θ(l)i,ja(l)j(1−a(l)j),2≤l≤L−1,${\displaystyle \frac{\mathrm{\partial }{z}_i^{\left(l+1\right)}}{\mathrm{\partial }{z}_j^{\left(l\right)}}}={\theta }_{i,j}^{\left(l\right)}{a}_j^{\left(l\right)}\left(1-a_j^{\left(l\right)}\right),2\le l\le L-1,$
所以 δ(l)j=∑i=1sl+1δ(l+1)i∂z(l+1)i∂z(l)j${\delta }_j^{\left(l\right)}=\sum _{i=1}^{s_{l+1}}{\delta }_i^{\left(l+1\right)}{\displaystyle \frac{\mathrm{\partial }{z}_i^{\left(l+1\right)}}{\mathrm{\partial }{z}_j^{\left(l\right)}}}$
               =∑i=1sl+1δ(l+1)iθ(l)i,ja(l)j(1−a(l)j)$=\sum _{i=1}^{s_{l+1}}{\delta }_i^{\left(l+1\right)}{\theta }_{i,j}^{\left(l\right)}{a}_j^{\left(l\right)}\left(1-a_j^{\left(l\right)}\right)$
               =[∑i=1sl+1θ(l)i,jδ(l+1)i]⋅δ(l)j(1−a(l)j),2≤l≤L−1,$=\left[\sum _{i=1}^{s_{l+1}}{\theta }_{i,j}^{\left(l\right)}{\delta }_i^{\left(l+1\right)}\right]\cdot {\delta }_j^{\left(l\right)}\left(1-a_j^{\left(l\right)}\right),2\le l\le L-1,$

由於 ∂∂a(L)kcost(θ;X,Y)=−[yk1a(L)k−(1−yk)11−a(L)k]${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{a}_k^{\left(L\right)}}}\mathrm{cost}\left(\theta ;X,Y\right)=-\left[y_k{\displaystyle \frac{1}{a_k^{\left(L\right)}}}-\left(1-y_k\right){\displaystyle \frac{1}{1-a_k^{\left(L\right)}}}\right]$
                                            =−(yk−a(L)k)1a(L)k(1−a(L)k)$=-\left(y_k-a_k^{\left(L\right)}\right){\displaystyle \frac{1}{a_k^{\left(L\right)}\left(1-a_k^{\left(L\right)}\right)}}$
                                            =(a(L)k−yk)1a(L)k(1−a(L)k),1≤k≤sL=K$=\left(a_k^{\left(L\right)}-y_k\right){\displaystyle \frac{1}{a_k^{\left(L\right)}\left(1-a_k^{\left(L\right)}\right)}},1\le k\le s_L=K$
因此 (δ(L))j=∂∂aL,jcost(θ;X,Y)da(L)jdzL,j${\left({\delta }^{\left(L\right)}\right)}_j={\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{a}_{L,j}}}\mathrm{cost}\left(\theta ;X,Y\right){\displaystyle \frac{\mathrm{d}{a}_j^{\left(L\right)}}{\mathrm{d}{z}_{L,j}}}$
                      =(a(L)j−yj)1a(L)j(1−a(L)j)a(L)j(1−a(L)j)$=\left(a_j^{\left(L\right)}-y_j\right){\displaystyle \frac{1}{a_j^{\left(L\right)}\left(1-a_j^{\left(L\right)}\right)}}{a}_j^{\left(L\right)}\left(1-a_j^{\left(L\right)}\right)$
                      =a(L)j−yj,1≤j≤sL$=a_j^{\left(L\right)}-y_j,1\le j\le s_L$
因此，命題成立。

cost(θ;X,Y)$\mathrm{cost}\left(\theta ;X,Y\right)$ 關於 θ$\theta$ 的梯度

∂∂θ(l)i,jcost(θ;X,Y)=δ(l+1)ia(l)j,1≤l<L−1${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }_{i,j}^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X,Y\right)={\delta }_i^{\left(l+1\right)}{a}_j^{\left(l\right)},1\le l<L-1$

證明

由 ∂z(l+1)i∂θ(l)i,j=a(l)j,1≤l≤L−1,${\displaystyle \frac{\mathrm{\partial }{z}_i^{\left(l+1\right)}}{\mathrm{\partial }{\theta }_{i,j}^{\left(l\right)}}}=a_j^{\left(l\right)},1\le l\le L-1,$
得 ∂∂θ(l)i,jcost(θ;X,Y)=δ(l+1)i∂z(l+1)i∂θ(l)i,j=δ(l+1)ia(l)j,1≤l<L−1${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }_{i,j}^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X,Y\right)={\delta }_i^{\left(l+1\right)}{\displaystyle \frac{\mathrm{\partial }{z}_i^{\left(l+1\right)}}{\mathrm{\partial }{\theta }_{i,j}^{\left(l\right)}}}={\delta }_i^{\left(l+1\right)}{a}_j^{\left(l\right)},1\le l<L-1$

推論

∂∂θ(l)cost(θ;X,Y)=δ(l+1)(a(l))⊺,1≤l<L−1${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X,Y\right)={\delta }^{\left(l+1\right)}{\left(a^{\left(l\right)}\right)}^{⊺},1\le l<L-1$

損失函數 J(θ)$\mathrm{J}\left(\theta \right)$ 關於 θ$\theta$ 的梯度

∀t∈N,1≤t≤m,$\mathrm{\forall }t\in \mathbb{N},1\le t\le m,$
令 a(t,1)=X(t),$a^{\left(t,1\right)}=X^{\left(t\right)},$
Z(t,l+1)=θ(l)a(t,l),1≤l≤L−1,$Z^{\left(t,l+1\right)}={\theta }^{\left(l\right)}{a}^{\left(t,l\right)},1\le l\le L-1,$
a(t,l)=g(Z(t,l)),2≤l≤L,$a^{\left(t,l\right)}=g\left(Z^{\left(t,l\right)}\right),2\le l\le L,$
則 a(t,L)=hθ(X(t))$a^{\left(t,L\right)}=h_{\theta }\left(X^{\left(t\right)}\right)$
令 δ(t,l)=∂∂Z(t,l)cost(θ;X(t),Y(t))=⎛⎝⎜⎜⎜⎜⎜⎜⎜⎜∂∂z(t,l)1cost(θ;X(t),Y(t))⋮∂∂z(t,l)slcost(θ;X(t),Y(t))⎞⎠⎟⎟⎟⎟⎟⎟⎟⎟,2≤l≤L,${\delta }^{\left(t,l\right)}={\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{Z}^{\left(t,l\right)}}}\mathrm{cost}\left(\theta ;X^{\left(t\right)},Y^{\left(t\right)}\right)=\left(\begin{array}{c}{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_1^{\left(t,l\right)}}}\mathrm{cost}\left(\theta ;X^{\left(t\right)},Y^{\left(t\right)}\right)\\ ⋮\\ {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_{s_l}^{\left(t,l\right)}}}\mathrm{cost}\left(\theta ;X^{\left(t\right)},Y^{\left(t\right)}\right)\end{array}\right),2\le l\le L,$
則 δ(t,l)={a(t,L)−Y(t),(θ(l))⊺δ(t,l+1) .∗ a(t,l) .∗ (1−a(t,l)),l=L,2≤l≤L−1,${\delta }^{\left(t,l\right)}=\{\begin{array}{ll}{a}^{\left(t,L\right)}-Y^{\left(t\right)},& l=L,\\ {\left({\theta }^{\left(l\right)}\right)}^{⊺}{\delta }^{\left(t,l+1\right)}.\ast a^{\left(t,l\right)}.\ast \left(1-a^{\left(t,l\right)}\right),& 2\le l\le L-1,\end{array}$
於是 ∂∂θ(l)i,jcost(θ;X(t),Y(t))=δ(t,l+1)ia(t,l)j,1≤l<L−1${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }_{i,j}^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X^{\left(t\right)},Y^{\left(t\right)}\right)={\delta }_i^{\left(t,l+1\right)}{a}_j^{\left(t,l\right)},1\le l<L-1$
因此  ∂∂θ(l)i,jJ(θ)=1m∑t=1m∂∂θ(l)i,jcost(θ;X(t),Y(t))+λmθ(l)i,j${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }_{i,j}^{\left(l\right)}}}\mathrm{J}\left(\theta \right)={\displaystyle \frac{1}{m}}\sum _{t=1}^m{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }_{i,j}^{\left(l\right)}}}\mathrm{cost}\left(\theta ;X^{\left(t\right)},Y^{\left(t\right)}\right)+{\displaystyle \frac{\lambda }{m}}{\theta }_{i,j}^{\left(l\right)}$
                           =1m∑i=1mδ(t,l+1)ia(t,l)j+λmθ(l)i,j,1≤l≤L−1$={\displaystyle \frac{1}{m}}\sum _{i=1}^m{\delta }_i^{\left(t,l+1\right)}{a}_j^{\left(t,l\right)}+{\displaystyle \frac{\lambda }{m}}{\theta }_{i,j}^{\left(l\right)},1\le l\le L-1$

推論

 ∂∂θ(l)J(θ)=1m∑i=1mδ(t,l+1)(a(t,l))⊺+λmθ(l),1≤l≤L−1${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }^{\left(l\right)}}}\mathrm{J}\left(\theta \right)={\displaystyle \frac{1}{m}}\sum _{i=1}^m{\delta }^{\left(t,l+1\right)}{\left(a^{\left(t,l\right)}\right)}^{⊺}+{\displaystyle \frac{\lambda }{m}}{\theta }^{\left(l\right)},1\le l\le L-1$