Shallow Neural Network Week 3

Single Sample

Symbols

X=⎛⎝⎜⎜x1⋮xnx⎞⎠⎟⎟,Y=⎛⎝⎜⎜y1⋮yny⎞⎠⎟⎟,$X=\left(\begin{array}{c}{x}_1\\ ⋮\\ x_{n_x}\end{array}\right),Y=\left(\begin{array}{c}{y}_1\\ ⋮\\ y_{n_y}\end{array}\right),$
Z[l]=⎛⎝⎜⎜⎜z[l]1⋮z[l]nl⎞⎠⎟⎟⎟,1≤l≤L$Z^{[l]}=\left(\begin{array}{c}{z}_1^{[l]}\\ ⋮\\ z_{n_l}^{[l]}\end{array}\right),1\le l\le L$
A[l]=⎛⎝⎜⎜⎜a[l]1⋮a[l]nl⎞⎠⎟⎟⎟,A~[l]=⎛⎝⎜⎜⎜⎜⎜⎜a[l]0a[l]1⋮a[l]nl⎞⎠⎟⎟⎟⎟⎟⎟=(1A[l]),0≤l≤L$A^{[l]}=\left(\begin{array}{c}{a}_1^{[l]}\\ ⋮\\ a_{n_l}^{[l]}\end{array}\right),{\tilde{A}}^{[l]}=\left(\begin{array}{c}{a}_0^{[l]}\\ a_1^{[l]}\\ ⋮\\ a_{n_l}^{[l]}\end{array}\right)=\left(\begin{array}{c}1\\ A^{[l]}\end{array}\right),0\le l\le L$
W[l]=(w[l]ij)nl×nl−1,w′[l]=⎛⎝⎜⎜⎜w[l]1,0⋮w[l]nl,0⎞⎠⎟⎟⎟,W~[l]=(w′[l]W[l]),1≤l<L$W^{[l]}={\left(\begin{array}{c}{w}_{ij}^{[l]}\end{array}\right)}_{n_l\times n_{l-1}},w^{\prime [l]}=\left(\begin{array}{c}{w}_{1,0}^{[l]}\\ ⋮\\ w_{n_l,0}^{[l]}\end{array}\right),{\tilde{W}}^{[l]}=\left(\begin{array}{cc}{w}^{\prime [l]}& W^{[l]}\end{array}\right),1\le l<L$

Neural Network Architecture

X=A[0]→Z[1]→A[1]→⋯→Z[L]→A[L]=Y^$X=A^{[0]}\to Z^{[1]}\to A^{[1]}\to \cdots \to Z^{[L]}\to A^{[L]}=\hat{Y}$

Loss Function

z[l]i=∑j=0nl−1w[l]ija~[l−1]j,1≤i≤nl,1≤l≤L$z_i^{[l]}=\sum _{j=0}^{n_{l-1}}{w}_{ij}^{[l]}{\tilde{a}}_j^{[l-1]},1\le i\le n_l,1\le l\le L$
即 Zl=W[l]A~[l−1],1≤l≤L$Z^l=W^{[l]}{\tilde{A}}^{[l-1]},1\le l\le L$
a[l]i=g(z[l]i),1≤i≤nl,1≤l≤L$a_i^{[l]}=g\left(z_i^{[l]}\right),1\le i\le n_l,1\le l\le L$
即 A[l]=g(Z[l]),1≤l≤L$A^{[l]}=g\left(Z^{[l]}\right),1\le l\le L$
loss(X,Y)=−∑i=1ny[yilny^i+(1−yi)ln(1−y^i)]$\mathrm{loss}(X,Y)=-\sum _{i=1}^{n_y}\left[y_i\ln {\hat{y}}_i+\left(1-y_i\right)\ln \left(1-{\hat{y}}_i\right)\right]$

公式

∂∂z[L]iloss(X,Y)=dy^idz[L]i⋅∂∂y^iloss(X,Y)${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_i^{[L]}}}\mathrm{loss}(X,Y)={\displaystyle \frac{\mathrm{d}{\hat{y}}_i}{\mathrm{d}{z}_i^{[L]}}}\cdot {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\hat{y}}_i}}\mathrm{loss}(X,Y)$
=−g′(z[L])[yi⋅1y^i−(1−yi)⋅11−y^i]$=-g^{\prime }\left(z^{[L]}\right)\left[y_i\cdot {\displaystyle \frac{1}{{\hat{y}}_i}}-\left(1-y_i\right)\cdot {\displaystyle \frac{1}{1-{\hat{y}}_i}}\right]$
=−y^i(1−y^i)[yi⋅1y^i−(1−yi)⋅11−y^i]$=-{\hat{y}}_i\left(1-{\hat{y}}_i\right)\left[y_i\cdot {\displaystyle \frac{1}{{\hat{y}}_i}}-\left(1-y_i\right)\cdot {\displaystyle \frac{1}{1-{\hat{y}}_i}}\right]$
=(1−yi)y^i−yi(1−y^i)$=\left(1-y_i\right){\hat{y}}_i-y_i\left(1-{\hat{y}}_i\right)$
=y^i−yi,1≤i≤nL$={\hat{y}}_i-y_i,1\le i\le n_L$

∂∂z[l]jloss(X,Y)=∑i=1nl+1∂z[l+1]i∂z[l]j⋅∂∂z[l+1]iloss(X,Y)${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_j^{[l]}}}\mathrm{loss}(X,Y)=\sum _{i=1}^{n_{l+1}}{\displaystyle \frac{\mathrm{\partial }{z}_i^{[l+1]}}{\mathrm{\partial }{z}_j^{[l]}}}\cdot {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_i^{[l+1]}}}\mathrm{loss}(X,Y)$
=∑i=1nl+1g′(z[l]j)w[l]ij⋅∂∂z[l+1]iloss(X,Y)$=\sum _{i=1}^{n_{l+1}}{g}^{\prime }\left(z_j^{[l]}\right)w_{ij}^{[l]}\cdot {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_i^{[l+1]}}}\mathrm{loss}(X,Y)$
=g′(z[l]j)∑i=1nl+1w[l]ij⋅∂∂z[l+1]iloss(X,Y),1≤j≤sl,1≤l<L$=g^{\prime }\left(z_j^{[l]}\right)\sum _{i=1}^{n_{l+1}}{w}_{ij}^{[l]}\cdot {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_i^{[l+1]}}}\mathrm{loss}(X,Y),1\le j\le s_l,1\le l<L$
因此
∂∂Z[l]loss(X,Y)=⎧⎩⎨⎪⎪A[L]−Y,l=Lg′(Z[l]) .∗ ((W[l+1])⊺∂∂Z[l+1]loss(X,Y)),1≤l<L${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{Z}^{[l]}}}\mathrm{loss}(X,Y)=\{\begin{array}{l}{A}^{[L]}-Y,l=L\\ g^{\prime }\left(Z^{[l]}\right).\ast \left({\left(W^{[l+1]}\right)}^{⊺}{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{Z}^{[l+1]}}}\mathrm{loss}(X,Y)\right),1\le l<L\end{array}$
where .* is element-wise product.

∂∂w[l]ijloss(X,Y)=∂∂z[l]iloss(X,Y)⋅a~[l−1]j,1≤i≤sl+1,0≤j≤sl,1≤l≤L${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{w}_{ij}^{[l]}}}\mathrm{loss}(X,Y)={\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_i^{[l]}}}\mathrm{loss}(X,Y)\cdot {\tilde{a}}_j^{[l-1]},1\le i\le s_{l+1},0\le j\le s_l,1\le l\le L$
因此
∂∂W~[l]loss(X,Y)=∂∂Z[l]loss(X,Y)⋅A~[l−1]⊺,1≤l≤L${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\tilde{W}}^{[l]}}}\mathrm{loss}(X,Y)={\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{Z}^{[l]}}}\mathrm{loss}(X,Y)\cdot {{\tilde{A}}^{[l-1]}}^{⊺},1\le l\le L$

Multiple Samples

Symbols

X=(X(1),⋯,X(m)),$\mathbf{X}=\left(\begin{array}{ccc}{X}^{(1)},& \cdots ,& X^{(m)}\end{array}\right),$
Y=(Y(1),⋯,Y(m)),$\mathbf{Y}=\left(\begin{array}{ccc}{Y}^{(1)},& \cdots ,& Y^{(m)}\end{array}\right),$
Z[l]=(Z[l](1),⋯,Z[l](m)),1≤l≤L${\mathbf{Z}}^{[l]}=\left(\begin{array}{ccc}{Z}^{[l](1)},& \cdots ,& Z^{[l](m)}\end{array}\right),1\le l\le L$
A[l]=(A[l](1),⋯,A[l](m)),0≤l≤L${\mathbf{A}}^{[l]}=\left(\begin{array}{ccc}{A}^{[l](1)},& \cdots ,& A^{[l](m)}\end{array}\right),0\le l\le L$
A~[l]=(A~[l](1),⋯,A~[l](m)),0≤l≤L${\stackrel{\mathbf{~}}{\mathbf{A}}}^{[l]}=\left(\begin{array}{ccc}{\tilde{A}}^{[l](1)},& \cdots ,& {\tilde{A}}^{[l](m)}\end{array}\right),0\le l\le L$
∂Z[l]=(∂∂Z[l]loss(X(1),Y(1)),⋯,∂∂Z[l]loss(X(m),Y(m)))nl×m,1≤l≤L$\mathrm{\partial }{\mathbf{Z}}^{[l]}={\left(\begin{array}{ccc}{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{Z}^{[l]}}}\mathrm{loss}\left(X^{(1)},Y^{(1)}\right),& \cdots ,& {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{Z}^{[l]}}}\mathrm{loss}\left(X^{(m)},Y^{(m)}\right)\end{array}\right)}_{n_l\times m},1\le l\le L$

Cost Function

cost(X,Y)=1m∑i=1mloss(X(i),Y(i))$\mathrm{cost}(\mathbf{X},\mathbf{Y})={\displaystyle \frac{1}{m}}\sum _{i=1}^m\mathrm{loss}\left(X^{(i)},Y^{(i)}\right)$

公式

Z[l]=W[l]A~[l−1],1≤l<L${\mathbf{Z}}^{[l]}=W^{[l]}{\stackrel{\mathbf{~}}{\mathbf{A}}}^{[l-1]},1\le l<L$
A[l]=g(Z[l]),1≤l≤L${\mathbf{A}}^{[l]}=g\left({\mathbf{Z}}^{[l]}\right),1\le l\le L$
g′(Z[l])=A[l] .∗ (1nl×m−A[l]),1≤l≤L$g^{\prime }\left({\mathbf{Z}}^{[l]}\right)={\mathbf{A}}^{[l]}.\ast \left(1_{n_l\times m}-{\mathbf{A}}^{[l]}\right),1\le l\le L$

∂Z[l]={A[L]−Y,l=Lg′(Z[l]) .∗ ((W[l+1])⊺⋅∂Z[l+1]),1≤l<L$\mathrm{\partial }{\mathbf{Z}}^{[l]}=\{\begin{array}{l}{\mathbf{A}}^{[L]}-\mathbf{Y},l=L\\ g^{\prime }\left({\mathbf{Z}}^{[l]}\right).\ast \left({\left(W^{[l+1]}\right)}^{⊺}\cdot \mathrm{\partial }{\mathbf{Z}}^{[l+1]}\right),1\le l<L\end{array}$
∂∂W~[l]cost(X,Y)=1m∂Z[l]⋅A~[l−1]⊺,1≤l≤L${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\tilde{W}}^{[l]}}}\mathrm{cost}(\mathbf{X},\mathbf{Y})={\displaystyle \frac{1}{m}}\mathrm{\partial }{\mathbf{Z}}^{[l]}\cdot {{\stackrel{\mathbf{~}}{\mathbf{A}}}^{[l-1]}}^{⊺},1\le l\le L$