Lecture 4: Backpropagation and Neural Networks

CS231n

Lecture 4: Backpropagation and Neural Networks

Backpropagation

反向傳播的理論基礎：多元函數微分學中的鏈式法則
z=z(x,y),x=x(u,v),y=y(u,v)⇒dz=∂z∂xdx+∂z∂ydy=∂z∂x(∂x∂udu+∂x∂vdv)+∂z∂y(∂y∂udu+∂y∂vdv)⇒∂z∂u=∂z∂x∂x∂u+∂z∂y∂y∂u$z=z(x,y),x=x(u,v),y=y(u,v)\Rightarrow \mathrm{d}z=\frac{\mathrm{\partial }z}{\mathrm{\partial }x}\mathrm{d}x+\frac{\mathrm{\partial }z}{\mathrm{\partial }y}\mathrm{d}y=\frac{\mathrm{\partial }z}{\mathrm{\partial }x}(\frac{\mathrm{\partial }x}{\mathrm{\partial }u}\mathrm{d}u+\frac{\mathrm{\partial }x}{\mathrm{\partial }v}\mathrm{d}v)+\frac{\mathrm{\partial }z}{\mathrm{\partial }y}(\frac{\mathrm{\partial }y}{\mathrm{\partial }u}\mathrm{d}u+\frac{\mathrm{\partial }y}{\mathrm{\partial }v}\mathrm{d}v)\Rightarrow \frac{\mathrm{\partial }z}{\mathrm{\partial }u}=\frac{\mathrm{\partial }z}{\mathrm{\partial }x}\frac{\mathrm{\partial }x}{\mathrm{\partial }u}+\frac{\mathrm{\partial }z}{\mathrm{\partial }y}\frac{\mathrm{\partial }y}{\mathrm{\partial }u}$
於是梯度可以不斷向前傳遞
Q2: what does it look like?
A: Jacobian shape難道不是100×4096×4096$100\times 4096\times 4096$ ？

Neural Networks

[FC+ReLU]xN + Softmax/SVM
單層感知機來源於神經生物學，但是現在的神經網絡並沒有仿生人類神經