Backpropagation algorithms are a family of methods used to efficiently train artificial neural networks (ANNs) following a gradient descent approach that exploits the chain rule. The main feature of backpropagation is its iterative, recursive and efficient method for calculating the weights updates to improve in the network until it is able to perform the task for which it is being trained.[1] It is closely related to the Gauss–Newton algorithm.
1943年，lMcCuIIochandPitts，1943]將生物神經網絡抽象爲圖所示的簡單
理想中的激活數是圖5．2間所示的階躍函數，它將輸入值映射爲輸出
“與” ( x 1 ∧ x 2 ) \left(x_{1} \wedge x_{2}\right) ( x 1 ∧ x 2 ) w 1 = w 2 = 1 w_{1}=w_{2}=1 w 1 = w 2 = 1 θ = 2 \theta=2 θ = 2 y = f ( 1 ⋅ x 1 + 1 ⋅ x 2 − 2 ) y=f\left(1 \cdot x_{1}+1 \cdot x_{2}-2\right) y = f ( 1 ⋅ x 1 + 1 ⋅ x 2 − 2 ) x 1 = x 2 = 1 x_{1}=x_{2}=1 x 1 = x 2 = 1
“或” ( x 1 ∨ x 2 ) \left(x_{1} \vee x_{2}\right) ( x 1 ∨ x 2 ) w 1 = w 2 = 1 w_{1}=w_{2}=1 w 1 = w 2 = 1 θ = 0.5 \theta=0.5 θ = 0 . 5 y = f ( 1 ⋅ x 1 + 1 ⋅ x 2 − 0.5 ) y=f\left(1 \cdot x_{1}+1 \cdot x_{2}-0.5\right) y = f ( 1 ⋅ x 1 + 1 ⋅ x 2 − 0 . 5 ) x 1 = 1 x_{1}=1 x 1 = 1 x 2 = 1 x_{2}=1 x 2 = 1 ( ¬ x 1 ) \left(\neg x_{1}\right) ( ¬ x 1 ) w 1 = − 0.6 , w 2 = 0 w_{1}=-0.6,w_{2}=0 w 1 = − 0 . 6 , w 2 = 0 θ = − 0.5 \theta=-0.5 θ = − 0 . 5 y = f ( − 0.6 ⋅ x 1 + 0 ⋅ x 2 + 0.5 ) y=f\left(-0.6 \cdot x_{1}+0 \cdot x_{2}+0.5\right) y = f ( − 0 . 6 ⋅ x 1 + 0 ⋅ x 2 + 0 . 5 ) x 1 = 1 x_{1}=1 x 1 = 1 x 1 = 1 x_{1}=1 x 1 = 1
要解決非線性可分問題，需考慮使用多層功能神經元．例如圖中這個
給定訓練數據集，權重(i=1，2，… ，n)以及θ \theta θ θ \theta θ w n + 1 w_{n+1} w n + 1 w i ← w i + Δ w i Δ w i = η ( y − y ^ ) x i
\begin{array}{c}{w_{i} \leftarrow w_{i}+\Delta w_{i}} \\ {\Delta w_{i}=\eta(y-\hat{y}) x_{i}}\end{array}
w i ← w i + Δ w i Δ w i = η ( y − y ^ ) x i η ∈ ( 0 , 1 ) \eta \in(0,1) η ∈ ( 0 , 1 ) y ^ = y \hat{y}=y y ^ = y
啞節點示意圖 :
多層網絡的學習能力比單層感知機強得多、欲訓練多層網絡，上面的簡單感知機學習規則顯然不夠了，需要更強大的學習算法．誤差逆傳播（error BackPropagation，簡稱BP）算法就是其中最傑出的代表，它是迄今最成功的神D = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m ) } , x i ∈ R d , y i ∈ R l D=\left\{\left(\boldsymbol{x}_{1}, \boldsymbol{y}_{1}\right)\right.,\left(\boldsymbol{x}_{2}, \boldsymbol{y}_{2}\right), \ldots,\left(\boldsymbol{x}_{m}, \boldsymbol{y}_{m}\right) \}, \boldsymbol{x}_{i} \in \mathbb{R}^{d}, \boldsymbol{y}_{i} \in \mathbb{R}^{l} D = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m ) } , x i ∈ R d , y i ∈ R l θ j \theta_{j} θ j γ h \gamma_{h} γ h v i h v_{i h} v i h w h j w_{h}j w h j α h = ∑ i = 1 d v i h x i \alpha_{h}=\sum_{i=1}^{d} v_{i h} x_{i} α h = ∑ i = 1 d v i h x i β j = ∑ h = 1 q w h j b h \beta_{j}=\sum_{h=1}^{q} w_{h j} b_{h} β j = ∑ h = 1 q w h j b h b n b_{n} b n x k , y k x_{k},y_{k} x k , y k y ^ k = ( y ^ 1 k , y ^ 2 k , … , y ^ l k ) \hat{\boldsymbol{y}}_{k}=\left(\hat{y}_{1}^{k}, \hat{y}_{2}^{k}, \ldots, \hat{y}_{l}^{k}\right) y ^ k = ( y ^ 1 k , y ^ 2 k , … , y ^ l k ) y ^ j k = f ( β j − θ j ) ( 3 ) \hat{y}_{j}^{k}=f\left(\beta_{j}-\theta_{j}\right) \qquad \qquad \qquad (3) y ^ j k = f ( β j − θ j ) ( 3 ) x k , y k x_{k},y_{k} x k , y k E k = 1 2 ∑ j = 1 l ( y ^ j k − y j k ) 2 q q u a d ( 4 ) E_{k}=\frac{1}{2} \sum_{j=1}^{l}\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2}qquad \qquad \qquad (4) E k = 2 1 j = 1 ∑ l ( y ^ j k − y j k ) 2 q q u a d ( 4 ) v ← v + Δ v ( 5 ) v \leftarrow v+\Delta v \qquad \qquad \qquad (5) v ← v + Δ v ( 5 ) w h j w_{h j} w h j E k E_{k} E k η \eta η Δ w h j = − η ∂ E k ∂ w h j ( 6 )
\Delta w_{h j}=-\eta \frac{\partial E_{k}}{\partial w_{h j}}\qquad \qquad \qquad (6)
Δ w h j = − η ∂ w h j ∂ E k ( 6 ) w h j w_{hj} w h j β j \beta_{j} β j y ^ j k \hat{y}_{j}^{k} y ^ j k E k E_{k} E k ∂ E k ∂ w h j = ∂ E k ∂ y ^ j k ⋅ ∂ y ^ j k ∂ β j ⋅ ∂ β j ∂ w h j ( 7 ) \frac{\partial E_{k}}{\partial w_{h j}}=\frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial w_{h j}}\qquad \qquad \qquad (7) ∂ w h j ∂ E k = ∂ y ^ j k ∂ E k ⋅ ∂ β j ∂ y ^ j k ⋅ ∂ w h j ∂ β j ( 7 ) β j \beta_{j} β j ∂ β j ∂ w h j = b h ( 8 ) \frac{\partial \beta_{j}}{\partial w_{h j}}=b_{h}\qquad \qquad \qquad (8) ∂ w h j ∂ β j = b h ( 8 ) f ′ ( x ) = f ( x ) ( 1 − f ( x ) ) ( 9 ) f^{\prime}(x)=f(x)(1-f(x))
\qquad \qquad \qquad (9) f ′ ( x ) = f ( x ) ( 1 − f ( x ) ) ( 9 )
推導過程Simoid -> s ( x ) = 1 1 + e − x s(x)=\frac{1}{1+e^{-x}} s ( x ) = 1 + e − x 1 f ( x ) = 1 s ( x ) = 1 + e − x f(x)=\frac{1}{s(x)}=1+e^{-x} f ( x ) = s ( x ) 1 = 1 + e − x f ′ ( x ) = − s ′ ( x ) s ( x ) 2 ( 1 ) f^{\prime}(x)=-\frac{s^{\prime}(x)}{s(x)^{2}}\qquad(1) f ′ ( x ) = − s ( x ) 2 s ′ ( x ) ( 1 ) f ′ ( x ) = ( 1 + e − x ) d d x = − e − x = 1 − f ( x ) = 1 − 1 s ( x ) = s ( x ) − 1 s ( x ) ( 2 ) f^{\prime}(x)=\left(1+e^{-x}\right)\frac{d}{d x}=-e^{-x}=1-f(x)=1-\frac{1}{s(x)}=\frac{s(x)-1}{s(x)}\qquad(2) f ′ ( x ) = ( 1 + e − x ) d x d = − e − x = 1 − f ( x ) = 1 − s ( x ) 1 = s ( x ) s ( x ) − 1 ( 2 ) s ′ ( x ) = s ( x ) ⋅ ( 1 − s ( x ) ) s^{\prime}(x)=s(x) \cdot(1-s(x)) s ′ ( x ) = s ( x ) ⋅ ( 1 − s ( x ) )
根據公式(3),(4),有g j = − ∂ E k ∂ y ^ j k ⋅ ∂ y ^ j k ∂ β j = − ( y ^ j k − y j k ) f ′ ( β j − θ j ) = y ^ j k ( 1 − y ^ j k ) ( y j k − y ^ j k ) ( 10 ) \begin{aligned} g_{j} &=-\frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \\ &=-\left(\hat{y}_{j}^{k}-y_{j}^{k}\right) f^{\prime}\left(\beta_{j}-\theta_{j}\right) \\ &=\hat{y}_{j}^{k}\left(1-\hat{y}_{j}^{k}\right)\left(y_{j}^{k}-\hat{y}_{j}^{k}\right) \end{aligned}
\qquad \qquad \qquad (10) g j = − ∂ y ^ j k ∂ E k ⋅ ∂ β j ∂ y ^ j k = − ( y ^ j k − y j k ) f ′ ( β j − θ j ) = y ^ j k ( 1 − y ^ j k ) ( y j k − y ^ j k ) ( 1 0 ) W h j W_{hj} W h j Δ w h j = η g j b h \Delta w_{h j}=\eta g_{j} b_{h} Δ w h j = η g j b h Δ θ j = − η g j Δ v i h = η e h x i Δ γ h = − η e h \begin{aligned} \Delta \theta_{j} &=-\eta g_{j} \\ \Delta v_{i h} &=\eta e_{h} x_{i} \\ \Delta \gamma_{h} &=-\eta e_{h} \end{aligned} Δ θ j Δ v i h Δ γ h = − η g j = η e h x i = − η e h e h = − ∂ E k ∂ b h ⋅ ∂ b h ∂ α h = − ∑ j = 1 l ∂ E k ∂ β j ⋅ ∂ β j ∂ b h f ′ ( α h − γ h ) = ∑ j = 1 l w h j g j f ′ ( α h − γ h ) = b h ( 1 − b h ) ∑ j = 1 l w h j g j \begin{aligned} e_{h} &=-\frac{\partial E_{k}}{\partial b_{h}} \cdot \frac{\partial b_{h}}{\partial \alpha_{h}} \\ &=-\sum_{j=1}^{l} \frac{\partial E_{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial b_{h}} f^{\prime}\left(\alpha_{h}-\gamma_{h}\right) \end{aligned}\\=\sum_{j=1}^{l} w_{h j} g_{j} f^{\prime}\left(\alpha_{h}-\gamma_{h}\right) \\ =b_{h}\left(1-b_{h}\right) \sum_{j=1}^{l} w_{h j} g_{j} e h = − ∂ b h ∂ E k ⋅ ∂ α h ∂ b h = − j = 1 ∑ l ∂ β j ∂ E k ⋅ ∂ b h ∂ β j f ′ ( α h − γ h ) = j = 1 ∑ l w h j g j f ′ ( α h − γ h ) = b h ( 1 − b h ) j = 1 ∑ l w h j g j η ∈ ( 0 , 1 ) \eta \in(0,1) η ∈ ( 0 , 1 )
指數衰減學習率。這個算法實現達到的效果是:隨着迭代次數的增加逐步減小學習率，是模型前期訓練速度加快，後期訓練模型更加穩定而不會出現極優值兩邊跳動的情況。
E = 1 m ∑ k = 1 m E k E=\frac{1}{m} \sum_{k=1}^{m} E_{k} E = m 1 k = 1 ∑ m E k
補充:E k E_{k} E k w i w_{i} w i E = λ 1 m ∑ k = 1 m E k + ( 1 − λ ) ∑ i w i 2 E=\lambda \frac{1}{m} \sum_{k=1}^{m} E_{k}+(1-\lambda) \sum_{i} w_{i}^{2} E = λ m 1 k = 1 ∑ m E k + ( 1 − λ ) i ∑ w i 2
