Softmax函數和交叉熵Cross-entropy以及KL散度求導

參考鏈接：https://blog.csdn.net/qian99/article/details/78046329

交叉熵cross-entropy

對一個分類神經網絡$f$，輸出爲$z=f(x;\theta),z=[z_{0},z_{1},\cdots,z_{C-1}]$,$z$爲logits，其中類別數量爲$C$,$y$爲$x$的one-hot標籤。通過softmax歸一化來得到概率：
$p_{i}=\frac{\exp{z_{i}}}{\sum_{j}{\exp{z_{j}}}}$
負交叉熵誤差爲：
$\mathcal{L}=-\sum_{i}y_{i}\log{p_{i}}$
誤差對於概率的梯度爲：
$\frac{\partial \mathcal{L}}{\partial p_{i}}=-y_{i}\frac{1}{p_{i}}$
緊接着計算$\frac{\partial \mathcal{p_{i}}}{\partial z_{k}},k=0,1,...,C-1$:
（1）當$k=i$時，
$\frac{\partial \mathcal{p_{i}}}{\partial z_{i}}=\frac{\partial ( \frac{\exp{z_{i}}}{\sum_{j}{\exp{z_{j}}}})}{\partial z_{i}}=\frac{\exp{z_{i}}\sum_{j}\exp{z_{j}}-(\exp{z_{i}})^{2}}{(\sum_{j}{\exp{z_{j}}})^{2}} \\ =( \frac{\exp{z_{i}}}{\sum_{j}{\exp{z_{j}}}})(1- \frac{\exp{z_{i}}}{\sum_{j}{\exp{z_{j}}}})=p_{i}(1-p_{i})$

（2）當$k\neq i$時，
$\frac{\partial \mathcal{p_{i}}}{\partial z_{k}}=\frac{\partial ( \frac{\exp{z_{i}}}{\sum_{j}{\exp{z_{j}}}})}{\partial z_{k}}=\frac{-\exp{z_{i}}\exp{z_{k}}}{(\sum_{j}{\exp{z_{j}}})^{2}} =-p_{i}p_{k}$
根據求導的鏈式法則：
$\frac{\partial \mathcal{\mathcal{L}}}{\partial z_{k}}=\sum_{j}(\frac{\partial \mathcal{L}}{\partial p_{j}}\frac{\partial \mathcal{p_{j}}}{\partial z_{k}})\\ =\sum_{j=/k}(\frac{\partial \mathcal{L}}{\partial p_{j}}\frac{\partial \mathcal{p_{j}}}{\partial z_{k}})+(\frac{\partial \mathcal{L}}{\partial p_{k}}\frac{\partial \mathcal{p_{k}}}{\partial z_{k}})\\ =\sum_{j=/k}(-y_{j}\frac{1}{p_{j}}*-p_{j}p_{k})+(-y_{k}\frac{1}{p_{k}}*p_{k}(1-p_{k}))\\ =\sum_{j=/k}(y_{j}p_{k})-y_{k}+y_{k}p_{k}\\ =p_{k}\sum_{j}y_{j}-y_{k}$
因爲$y$爲one-hot編碼，所以$\sum_{j}y_{j}=1$,i.e.,
$\frac{\partial \mathcal{\mathcal{L}}}{\partial z_{k}}=p_{k}-y_{k}$

相對熵KL散度

預測的概率分佈$p$,真實概率分佈爲$q$，KL的散度爲：
$\mathcal{L}=KL(q||p)=\sum_{k}q_{c}\log{\frac{q_{k}}{p_{k}}}$
求解對概率$p_{k}$的梯度
$\frac{\partial \mathcal{\mathcal{L}}}{\partial p_{k}}=-\frac{q_{k}}{p_{k}}$
求解對logits $z_{k}$的梯度:
$\frac{\partial \mathcal{\mathcal{L}}}{\partial z_{c}}= \sum_{j}(\frac{\partial \mathcal{L}}{\partial p_{j}}\frac{\partial \mathcal{p_{j}}}{\partial z_{k}})\\ =\sum_{j=/k}(\frac{\partial \mathcal{L}}{\partial p_{j}}\frac{\partial \mathcal{p_{j}}}{\partial z_{k}})+(\frac{\partial \mathcal{L}}{\partial p_{k}}\frac{\partial \mathcal{p_{k}}}{\partial z_{k}})\\ =\sum_{j=/k}(-\frac{q_{j}}{p_{j}}*-p_{j}p_{k})+(-\frac{q_{k}}{p_{k}}*p_{k}(1-p_{k}))\\ =\sum_{j=/k}(q_{j}p_{k})+q_{k}p_{k}-q_{k}\\ =\sum_{j}q_{j}p_{k}-q_{k}$