Softmax Derivation

討論最簡單情況：

     以神經網絡爲例：

     假設在softmax層，輸入的數據是N維的一維數組，輸出結果是多分類的各個概率，假設爲C類。

--1. input: x --> import data with dimension N, can be writen as ,  in neural network, means the last hidden layer output.

               W, b --> the affine weight with shape:(N, C) and (C, )

                 y --> the target label of the data, the value of y is in (0, C-1). I will transform y to be an one-hot vector. In the form .

--2: Derivation

      定義隱層輸出爲.

     定義損失函數：

                                                        

     求反向傳播參數dx, dW:

     這裏我把它變成求中間變量dz, 然後用dz 推導dx, dW.

                                                   

                                                    

                                                     

                                                    

So:

                                                     

where:

                                                    

                                                   

改寫成：

                                                   , one-hot 性質決定了結果中實際只有一個值有效。因此這個形式可以化簡成其中爲1的那項。

                                                  

同理可以求出其他z,然後根據z求出dx, dW.

                歡迎討論！