手推邏輯斯蒂迴歸——以向量形式

LR的決策函數爲

$h(\boldsymbol x)=\sigma(\boldsymbol \theta^T \boldsymbol x)=\frac{1}{1+e^{-\boldsymbol \theta^T \boldsymbol x}} \tag1$

其中$\sigma(z)=\frac 1{1+e^{-z}}$，稱爲sigmoid函數

設$h(\boldsymbol x)$表示該樣本爲正例的概率，將其視爲類後驗概率估計$p(y=1|\boldsymbol x;\boldsymbol \theta)$，則：

$p(y=1|\boldsymbol x;\boldsymbol \theta)=h (\boldsymbol x) \tag2$

$p(y=0|\boldsymbol x;\boldsymbol \theta)=1-h (\boldsymbol x) \tag3$

合併式$(2)(3)$得到

$p(y|\boldsymbol x;\boldsymbol \theta)=h (\boldsymbol x)^y(1-h(\boldsymbol x))^{1-y} \tag4$

我們可以使用極大似然估計來得到參數$\theta$，似然函數爲

$L(\boldsymbol \theta)=\prod_{i=1}^mp(y^{(i)}|\boldsymbol x^{(i)};\boldsymbol \theta)=\prod_{i=1}^m h(\boldsymbol x^{(i)})^{y^{(i)}} (1-h(\boldsymbol x^{(i)}))^{1-y^{(i)}} \tag5$

其中$m$爲數據集的樣本個數.

由於取對數不影響單調性且可以避免一些數值問題，取對數可得

$\log L(\boldsymbol \theta)= \sum_{i=1}^m y^{(i)}\log(h(\boldsymbol x^{(i)})) + (1-y^{(i)})\log(1-h(\boldsymbol x^{(i)})) \tag6$

最大化式$(6)$等價於最小化下列損失函數，剛好就是交叉熵損失函數：

$J(\boldsymbol \theta)= -\frac1m\sum_{i=1}^m y^{(i)}\log(h(\boldsymbol x^{(i)})) + (1-y^{(i)})\log(1-h(\boldsymbol x^{(i)})) \tag7$

爲推導簡便，令$J_i$表示$J(\theta)$的第$i$項，對應了第$i$個樣本，即

$J(\boldsymbol \theta)= -\frac1m\sum_{i=1}^m J_i(\boldsymbol \theta) \tag8$

$J_i(\boldsymbol \theta)=y^{(i)}\log(h(\boldsymbol x^{(i)})) + (1-y^{(i)})\log(1-h(\boldsymbol x^{(i)})) \tag{9}$

下面先推導出$\frac{\partial J_i}{\partial \boldsymbol \theta}$，省略$J_i$表達式中$\boldsymbol x^{(i)}$、$y^{(i)}$和$h^{(i)}$的上標$(i)$，有：

$\begin{aligned} \frac{\partial J_i(\boldsymbol \theta)}{\partial \boldsymbol \theta} &=y\frac{\partial \log h}{\partial \boldsymbol \theta} + (1-y)\frac{\partial \log (1-h)}{\partial \boldsymbol \theta} \\ &=\frac yh \frac{\partial h}{\partial \boldsymbol \theta} +\frac{ (1-y)}{(1-h)}\frac{\partial(1-h)}{\partial \boldsymbol \theta} \\ &=\frac{y-h}{h(1-h)} \frac{\partial h}{\partial \boldsymbol \theta} \\ &=\frac{y-h}{h(1-h)} \frac{\partial \sigma(z)}{\partial \boldsymbol \theta}\\ &=\frac{y-h}{h(1-h)} \frac{\partial \sigma(z)}{\partial z} \frac{\partial z}{\partial \boldsymbol \theta}\\ &=\frac{y-h}{h(1-h)} h(1-h) \frac{\partial \boldsymbol \theta^T \boldsymbol x}{\partial \boldsymbol \theta}\\ &=(y-h)\boldsymbol x\\ \end{aligned}$

補好上標$(i)$則是：

$\frac{\partial J_i}{\partial \boldsymbol \theta}=(y^{(i)}-h^{(i)})\boldsymbol x^{(i)}\tag{10}$

由式$(8)$和式$(10)$得

$\frac{\partial J}{\partial \boldsymbol \theta}=-\frac1m\sum_{i=1}^m \frac{\partial J_i}{\partial \boldsymbol \theta}=\frac1m\sum_{i=1}^m (h^{(i)}-y^{(i)})\boldsymbol x^{(i)} \tag{11}$

故梯度更新式爲$\boldsymbol \theta \leftarrow \boldsymbol \theta-\alpha \frac 1 m\sum_{i=1}^m (h^{(i)}-y^{(i)})\boldsymbol x^{(i)} \tag{12}$

References:
[1] 機器學習 3.3節. 周志華