標準方程的證明

標準方程的證明

線性迴歸模型公式(第i個實例的預測值$\hat{y_i}$):
$\hat{y_i}=\theta_0+\theta_1 x_{i,1}+\theta_2 x_{i,2} + ... + \theta_n x_{i,n}$
轉化成矩陣:
$\hat{y_i}= \begin{bmatrix} 1 & x_{i,1} & x_{i,2} & \cdots & x_{i,n} \end{bmatrix} \begin{bmatrix} \theta_0 \\ \theta_1 \\ \theta_2 \\ \vdots \\ \theta_n \end{bmatrix}$
簡化爲:
$\hat{y_i}=\mathbf{x_i}^{T}\theta$
誤差公式爲:
$MSE(\mathbf{\theta})=\frac{1}{m} \sum_{i=1}^m (\hat{y}_i-y_i)^{2} =\frac{1}{m}\sum_{i=1}^{m}(\mathbf{x_i}^{T} \mathbf{\theta} -y_i)^{2}$
設:
$\mathbf c = \begin{bmatrix} \mathbf{x_1}^{T} \mathbf{\theta} -y_1 \\ \mathbf{x_2}^{T} \mathbf{\theta} -y_2 \\ \vdots \\ \mathbf{x_m}^{T} \mathbf{\theta} -y_m \\ \end{bmatrix}= \begin{bmatrix} \mathbf{x_1}^{T} \mathbf{\theta} \\ \mathbf{x_2}^{T} \mathbf{\theta} \\ \vdots \\ \mathbf{x_m}^{T} \mathbf{\theta} \\ \end{bmatrix}- \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_m \end{bmatrix}= \begin{bmatrix} \mathbf{x_1}^{T} \\ \mathbf{x_2}^{T} \\ \vdots \\ \mathbf{x_m}^{T} \\ \end{bmatrix} \mathbf{\theta} -\mathbf{y} =\mathbf{X}\mathbf{\theta}-\mathbf{y}$

則:
$MSE(\mathbf{\theta})=\frac{1}{m} \left \| \mathbf{c} \right \|^{2} =\frac{1}{m} \left \| \mathbf{X}\mathbf{\theta}-\mathbf{y} \right \|^{2}$

$MSE(\mathbf{\theta})$要取到最小值,則對$MSE(\mathbf{\theta})=MSE(\theta_0,\theta_1,\cdots,\theta_n)=E$,相當於求解該多變量函數梯度爲0的點,梯度向量爲E函數對$\mathbf{\theta}$的偏導數:
$\frac{\partial{E}}{\partial{\mathbf{\theta}}}= \begin{bmatrix} \frac{\partial{E}}{\partial{\theta_0}} & \frac{\partial{E}}{\partial{\theta_1}} & \cdots & \frac{\partial{E}}{\partial{\theta_n}} & \end{bmatrix}$
由矩陣的求導法則及下一節證明出的公式可證:

設$g(\mathbf \theta)=\mathbf X \mathbf \theta - \mathbf y=\mathbf u$,則
$f(\mathbf u)=MSE(\mathbf \theta)=\frac{1}{m}\left\| g(\mathbf \theta) \right\|^2=\frac{1}{m} \left\| \mathbf u \right\|^2$
$\frac{\partial MSE(\mathbf \theta)}{\partial \mathbf \theta}=\frac{\partial f(\mathbf u)}{\partial \mathbf \theta}=\frac{\partial f(\mathbf u)}{\partial \mathbf u} \frac{\partial \mathbf u}{\partial \mathbf \theta}=\frac{\partial \frac{1}{m} \left\| \mathbf u \right\|^2}{\partial \mathbf u} \frac{\partial \mathbf X \mathbf \theta - \mathbf y}{\partial \mathbf \theta} =\frac{1}{m}\frac{\partial \mathbf u^T\mathbf u}{\partial \mathbf u}\mathbf X=\frac{2}{m}\mathbf u^T\mathbf X$

則求解梯度全爲0時$\mathbf \theta$的值$\hat{\mathbf \theta}$:
$\frac{2}{m}\left( \mathbf X\hat{\mathbf \theta}-\mathbf y \right)^T\mathbf X=\mathbf 0$
$\hat{\mathbf \theta}^T \mathbf X^T \mathbf X-\mathbf y^T \mathbf X=\mathbf 0$
$\hat{\mathbf \theta}^T=\mathbf y^T\mathbf X\left( \mathbf X^T\mathbf X \right)^{-1}$
$\hat{\mathbf \theta}=\left( \mathbf X^T\mathbf X \right)^{-1}\mathbf X^T\mathbf y$

本質上來說是矩陣求導的應用，特殊多項式求最小值,該計算涉及到求逆操作，對n×n矩陣的求逆的計算複雜度通常爲$O(n^{2.4})$到$O(n^{3})$之間，當特徵數量比較大時(例如100000時),標準方程的計算會極其緩慢