Introduction of Machine Learning
Liner Regression with one Variable
Liner Algebra Review
機器學習：一臺計算機，從 E 中學習實現 T ，用 P 衡量
Example:
監督學習：給定待訓練數據集（帶正確輸出結果的label，即知道這些數據集的輸出結果 ）Regression :continuous function .（連續函數）Classification :discrete categories .（離散範疇；yes or no）
Example: whether the tumor is maligant or benign.
無監督學習：給定待訓練數據集（不知道這些數據集的輸出結果 ）
Notation:
input variables: x ( i ) x^{(i)} x ( i ) y ( i ) y^{(i)} y ( i )
a training example: ( x ( i ) , y ( i ) ) (x^{(i)} ,y^{(i)}) ( x ( i ) , y ( i ) ) m m m ( x ( i ) , y ( i ) ) ; i = 1 , . . . , m (x^{(i)} ,y^{(i)}); i=1,...,m ( x ( i ) , y ( i ) ) ; i = 1 , . . . , m
the space of input values: X X X x ( i ) x^{(i)} x ( i ) Y Y Y
Example: X = Y = R X=Y=R X = Y = R h h h
Our goal is , given a training set, to learn a function h : X → Y h:X \to Y h : X → Y h ( x ) h(x) h ( x ) y y y
Hypothesis:h θ ( x ) = θ 0 + θ 1 x h_{\theta}(x)=\theta_0 +\theta_1 x h θ ( x ) = θ 0 + θ 1 x h θ ( x ) h_{\theta}(x) h θ ( x ) h ( x ) h(x) h ( x )
Parameters:θ 0 , θ 1 \theta_0,\theta_1 θ 0 , θ 1
Cost Function:J ( θ 0 , θ 1 ) = 1 2 m ∑ i = 1 m ( h θ ( x i ) − y i ) 2 = 1 2 m ∑ i = 1 m ( y i ^ − y i ) 2 J(\theta_0,\theta_1)=\frac{1}{2m} \sum_{i=1}^{m}(h_\theta(x_i)-y_i)^{2}=\frac{1}{2m} \sum_{i=1}^{m}(\hat{y_i}-y_i)^{2} J ( θ 0 , θ 1 ) = 2 m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) 2 = 2 m 1 ∑ i = 1 m ( y i ^ − y i ) 2 1 2 \frac{1}{2} 2 1 ( x 2 ) ′ = 2 x (x^{2})'=2x ( x 2 ) ′ = 2 x
Goal:m i n i m i z e θ 0 , θ 1 J ( θ 0 , θ 1 ) \frac{minimize}{\theta_0,\theta_1}J(\theta_0,\theta_1) θ 0 , θ 1 m i n i m i z e J ( θ 0 , θ 1 )
Idea:θ 0 , θ 1 \theta_0,\theta_1 θ 0 , θ 1 h θ ( x ) h_{\theta}(x) h θ ( x ) ( x , y ) (x,y) ( x , y )
In order to better visualize the cost function J J J θ 0 = 0 \theta_0=0 θ 0 = 0
Hypothesis:h θ ( x ) = θ 1 x h_{\theta}(x)=\theta_1 x h θ ( x ) = θ 1 x
Parameters:θ 1 \theta_1 θ 1
Cost Function:J ( θ 1 ) = 1 2 m ∑ i = 1 m ( h θ ( x i ) − y i ) 2 = 1 2 m ∑ i = 1 m ( y i ^ − y i ) 2 J(\theta_1)=\frac{1}{2m} \sum_{i=1}^{m}(h_\theta(x_i)-y_i)^{2}=\frac{1}{2m} \sum_{i=1}^{m}(\hat{y_i}-y_i)^{2} J ( θ 1 ) = 2 m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) 2 = 2 m 1 ∑ i = 1 m ( y i ^ − y i ) 2
Goal:m i n i m i z e θ 1 J ( θ 1 ) \frac{minimize}{\theta_1}J(\theta_1) θ 1 m i n i m i z e J ( θ 1 )
某個訓練集的J ( θ 1 ) J(\theta_1) J ( θ 1 )
Hypothesis:h θ ( x ) = θ 0 + θ 1 x h_{\theta}(x)=\theta_0 +\theta_1 x h θ ( x ) = θ 0 + θ 1 x
Parameters:θ 0 , θ 1 \theta_0,\theta_1 θ 0 , θ 1
Cost Function:J ( θ 0 , θ 1 ) = 1 2 m ∑ i = 1 m ( h θ ( x i ) − y i ) 2 = 1 2 m ∑ i = 1 m ( y i ^ − y i ) 2 J(\theta_0,\theta_1)=\frac{1}{2m} \sum_{i=1}^{m}(h_\theta(x_i)-y_i)^{2}=\frac{1}{2m} \sum_{i=1}^{m}(\hat{y_i}-y_i)^{2} J ( θ 0 , θ 1 ) = 2 m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) 2 = 2 m 1 ∑ i = 1 m ( y i ^ − y i ) 2
Goal:m i n i m i z e θ 0 , θ 1 J ( θ 0 , θ 1 ) \frac{minimize}{\theta_0,\theta_1}J(\theta_0,\theta_1) θ 0 , θ 1 m i n i m i z e J ( θ 0 , θ 1 )
某個訓練集的J ( θ 0 , θ 1 ) J(\theta_0,\theta_1) J ( θ 0 , θ 1 )
某個訓練集的J ( θ 0 , θ 1 ) J(\theta_0,\theta_1) J ( θ 0 , θ 1 ) J ( θ 0 , θ 1 ) J(\theta_0,\theta_1) J ( θ 0 , θ 1 )
Have some function J ( θ 0 , θ 1 ) J(\theta_0,\theta_1) J ( θ 0 , θ 1 ) m i n θ 0 , θ 1 J ( θ 0 , θ 1 ) \frac{min}{\theta_0,\theta_1}J(\theta_0,\theta_1) θ 0 , θ 1 m i n J ( θ 0 , θ 1 )
Start with some θ 0 , θ 1 \theta_0,\theta_1 θ 0 , θ 1
Keep changing θ 0 , θ 1 \theta_0,\theta_1 θ 0 , θ 1 J ( θ 0 , θ 1 ) J(\theta_0,\theta_1) J ( θ 0 , θ 1 )
local optimal：
Gradient descent algorithm θ j : = θ j − α ∂ ∂ θ j J ( θ 0 , θ 1 ) \theta_j:=\theta_j - \alpha \frac{\partial}{\partial\theta_j}
J(\theta_0,\theta_1) θ j : = θ j − α ∂ θ j ∂ J ( θ 0 , θ 1 ) notation: α \alpha α θ j \theta_j θ j
Corret: simultaneous update θ 0 , θ 1 \theta_0,\theta_1 θ 0 , θ 1 t e m p 0 : = θ 0 − α ∂ ∂ θ 0 J ( θ 0 , θ 1 ) temp0:=\theta_0 - \alpha \frac{\partial}{\partial\theta_0}
J(\theta_0,\theta_1) t e m p 0 : = θ 0 − α ∂ θ 0 ∂ J ( θ 0 , θ 1 ) t e m p 1 : = θ 1 − α ∂ ∂ θ 1 J ( θ 0 , θ 1 ) temp1:=\theta_1 - \alpha \frac{\partial}{\partial\theta_1}
J(\theta_0,\theta_1) t e m p 1 : = θ 1 − α ∂ θ 1 ∂ J ( θ 0 , θ 1 ) θ 0 : = t e m p 0 \theta_0:=temp0 θ 0 : = t e m p 0 θ 1 : = t e m p 1 \theta_1:=temp1 θ 1 : = t e m p 1
在J ( θ 1 ) J(\theta_1) J ( θ 1 ) d d θ 1 J ( θ 1 ) \frac{d}{d\theta_1}J(\theta_1) d θ 1 d J ( θ 1 )
α \alpha α
When α \alpha α , gradient descent can converge to a lcoal minimum, even with the learning rate α \alpha α ∂ ∂ θ j J ( θ 0 , θ 1 ) \frac{\partial}{\partial\theta_j}J(\theta_0,\theta_1) ∂ θ j ∂ J ( θ 0 , θ 1 ) automatically take smaller steps （邁的步子越來越小）. So, no need to decrease α \alpha α
Hypothesis:h θ ( x ) = θ 0 + θ 1 x h_{\theta}(x)=\theta_0 +\theta_1 x h θ ( x ) = θ 0 + θ 1 x
Cost function:J ( θ 0 , θ 1 ) = 1 2 m ∑ i = 1 m ( h θ ( x i ) − y i ) 2 J(\theta_0,\theta_1)=\frac{1}{2m} \sum_{i=1}^{m}(h_\theta(x_i)-y_i)^{2} J ( θ 0 , θ 1 ) = 2 m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) 2 = 1 2 m ∑ i = 1 m ( θ 0 + θ 1 x i − y i ) 2 =\frac{1}{2m} \sum_{i=1}^{m}(\theta_0 +\theta_1 x_i-y_i)^{2} = 2 m 1 ∑ i = 1 m ( θ 0 + θ 1 x i − y i ) 2
Gradient descent:θ j : = θ j − α ∂ ∂ θ j J ( θ 0 , θ 1 ) \theta_j:=\theta_j - \alpha \frac{\partial}{\partial\theta_j}
J(\theta_0,\theta_1) θ j : = θ j − α ∂ θ j ∂ J ( θ 0 , θ 1 )
結合爲：j = 0 , ∂ ∂ θ 0 J ( θ 0 , θ 1 ) = 1 m ∑ i = 1 m ( h θ ( x i ) − y i ) j=0, \frac{\partial}{\partial\theta_0}J(\theta_0,\theta_1)
=\frac{1}{m}\sum_{i=1}^{m}(h_\theta(x_i)-y_i) j = 0 , ∂ θ 0 ∂ J ( θ 0 , θ 1 ) = m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) j = 1 , ∂ ∂ θ 1 J ( θ 0 , θ 1 ) = 1 m ∑ i = 1 m ( h θ ( x i ) − y i ) x i j=1, \frac{\partial}{\partial\theta_1}J(\theta_0,\theta_1)
=\frac{1}{m}\sum_{i=1}^{m}(h_\theta(x_i)-y_i)x_i j = 1 , ∂ θ 1 ∂ J ( θ 0 , θ 1 ) = m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) x i
repeat until convergence{simultaneous update:θ 0 : = θ 0 − α 1 m ∑ i = 1 m ( h θ ( x i ) − y i ) \theta_0:=\theta_0 - \alpha
\frac{1}{m}\sum_{i=1}^{m}(h_\theta(x_i)-y_i) θ 0 : = θ 0 − α m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) θ 1 : = θ 1 − α 1 m ∑ i = 1 m ( h θ ( x i ) − y i ) x i \theta_1:=\theta_1 - \alpha \frac{1}{m}\sum_{i=1}^{m}(h_\theta(x_i)-y_i)x_i θ 1 : = θ 1 − α m 1 ∑ i = 1 m ( h θ ( x i ) − y i ) x i
Batch: Each step of gradient descent uses all the training examples.（1 m ∑ i = 1 m ( h θ ( x i ) − y i ) \frac{1}{m}\sum_{i=1}^{m}(h_\theta(x_i)-y_i) m 1 ∑ i = 1 m ( h θ ( x i ) − y i )
Matrix: （大寫表示）4 × 2 4\times2 4 × 2 R 4 × 2 R^{4\times2} R 4 × 2 A i j A_{ij} A i j i t h i^{th} i t h j t h j^{th} j t h
Vector: An n × 1 n\times1 n × 1 y = [ 460 235 315 178 ] y=\begin{bmatrix}
460 \\
235 \\
315\\
178
\end{bmatrix} y = ⎣ ⎢ ⎢ ⎡ 4 6 0 2 3 5 3 1 5 1 7 8 ⎦ ⎥ ⎥ ⎤ R 4 R^{4} R 4 y i = i t h y_i = i^{th} y i = i t h y = [ y 1 y 2 y 3 y 4 ] y=\begin{bmatrix}
y_1 \\
y_2 \\
y_3\\
y_4
\end{bmatrix} y = ⎣ ⎢ ⎢ ⎡ y 1 y 2 y 3 y 4 ⎦ ⎥ ⎥ ⎤
0-indexed: y = [ y 0 y 1 y 2 y 3 ] y=\begin{bmatrix}
y_0 \\
y_1 \\
y_2\\
y_3
\end{bmatrix} y = ⎣ ⎢ ⎢ ⎡ y 0 y 1 y 2 y 3 ⎦ ⎥ ⎥ ⎤
matrix addition:…× \times ×
Conditions :h θ ( x ) = − 40 + 0.25 x x = [ 2104 1416 1534 852 ] h_\theta(x)=-40+0.25x\\
x = \begin{bmatrix}
2104 \\
1416 \\
1534\\
852
\end{bmatrix} h θ ( x ) = − 4 0 + 0 . 2 5 x x = ⎣ ⎢ ⎢ ⎡ 2 1 0 4 1 4 1 6 1 5 3 4 8 5 2 ⎦ ⎥ ⎥ ⎤
Result: p r e d i c t i o n = D a t a M a t r i x × p a r a m e t e r s = [ 1 , 2104 1 , 1416 1 , 1534 1 , 852 ] × [ − 40 0.25 ] prediction = DataMatrix \times parameters\\
= \begin{bmatrix}
1,2104 \\
1,1416 \\
1,1534\\
1, 852
\end{bmatrix} \times \begin{bmatrix}
-40\\
0.25
\end{bmatrix} p r e d i c t i o n = D a t a M a t r i x × p a r a m e t e r s = ⎣ ⎢ ⎢ ⎡ 1 , 2 1 0 4 1 , 1 4 1 6 1 , 1 5 3 4 1 , 8 5 2 ⎦ ⎥ ⎥ ⎤ × [ − 4 0 0 . 2 5 ]
不僅在octave，在高級語言中，矩陣運算 比用 for 循環計算代碼少，效率高 。
x = [2014, 1416, 1534, 852]
for i in range(4):
prediction[i] = -40 + 0.25*x
Conditions :1. h θ ( x ) = − 40 + 0.25 x 2. h θ ( x ) = 200 + 0.1 x 3. h θ ( x ) = − 150 + 0.4 x x = [ 2104 1416 1534 852 ] 1. h_\theta(x)=-40+0.25x\\
2. h_\theta(x)=200+0.1x\\
3. h_\theta(x)=-150+0.4x\\
x = \begin{bmatrix}
2104 \\
1416 \\
1534\\
852
\end{bmatrix} 1 . h θ ( x ) = − 4 0 + 0 . 2 5 x 2 . h θ ( x ) = 2 0 0 + 0 . 1 x 3 . h θ ( x ) = − 1 5 0 + 0 . 4 x x = ⎣ ⎢ ⎢ ⎡ 2 1 0 4 1 4 1 6 1 5 3 4 8 5 2 ⎦ ⎥ ⎥ ⎤
Result: P r e d i c t i o n M a t r i x = D a t a M a t r i x × P a r a m e t e r s M a t r i x = [ 1 , 2104 1 , 1416 1 , 1534 1 , 852 ] × [ − 40 , 200 , − 150 0.25 , 0.1 , 0.4 ] = [ 486 , 410 , 692 314 , 342 , 416 344 , 353 , 464 173 , 285 , 191 ] PredictionMatrix = DataMatrix \times ParametersMatrix\\
= \begin{bmatrix}
1,2104 \\
1,1416 \\
1,1534\\
1, 852
\end{bmatrix} \times \begin{bmatrix}
-40,200, -150\\
0.25,0.1,0.4
\end{bmatrix}\\
=\begin{bmatrix}
486,410,692\\
314, 342, 416\\
344, 353, 464\\
173, 285, 191
\end{bmatrix} P r e d i c t i o n M a t r i x = D a t a M a t r i x × P a r a m e t e r s M a t r i x = ⎣ ⎢ ⎢ ⎡ 1 , 2 1 0 4 1 , 1 4 1 6 1 , 1 5 3 4 1 , 8 5 2 ⎦ ⎥ ⎥ ⎤ × [ − 4 0 , 2 0 0 , − 1 5 0 0 . 2 5 , 0 . 1 , 0 . 4 ] = ⎣ ⎢ ⎢ ⎡ 4 8 6 , 4 1 0 , 6 9 2 3 1 4 , 3 4 2 , 4 1 6 3 4 4 , 3 5 3 , 4 6 4 1 7 3 , 2 8 5 , 1 9 1 ⎦ ⎥ ⎥ ⎤ j j j j j j
Matrix × \times × I n × n I_{n\times n} I n × n
Matrix Inverse：…（逆）
