首先對於d維的隨機變量x ∈ R d × 1 \displaystyle \mathbf{x} \in R^{d\times 1} x ∈ R d × 1 s ∈ R d × 1 \displaystyle \mathbf{s} \in R^{d\times 1} s ∈ R d × 1 A ∈ R d × d \displaystyle A\in R^{d\times d} A ∈ R d × d
x = A s
\mathbf{x} =\mathbf{As}
x = A s
求解ICA的第一步是對x白化得到互不相關的列x ~ = E D − 1 / 2 E T x \displaystyle \tilde{\mathbf{x}} =\mathbf{ED}^{-1/2}\mathbf{E}^{T}\mathbf{x} x ~ = E D − 1 / 2 E T x
x ~ = E D − 1 / 2 E T A s = A ~ s
\tilde{\mathbf{x}} =\mathbf{ED}^{-1/2}\mathbf{E}^{T}\mathbf{As} =\tilde{\mathbf{A}}\mathbf{s}
x ~ = E D − 1 / 2 E T A s = A ~ s
其中EDE其實的協方差x的特徵值分解，即x x T = A A T = E D E T \displaystyle \mathbf{xx}^{T} =\mathbf{AA}^{T} =\mathbf{EDE}^{T} x x T = A A T = E D E T A ~ \displaystyle \tilde{\mathbf{A}} A ~ E ( x ~ x ~ T ) = I \displaystyle E\left(\mathbf{\tilde{\mathbf{x}}}\tilde{\mathbf{x}}^{T}\right) =I E ( x ~ x ~ T ) = I
x ~ x ~ T = E D − 1 / 2 E T A s s T A T E D − 1 / 2 E T = E D − 1 / 2 E T E D E T E D − 1 / 2 E T = I
\tilde{\mathbf{x}}\tilde{\mathbf{x}}^{T} =\mathbf{ED}^{-1/2}\mathbf{E}^{T}\mathbf{Ass}^{T}\mathbf{A}^{T}\mathbf{ED^{-1/2} E}^{T} =\mathbf{ED}^{-1/2}\mathbf{E}^{T}\mathbf{EDE^{T} ED^{-1/2} E}^{T} =I
x ~ x ~ T = E D − 1 / 2 E T A s s T A T E D − 1 / 2 E T = E D − 1 / 2 E T E D E T E D − 1 / 2 E T = I
這個白化使得我們需要學習的參數減少了一半，爲什麼呢，因爲正交矩陣的自由度是n ( n − 1 ) 2 \displaystyle \frac{n( n-1)}{2} 2 n ( n − 1 ) n 2 \displaystyle n^{2} n 2
接來下開始就是ICA了，如果s的服從高斯分佈的，那麼故事結束，我們不能恢復出唯一的s，因爲不管哪個方向都是等價的。而如果s是非高斯的，那麼ICA要做的就是就是要找到一個最優方向w \displaystyle \mathbf{w} w
max J ( w T x )
\max J\ \left(\mathbf{w}^{T}\mathbf{x}\right)
max J ( w T x )
因此，這只是一個優化問題。那麼怎麼衡量非高斯性呢，我們可以用負熵來衡量：
J ( y ) = H ( y g a u s s ) − H ( y )
J(\mathbf{y} )=H(\mathbf{y}_{gauss}) -H(\mathbf{y} )
J ( y ) = H ( y g a u s s ) − H ( y )
其中y g a u s s \displaystyle \mathbf{y}_{gauss} y g a u s s y \displaystyle y y
J ( y ) ≈ 1 12 E { y 3 } 2 + 1 48 kurt ( y ) 2
J(y)\approx \frac{1}{12} E\left\{y^{3}\right\}^{2} +\frac{1}{48}\operatorname{kurt} (y)^{2}
J ( y ) ≈ 1 2 1 E { y 3 } 2 + 4 8 1 k u r t ( y ) 2
而在fastICA中，使用的是：
J G ( w ) = [ E { G ( w T x ) } − E { G ( ν ) } ] 2
J_{G} (\mathbf{w} )=\left[ E\left\{G\left(\mathbf{w}^{T}\mathbf{x}\right)\right\} -E\{G(\nu )\}\right]^{2}
J G ( w ) = [ E { G ( w T x ) } − E { G ( ν ) } ] 2
其中E ( ( w T x ) 2 ) = E ( w T x x T w ) = E ( w T w ) = ∥ w ∥ 2 = 1 \displaystyle E\left(\left(\mathbf{w}^{T}\mathbf{x}\right)^{2}\right) =E\left(\mathbf{w}^{T}\mathbf{xx}^{T}\mathbf{w}\right) =E\left(\mathbf{w}^{T}\mathbf{w}\right) =\| \mathbf{w} \| ^{2} =1 E ( ( w T x ) 2 ) = E ( w T x x T w ) = E ( w T w ) = ∥ w ∥ 2 = 1
∂ J G ( w ) − β ( ∥ w ∥ 2 − 1 ) ∂ w = 2 E { x G ′ ( w T x ) } − 2 β w = 0
\frac{\partial J_{G} (\mathbf{w} )-\beta \left( \| \mathbf{w} \| ^{2} -1\right)}{\partial \mathbf{w}} =2E\left\{\mathbf{x} G'\left(\mathbf{w}^{T}\mathbf{x}\right)\right\} -2\beta \mathbf{w} =0
∂ w ∂ J G ( w ) − β ( ∥ w ∥ 2 − 1 ) = 2 E { x G ′ ( w T x ) } − 2 β w = 0
當這個方程等於0的時候就得到最優的w了。此外β \displaystyle \beta β w 0 \displaystyle w_{0} w 0 E { x G ′ ( w 0 T x ) } − β w o = 0 ⟹ β = E { w 0 T x G ′ ( w 0 T x ) } \displaystyle E\left\{\mathbf{x} G'\left(\mathbf{w}^{T}_{0}\mathbf{x}\right)\right\} -\beta \mathbf{w}_{o} =0\Longrightarrow \beta =E\left\{\mathbf{w^{T}_{0} x} G'\left(\mathbf{w}^{T}_{0}\mathbf{x}\right)\right\} E { x G ′ ( w 0 T x ) } − β w o = 0 ⟹ β = E { w 0 T x G ′ ( w 0 T x ) }
x n + 1 = x n − f ( x n ) f ′ ( x n )
x_{n+1} =x_{n} -\frac{f( x_{n})}{f^{\prime }( x_{n})}
x n + 1 = x n − f ′ ( x n ) f ( x n )
簡單地介紹一下，我們知道方程的根就是他跟x軸的交點，那如果我們用切線來近似曲線，那麼可以“認爲“切線與x軸的交點就是根，如果不是的話，我們可以一直重複這個流程直到收斂。
∂ E { x G ′ ( w T x ) } − β w ∂ w = E { x x T G ′ ′ ( w T x ) } − β I
\frac{\partial E\left\{\mathbf{x} G'\left(\mathbf{w}^{T}\mathbf{x}\right)\right\} -\beta \mathbf{w}}{\partial w} =E\left\{\mathbf{xx}^{T} G''\left(\mathbf{w}^{T}\mathbf{x}\right)\right\} -\beta \mathbf{I}
∂ w ∂ E { x G ′ ( w T x ) } − β w = E { x x T G ′ ′ ( w T x ) } − β I
其中用一個近似的方法E { x x T G ′ ′ ( w T x ) } ≈ E { x x T } E { G ′ ′ ( w T x ) } = E { G ′ ′ ( w T x ) } I \displaystyle E\left\{\mathbf{xx}^{T} G''\left(\mathbf{w}^{T}\mathbf{x}\right)\right\} \approx E\left\{\mathbf{xx}^{T}\right\} E\left\{G''\left(\mathbf{w}^{T}\mathbf{x}\right)\right\} =E\left\{G''\left(\mathbf{w}^{T}\mathbf{x}\right)\right\}\mathbf{I} E { x x T G ′ ′ ( w T x ) } ≈ E { x x T } E { G ′ ′ ( w T x ) } = E { G ′ ′ ( w T x ) } I
w n + 1 = w n − E { x G ′ ( w n T x ) } − β w n E { G ′ ′ ( w n T x ) } − β I
\mathbf{w}_{n+1} =\mathbf{w}_{n} -\frac{E\left\{\mathbf{x} G'\left(\mathbf{w}_{n}^{T}\mathbf{x}\right)\right\} -\beta \mathbf{w}_{n}}{E\left\{G''\left(\mathbf{w}^{T}_{n}\mathbf{x}\right)\right\} -\beta \mathbf{I}}
w n + 1 = w n − E { G ′ ′ ( w n T x ) } − β I E { x G ′ ( w n T x ) } − β w n
我們用w n \displaystyle \mathbf{w}_{n} w n w 0 \displaystyle \mathbf{w}_{0} w 0 β \displaystyle \beta β
w n + 1 = E { G ′ ′ ( w n T x ) } w n − E { x G ′ ( w n T x ) } E { G ′ ′ ( w n T x ) } − β
\mathbf{w}_{n+1} =\frac{\mathbf{E\left\{G''\left( w^{T}_{n} x\right)\right\} w_{n}} -E\left\{\mathbf{x} G'\left(\mathbf{w}_{n}^{T}\mathbf{x}\right)\right\}}{E\left\{G''\left(\mathbf{w}^{T}_{n}\mathbf{x}\right)\right\} -\beta }
w n + 1 = E { G ′ ′ ( w n T x ) } − β E { G ′ ′ ( w n T x ) } w n − E { x G ′ ( w n T x ) }
最後兩邊同時乘以β − E { G ′ ′ ( w n T x ) \displaystyle \beta -E\{G''\left(\mathbf{w}^{T}_{n}\mathbf{x}\right) β − E { G ′ ′ ( w n T x )
w n + 1 = E { x G ′ ( w n T x ) } − E { G ′ ′ ( w n T x ) } w n w n + 1 = w n / ∥ w n ∥
\mathbf{w}_{n+1} =E\left\{\mathbf{x} G'\left(\mathbf{w}_{n}^{T}\mathbf{x}\right)\right\} -\mathbf{E\left\{G''\left( w^{T}_{n} x\right)\right\} w_{n}}\\
\mathbf{w}_{n+1} =\mathbf{w}_{n} /\| \mathbf{w}_{n} \|
w n + 1 = E { x G ′ ( w n T x ) } − E { G ′ ′ ( w n T x ) } w n w n + 1 = w n / ∥ w n ∥
這個就是fastICA所使用的迭代公式了。
最後，這個G到底是什麼，常用的G:
G 1 ( u ) = 1 a 1 log cosh ( a 1 u ) G 1 ′ ( u ) = tanh ( a 1 u ) G 2 ( u ) = − 1 a 2 exp ( − a 2 u 2 / 2 ) G 2 ′ ( u ) = u exp ( − a 2 u 2 / 2 ) G 3 ( u ) = 1 4 u 4 G 3 ′ ( u ) = u 3
\begin{aligned}
G _ { 1 } ( u ) & = \frac { 1 } { a _ { 1 } } \log \cosh \left( a _ { 1 } u \right) \\
G' _ { 1 } ( u ) & = \tanh \left( a _ { 1 } u \right) \\
G _ { 2 } ( u ) & = - \frac { 1 } { a _ { 2 } } \exp \left( - a _ { 2 } u ^ { 2 } / 2 \right) \\
G' _ { 2 } ( u ) & = u \exp \left( - a _ { 2 } u ^ { 2 } / 2 \right) \\
G _ { 3 } ( u ) & = \frac { 1 } { 4 } u ^ { 4 } \\
G' _ { 3 } ( u ) & = u ^ { 3 }
\end{aligned}
G 1 ( u ) G 1 ′ ( u ) G 2 ( u ) G 2 ′ ( u ) G 3 ( u ) G 3 ′ ( u ) = a 1 1 log cosh ( a 1 u ) = tanh ( a 1 u ) = − a 2 1 exp ( − a 2 u 2 / 2 ) = u exp ( − a 2 u 2 / 2 ) = 4 1 u 4 = u 3
其中1 ≥ a 1 ≤ 2 , a 2 ≈ 1 1\ge a_1 \le2,a_2\approx1 1 ≥ a 1 ≤ 2 , a 2 ≈ 1
G 1 G_1 G 1 如果獨立成分s是非常超高斯的，那麼用G 2 G_2 G 2
如果獨立成分是sub-gaussian ，且沒有異常值，則用G 3 G_3 G 3
如果要減少計算量，那麼或許可以用G 1 G_1 G 1 G 2 G_2 G 2
[1] HYVÄRINEN A, OJA E. Independent component analysis: algorithms and applications[J]. Neural networks, Elsevier, 2000, 13(4–5): 411–430.
[2] HYVARINEN A. Fast and robust fixed-point algorithms for independent component analysis[J]. IEEE transactions on Neural Networks, IEEE, 1999, 10(3): 626–634.
