給定n n n d d d x 1 , x 2 , … , x n x_1,x_2, \dots,x_n x 1 , x 2 , … , x n 降維的標準是降維後數據點之間的距離d i j ( Y ) d_{ij}^{(Y)} d i j ( Y ) d i j ( X ) d_{ij}^{(X)} d i j ( X )
X = [ x 1 x 2 … x n ] d ∗ n , x i ∈ R d X=\left[ \begin{matrix} x_1 & x_2 & \dots & x_n\end{matrix} \right]_{d*n}, x_i \in R^d X = [ x 1 x 2 … x n ] d ∗ n , x i ∈ R d Y = [ y 1 y 2 … y n ] p ∗ n , y i ∈ R p , p < d Y=\left[ \begin{matrix} y_1 & y_2 & \dots & y_n\end{matrix}\right]_{p*n}, y_i \in R^p, p < d Y = [ y 1 y 2 … y n ] p ∗ n , y i ∈ R p , p < d
用數學表達是就是min Y ∑ i = 1 n ∑ j = 1 n ( d i j ( X ) − d i j ( Y ) ) 2 \min_Y \sum_{i=1}^n \sum_{j=1}^n (d_{ij}^{(X)}-d_{ij}^{(Y)})^2 Y min i = 1 ∑ n j = 1 ∑ n ( d i j ( X ) − d i j ( Y ) ) 2
距離矩陣D = [ d 11 d 12 … d 1 n d 21 d 22 … d 2 n ⋮ ⋮ ⋮ d n 1 d n 2 … d n n ] D=\left[\begin{matrix} d_{11} & d_{12} & \dots & d_{1n} \\ d_{21} & d_{22} & \dots & d_{2n} \\ \vdots & \vdots & \vdots \\ d_{n1} & d_{n2} & \dots & d_{nn}\end{matrix}\right] D = ⎣ ⎢ ⎢ ⎢ ⎡ d 1 1 d 2 1 ⋮ d n 1 d 1 2 d 2 2 ⋮ d n 2 … … ⋮ … d 1 n d 2 n d n n ⎦ ⎥ ⎥ ⎥ ⎤
d i j d_{ij} d i j i i i j j j
滑鐵盧大學Ali Ghodsi教授直接從正面推導 的過程我實在看不懂。
可以換個角度看問題，新的數據點只利用了點之間的距離，可以用距離矩陣D D D D D D X X X X X X Y Y Y
利用一箇中間矩陣B = X T X B=X^TX B = X T X D D D B B B X X X D → B → X D\rightarrow B \rightarrow X D → B → X
B = X T X = [ x 1 T x 2 T ⋮ x n T ] [ x 1 x 2 … x n ] = [ x 1 T x 1 x 1 T x 2 … x 1 T x n x 2 T x 1 x 2 T x 2 … x 2 T x n ⋮ ⋮ … ⋮ x n T x 1 x n T x 2 … x n T x n ] B=X^TX=\left[\begin{matrix} x_1^T \\ x_2^T \\ \vdots \\ x_n^T\end{matrix}\right]\left[\begin{matrix} x_1 & x_2 & \dots & x_n \end{matrix}\right]=\left[\begin{matrix} x_1^Tx_1 & x_1^Tx_2 & \dots &x_1^Tx_n \\ x_2^Tx_1 & x_2^Tx_2 & \dots & x_2^Tx_n \\ \vdots & \vdots & \dots & \vdots \\x_n^Tx_1 & x_n^Tx_2 & \dots & x_n^Tx_n \end{matrix}\right] B = X T X = ⎣ ⎢ ⎢ ⎢ ⎡ x 1 T x 2 T ⋮ x n T ⎦ ⎥ ⎥ ⎥ ⎤ [ x 1 x 2 … x n ] = ⎣ ⎢ ⎢ ⎢ ⎡ x 1 T x 1 x 2 T x 1 ⋮ x n T x 1 x 1 T x 2 x 2 T x 2 ⋮ x n T x 2 … … … … x 1 T x n x 2 T x n ⋮ x n T x n ⎦ ⎥ ⎥ ⎥ ⎤
b i j = x i T x j b_{ij}=x_i^Tx_j b i j = x i T x j d i j 2 = ∥ x i − x j ∥ 2 = ( x i − x j ) T ( x i − x j ) = x i T x i + x j T x j − 2 x i x j = b i i + b j j − 2 b i j d_{ij}^2=\left \lVert x_i-x_j\right \rVert^2=(x_i-x_j)^T(x_i-x_j)=x_i^Tx_i+x_j^Tx_j-2x_ix_j=b_{ii}+b_{jj}-2b_{ij} d i j 2 = ∥ x i − x j ∥ 2 = ( x i − x j ) T ( x i − x j ) = x i T x i + x j T x j − 2 x i x j = b i i + b j j − 2 b i j
設想一下，只保持點之間的距離不變，點們不是唯一的，把一堆點進行平移、旋轉、翻轉，是不會影響它們之間的距離的。
所以對X X X ∑ i = 1 n x i = 0 \sum_{i=1}^nx_i=0 ∑ i = 1 n x i = 0
我們要用D D D b i j b_{ij} b i j B B B d i j 2 = b i i + b j j − 2 b i j (1) d_{ij}^2=b_{ii}+b_{jj}-2b_{ij} \tag{1} d i j 2 = b i i + b j j − 2 b i j ( 1 ) B B B T = ∑ i = 1 n b i i T=\sum_{i=1}^nb_{ii} T = ∑ i = 1 n b i i
對( 1 ) (1) ( 1 ) i i i ∑ i = 1 n d i j 2 = ∑ i = 1 n b i i + ∑ i = 1 n b j j − 2 ∑ i = 1 n b i j = T + n b j j − 0 (2.1) \sum_{i=1}^nd_{ij}^2=\sum_{i=1}^nb_{ii} + \sum_{i=1}^nb_{jj}-2\sum_{i=1}^nb_{ij}=T+nb_{jj}-0 \tag{2.1} i = 1 ∑ n d i j 2 = i = 1 ∑ n b i i + i = 1 ∑ n b j j − 2 i = 1 ∑ n b i j = T + n b j j − 0 ( 2 . 1 )
其中∑ i = 1 n b i j = ∑ i = 1 n x i T x j = ( ∑ i = 1 n x i ) T x j = 0 \sum_{i=1}^nb_{ij}=\sum_{i=1}^nx_i^Tx_j=(\sum_{i=1}^nx_i)^Tx_j=0 ∑ i = 1 n b i j = ∑ i = 1 n x i T x j = ( ∑ i = 1 n x i ) T x j = 0 X X X
同理對( 1 ) (1) ( 1 ) ∑ i = 1 n d i j 2 = ∑ i = 1 n b i i + ∑ i = 1 n b j j − 2 ∑ i = 1 n b i j = T + n b j j − 0 (2.2) \sum_{i=1}^nd_{ij}^2=\sum_{i=1}^nb_{ii} + \sum_{i=1}^nb_{jj}-2\sum_{i=1}^nb_{ij}=T+nb_{jj}-0 \tag{2.2} i = 1 ∑ n d i j 2 = i = 1 ∑ n b i i + i = 1 ∑ n b j j − 2 i = 1 ∑ n b i j = T + n b j j − 0 ( 2 . 2 )
對( 1 ) (1) ( 1 ) i , j i,j i , j ∑ i = 1 n ∑ j = 1 n d i j 2 = ∑ i = 1 n ∑ j = 1 n b i i + ∑ i = 1 n ∑ j = 1 n b j j − 2 ∑ i = 1 n ∑ j = 1 n b i j = n T + n T − 0 (2.3) \sum_{i=1}^n \sum_{j=1}^nd_{ij}^2=\sum_{i=1}^n \sum_{j=1}^nb_{ii}+\sum_{i=1}^n \sum_{j=1}^nb_{jj}-2\sum_{i=1}^n \sum_{j=1}^nb_{ij}=nT+nT-0 \tag{2.3} i = 1 ∑ n j = 1 ∑ n d i j 2 = i = 1 ∑ n j = 1 ∑ n b i i + i = 1 ∑ n j = 1 ∑ n b j j − 2 i = 1 ∑ n j = 1 ∑ n b i j = n T + n T − 0 ( 2 . 3 )
由上面四個式子就可以得出b i j b_{ij} b i j b i j = − 1 2 ( d i j 2 − 1 n ∑ i d i j 2 − 1 n ∑ j d i j 2 + 1 n 2 ∑ i ∑ j d i j 2 ) (3) b_{ij}=-\frac{1}{2}(d_{ij}^2-\frac{1}{n}\sum_id_{ij}^2-\frac{1}{n}\sum_jd_{ij}^2+\frac{1}{n^2}\sum_i\sum_jd_{ij}^2) \tag{3} b i j = − 2 1 ( d i j 2 − n 1 i ∑ d i j 2 − n 1 j ∑ d i j 2 + n 2 1 i ∑ j ∑ d i j 2 ) ( 3 )
第一步B B B B = X T X B=X^TX B = X T X B B B B = Q Λ Q T = ( Q Λ 1 2 ) ( Q Λ 1 2 ) T B=Q\Lambda Q^T=(Q\Lambda^{\frac{1}{2}})(Q\Lambda^{\frac{1}{2}})^T B = Q Λ Q T = ( Q Λ 2 1 ) ( Q Λ 2 1 ) T X = Λ 1 2 Q X=\Lambda^\frac{1}{2}Q X = Λ 2 1 Q Q Q Q B B B p p p Λ \Lambda Λ B B B p p p
Ali Ghodsi教授的推導我實在看不明白，不過最後的結論是一樣的。K = − 1 2 H D H = X T X K=-\frac{1}{2}HDH=X^TX K = − 2 1 H D H = X T X B = X T X = − 1 2 H D ′ H B=X^TX=-\frac{1}{2}HD'H B = X T X = − 2 1 H D ′ H d i j ′ = d i j 2 d'_{ij}=d_{ij}^2 d i j ′ = d i j 2
H = I − 1 n e e T = [ 1 − 1 n − 1 n … − 1 n − 1 n 1 − 1 n … − 1 n ⋮ ⋮ … ⋮ − 1 n − 1 n … 1 − 1 n ] n ∗ n H=I-\frac{1}{n}ee^T=\left[\begin{matrix} 1-\frac{1}{n} & -\frac{1}{n} & \dots & -\frac{1}{n} \\ -\frac{1}{n} & 1-\frac{1}{n} & \dots & -\frac{1}{n} \\ \vdots & \vdots & \dots &\vdots \\ -\frac{1}{n} & -\frac{1}{n} & \dots & 1-\frac{1}{n} \end{matrix}\right]_{n*n} H = I − n 1 e e T = ⎣ ⎢ ⎢ ⎢ ⎡ 1 − n 1 − n 1 ⋮ − n 1 − n 1 1 − n 1 ⋮ − n 1 … … … … − n 1 − n 1 ⋮ 1 − n 1 ⎦ ⎥ ⎥ ⎥ ⎤ n ∗ n
− 1 2 H D ′ H = = − 1 2 [ 1 − 1 n − 1 n … − 1 n − 1 n 1 − 1 n … − 1 n ⋮ ⋮ … ⋮ − 1 n − 1 n … 1 − 1 n ] [ d 11 2 d 12 2 … d 1 n 2 d 21 2 d 22 2 … d 2 n 2 ⋮ ⋮ … ⋮ d n 1 2 d n 2 2 … d n n 2 ] [ 1 − 1 n − 1 n … − 1 n − 1 n 1 − 1 n … − 1 n ⋮ ⋮ … ⋮ − 1 n − 1 n … 1 − 1 n ] = − 1 2 [ … … − 1 n ∑ i d i 1 2 + d i 1 − 1 n ∑ i d i 2 2 + d i 2 … − 1 n ∑ i d i n 2 + d i n … ] [ … … − 1 n … … … − 1 n … ⋮ ⋮ 1 − 1 n ⋮ … … − 1 n … ] = − 1 2 [ ⋮ … 1 n 2 ∑ i ∑ j d i j 2 − 1 n ∑ j d i j 2 + d i j 2 − 1 n ∑ i d i j 2 … ⋮ ] = B \begin{aligned}
-\frac{1}{2}HD'H &= \\
&= -\frac{1}{2}
\left[\begin{matrix} 1-\frac{1}{n} & -\frac{1}{n} & \dots & -\frac{1}{n} \\ -\frac{1}{n} & 1-\frac{1}{n} & \dots & -\frac{1}{n} \\ \vdots & \vdots & \dots &\vdots \\ -\frac{1}{n} & -\frac{1}{n} & \dots & 1-\frac{1}{n} \end{matrix}\right]
\left[ \begin{matrix} d_{11}^2 & d_{12}^2 & \dots & d_{1n}^2 \\ d_{21}^2 & d_{22}^2 & \dots & d_{2n}^2 \\ \vdots & \vdots & \dots & \vdots \\ d_{n1}^2 & d_{n2}^2 & \dots & d_{nn}^2 \end{matrix}\right]
\left[\begin{matrix} 1-\frac{1}{n} & -\frac{1}{n} & \dots & -\frac{1}{n} \\ -\frac{1}{n} & 1-\frac{1}{n} & \dots & -\frac{1}{n} \\ \vdots & \vdots & \dots &\vdots \\ -\frac{1}{n} & -\frac{1}{n} & \dots & 1-\frac{1}{n} \end{matrix}\right] \\
&=-\frac{1}{2}\left[\begin{matrix} \dots \\ \dots \\ -\frac{1}{n}\sum_id_{i1}^2+d_{i1} & -\frac{1}{n}\sum_id_{i2}^2+d_{i2} & \dots & -\frac{1}{n}\sum_id_{in}^2+d_{in} \\ \dots \end{matrix}\right]
\left[ \begin{matrix} \dots & \dots & -\frac{1}{n} & \dots \\ \dots & \dots & -\frac{1}{n} & \dots \\ \vdots &\vdots &1- \frac{1}{n} & \vdots \\\dots & \dots & -\frac{1}{n} & \dots \\ \end{matrix}\right] \\
&=-\frac{1}{2}\left[ \begin{matrix} \vdots \\ \dots \frac{1}{n^2}\sum_i\sum_jd_{ij}^2-\frac{1}{n}\sum_jd_{ij}^2+d_{ij}^2-\frac{1}{n}\sum_id_{ij}^2 & \dots \\ \vdots \end{matrix}\right] \\
&=B
\end{aligned}
− 2 1 H D ′ H = = − 2 1 ⎣ ⎢ ⎢ ⎢ ⎡ 1 − n 1 − n 1 ⋮ − n 1 − n 1 1 − n 1 ⋮ − n 1 … … … … − n 1 − n 1 ⋮ 1 − n 1 ⎦ ⎥ ⎥ ⎥ ⎤ ⎣ ⎢ ⎢ ⎢ ⎡ d 1 1 2 d 2 1 2 ⋮ d n 1 2 d 1 2 2 d 2 2 2 ⋮ d n 2 2 … … … … d 1 n 2 d 2 n 2 ⋮ d n n 2 ⎦ ⎥ ⎥ ⎥ ⎤ ⎣ ⎢ ⎢ ⎢ ⎡ 1 − n 1 − n 1 ⋮ − n 1 − n 1 1 − n 1 ⋮ − n 1 … … … … − n 1 − n 1 ⋮ 1 − n 1 ⎦ ⎥ ⎥ ⎥ ⎤ = − 2 1 ⎣ ⎢ ⎢ ⎡ … … − n 1 ∑ i d i 1 2 + d i 1 … − n 1 ∑ i d i 2 2 + d i 2 … − n 1 ∑ i d i n 2 + d i n ⎦ ⎥ ⎥ ⎤ ⎣ ⎢ ⎢ ⎢ ⎡ … … ⋮ … … … ⋮ … − n 1 − n 1 1 − n 1 − n 1 … … ⋮ … ⎦ ⎥ ⎥ ⎥ ⎤ = − 2 1 ⎣ ⎢ ⎢ ⎡ ⋮ … n 2 1 ∑ i ∑ j d i j 2 − n 1 ∑ j d i j 2 + d i j 2 − n 1 ∑ i d i j 2 ⋮ … ⎦ ⎥ ⎥ ⎤ = B
所以− 1 2 H D ′ H = B = X T X -\frac{1}{2}HD'H=B=X^TX − 2 1 H D ′ H = B = X T X
T. Cox and M. Cox. Multidimensional Scaling. Chapman Hall, Boca Raton, 2nd edition,
