假設讀者已經瞭解 LDA 的來龍去脈。
需要明確採樣的含義：
語料庫中的第 i i i z i z_i z i i = ( m , n ) i=(m,n) i = ( m , n ) i i i m m m n n n ¬ i \neg i ¬ i i i i i i i p ( z i = k ∣ Z ⃗ ¬ i , W ⃗ ) p(z_i=k|\vec{Z}_{\neg i},\vec{W}) p ( z i = k ∣ Z ¬ i , W ) p ( z i = k ∣ Z ⃗ ¬ i , W ⃗ ) ∝ p ( z i = k , w i = t ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) p(z_i=k|\vec{Z}_{\neg i},\vec{W})\propto p(z_i=k,w_i=t|\vec{Z}_{\neg i},\vec{W}_{\neg i}) p ( z i = k ∣ Z ¬ i , W ) ∝ p ( z i = k , w i = t ∣ Z ¬ i , W ¬ i ) z i = k , w i = t z_i=k,w_i=t z i = k , w i = t m m m k k k
α ⃗ → θ ⃗ m → z ⃗ m \vec{\alpha}\rightarrow\vec{\theta}_m\rightarrow\vec{z}_m α → θ m → z m β ⃗ → φ ⃗ k → w ⃗ k \vec{\beta}\rightarrow\vec{\varphi}_k\rightarrow\vec{w}_k β → φ k → w k
去掉了語料中第 i i i ( z i , w i ) (z_i,w_i) ( z i , w i ) θ ⃗ m , φ ⃗ k \vec{\theta}_m,\vec{\varphi}_k θ m , φ k p ( θ ⃗ m ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) = D i r ( θ ⃗ m ∣ n ⃗ m , ¬ i + α ⃗ ) p ( φ ⃗ k ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) = D i r ( φ ⃗ k ∣ n ⃗ k , ¬ i + β ⃗ ) p(\vec{\theta}_m|\vec{Z}_{\neg i},\vec{W}_{\neg i})=Dir(\vec{\theta}_m|\vec{n}_{m,\neg i}+\vec{\alpha})\\p(\vec{\varphi}_k|\vec{Z}_{\neg i},\vec{W}_{\neg i})=Dir(\vec{\varphi}_k|\vec{n}_{k,\neg i}+\vec{\beta}) p ( θ m ∣ Z ¬ i , W ¬ i ) = D i r ( θ m ∣ n m , ¬ i + α ) p ( φ k ∣ Z ¬ i , W ¬ i ) = D i r ( φ k ∣ n k , ¬ i + β ) n ⃗ m , ¬ i , n ⃗ k , ¬ i \vec{n}_{m,\neg i},\vec{n}_{k,\neg i} n m , ¬ i , n k , ¬ i n ⃗ m = ( n m ( 1 ) , ⋯  , n m ( K ) ) \vec{n}_{m}=(n_m^{(1)},\cdots,n_m^{(K)}) n m = ( n m ( 1 ) , ⋯ , n m ( K ) ) n m ( k ) n_m^{(k)} n m ( k ) n ⃗ k = ( n k ( 1 ) , ⋯  , n k ( V ) ) \vec{n}_k=(n_k^{(1)},\cdots,n_k^{(V)}) n k = ( n k ( 1 ) , ⋯ , n k ( V ) ) n k ( t ) n_k^{(t)} n k ( t ) ¬ i \neg i ¬ i i i i n ⃗ m , n ⃗ k \vec{n}_{m},\vec{n}_{k} n m , n k n ⃗ m , ¬ i , n ⃗ k , ¬ i \vec{n}_{m,\neg i},\vec{n}_{k,\neg i} n m , ¬ i , n k , ¬ i
Gibbs Sampling 公式的推導：p ( z i = k ∣ Z ⃗ ¬ i , W ⃗ ) ∝ p ( z i = k , w i = t ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) = ∫ p ( z i = k , w i = t , θ ⃗ m , φ ⃗ k ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) d θ ⃗ m d φ ⃗ k = ∫ p ( z i = k , θ ⃗ m ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) ⋅ p ( w i = t , φ ⃗ k ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) d θ ⃗ m d φ ⃗ k = ∫ p ( z i = k ∣ θ ⃗ m ) p ( θ ⃗ m ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) ⋅ p ( w i = t ∣ φ ⃗ k ) p ( φ ⃗ k ∣ Z ⃗ ¬ i , W ⃗ ¬ i ) d θ ⃗ m d φ ⃗ k = ∫ p ( z i = k ∣ θ ⃗ m ) D i r ( θ ⃗ m ∣ n ⃗ m , ¬ i + α ⃗ ) d θ ⃗ m ⋅ ∫ p ( w i = t ∣ φ ⃗ k ) D i r ( φ ⃗ k ∣ n ⃗ k , ¬ i + β ⃗ ) d φ ⃗ k = ∫ θ m k D i r ( θ ⃗ m ∣ n ⃗ m , ¬ i + α ⃗ ) d θ ⃗ m ⋅ ∫ φ k t D i r ( φ ⃗ k ∣ n ⃗ k , ¬ i + β ⃗ ) d φ ⃗ k = E ( θ m k ) ⋅ E ( φ k t ) = θ ^ m k ⋅ φ ^ k t
\begin{aligned}
p(z_i=k|\vec{Z}_{\neg i},\vec{W})\propto & p(z_i=k,w_i=t|\vec{Z}_{\neg i},\vec{W}_{\neg i})\\
=&\int p(z_i=k,w_i=t,\vec{\theta}_m,\vec{\varphi}_k|\vec{Z}_{\neg i},\vec{W}_{\neg i})\mathrm d\vec{\theta}_m\mathrm d\vec{\varphi}_k\\
=&\int p(z_i=k,\vec{\theta}_m|\vec{Z}_{\neg i},\vec{W}_{\neg i})\cdot p(w_i=t,\vec{\varphi}_k|\vec{Z}_{\neg i},\vec{W}_{\neg i})\mathrm d\vec{\theta}_m\mathrm d\vec{\varphi}_k\\
=&\int p(z_i=k|\vec{\theta}_m)p(\vec{\theta}_m|\vec{Z}_{\neg i},\vec{W}_{\neg i})\cdot p(w_i=t|\vec{\varphi}_k)p(\vec{\varphi}_k|\vec{Z}_{\neg i},\vec{W}_{\neg i})\mathrm d\vec{\theta}_m\mathrm d\vec{\varphi}_k\\
=&\int p(z_i=k|\vec{\theta}_m)Dir(\vec{\theta}_m|\vec{n}_{m,\neg i}+\vec{\alpha})\mathrm d\vec{\theta}_m\cdot\int p(w_i=t|\vec{\varphi}_k)Dir(\vec{\varphi}_k|\vec{n}_{k,\neg i}+\vec{\beta})\mathrm d\vec{\varphi}_k\\
=&\int \theta_{mk} Dir(\vec{\theta}_m|\vec{n}_{m,\neg i}+\vec{\alpha})\mathrm d\vec{\theta}_m\cdot\int \varphi_{kt} Dir(\vec{\varphi}_k|\vec{n}_{k,\neg i}+\vec{\beta})\mathrm d\vec{\varphi}_k\\
=&E(\theta_{mk})\cdot E(\varphi_{kt})\\
=&\hat{\theta}_{mk}\cdot\hat{\varphi}_{kt}
\end{aligned}
p ( z i = k ∣ Z ¬ i , W ) ∝ = = = = = = = p ( z i = k , w i = t ∣ Z ¬ i , W ¬ i ) ∫ p ( z i = k , w i = t , θ m , φ k ∣ Z ¬ i , W ¬ i ) d θ m d φ k ∫ p ( z i = k , θ m ∣ Z ¬ i , W ¬ i ) ⋅ p ( w i = t , φ k ∣ Z ¬ i , W ¬ i ) d θ m d φ k ∫ p ( z i = k ∣ θ m ) p ( θ m ∣ Z ¬ i , W ¬ i ) ⋅ p ( w i = t ∣ φ k ) p ( φ k ∣ Z ¬ i , W ¬ i ) d θ m d φ k ∫ p ( z i = k ∣ θ m ) D i r ( θ m ∣ n m , ¬ i + α ) d θ m ⋅ ∫ p ( w i = t ∣ φ k ) D i r ( φ k ∣ n k , ¬ i + β ) d φ k ∫ θ m k D i r ( θ m ∣ n m , ¬ i + α ) d θ m ⋅ ∫ φ k t D i r ( φ k ∣ n k , ¬ i + β ) d φ k E ( θ m k ) ⋅ E ( φ k t ) θ ^ m k ⋅ φ ^ k t
根據狄利克雷分佈的期望公式有：θ ^ m k = n m , ¬ i ( k ) + α k ∑ k = 1 K n m , ¬ i ( k ) + α k φ ^ k t = n k , ¬ i ( t ) + β t ∑ t = 1 V n k , ¬ i ( t ) + β t \hat{\theta}_{mk}=\frac{n_{m,\neg i}^{(k)}+\alpha_k}{\sum_{k=1}^Kn_{m,\neg i}^{(k)}+\alpha_k}\\\hat{\varphi}_{kt}=\frac{n_{k,\neg i}^{(t)}+\beta_t}{\sum_{t=1}^Vn_{k,\neg i}^{(t)}+\beta_t} θ ^ m k = ∑ k = 1 K n m , ¬ i ( k ) + α k n m , ¬ i ( k ) + α k φ ^ k t = ∑ t = 1 V n k , ¬ i ( t ) + β t n k , ¬ i ( t ) + β t
所以 LDA 模型的採樣公式爲：p ( z i = k ∣ Z ⃗ ¬ i , W ⃗ ) ∝ n m , ¬ i ( k ) + α k ∑ k = 1 K n m , ¬ i ( k ) + α k ⋅ n k , ¬ i ( t ) + β t ∑ t = 1 V n k , ¬ i ( t ) + β t p(z_i=k|\vec{Z}_{\neg i},\vec{W})\propto\frac{n_{m,\neg i}^{(k)}+\alpha_k}{\sum_{k=1}^Kn_{m,\neg i}^{(k)}+\alpha_k}\cdot\frac{n_{k,\neg i}^{(t)}+\beta_t}{\sum_{t=1}^Vn_{k,\neg i}^{(t)}+\beta_t} p ( z i = k ∣ Z ¬ i , W ) ∝ ∑ k = 1 K n m , ¬ i ( k ) + α k n m , ¬ i ( k ) + α k ⋅ ∑ t = 1 V n k , ¬ i ( t ) + β t n k , ¬ i ( t ) + β t
第一類狄利克雷積分，後面直接當作結論使用，不做證明：Δ ( α ⃗ ) = ∫ ∑ x i = 1 ∏ i = 1 N x i α i − 1 d x ⃗ \Delta(\vec{\alpha})=\int_{\sum x_i=1}\prod_{i=1}^Nx_i^{\alpha_i-1}\mathrm d\vec{x} Δ ( α ) = ∫ ∑ x i = 1 i = 1 ∏ N x i α i − 1 d x p ( W ∣ α ⃗ ) = ∫ p ( W ∣ p ⃗ ) ⋅ p ( p ⃗ ∣ α ⃗ ) d p ⃗ p(W|\vec{\alpha})=\int{p(W|\vec{p})\cdot p(\vec{p}|\vec{\alpha})\mathrm d\vec{p}} p ( W ∣ α ) = ∫ p ( W ∣ p ) ⋅ p ( p ∣ α ) d p p ( W ∣ α ⃗ ) = ∫ ∏ n = 1 N M u l t ( w = w n ∣ p ⃗ , 1 ) ⋅ D i r ( p ⃗ ∣ α ⃗ ) d p ⃗ = ∫ ∏ v = 1 V p v n ( v ) ⋅ 1 Δ ( α ⃗ ) ∏ v = 1 V p v α v − 1 d p ⃗ = 1 Δ ( α ⃗ ) ∫ ∏ v = 1 V p v n ( v ) + α v − 1 d p ⃗ ∣ D i r i c h l e t ∫ = Δ ( n ⃗ + α ⃗ ) Δ ( α ⃗ ) , n ⃗ = { n ( v ) } v = 1 V
\begin{aligned}
p(W|\vec{\alpha})=&\int{\prod_{n=1}^N\mathrm{Mult}(w=w_n|\vec{p},1)\cdot \mathrm{Dir}(\vec{p}|\vec{\alpha})\mathrm d\vec{p}}\\
=&\int{\prod_{v=1}^Vp_v^{n^{(v)}}\cdot \frac{1}{\Delta(\vec\alpha)}\prod_{v=1}^Vp_v^{\alpha_v-1}\mathrm d\vec{p}}\\
=&\frac{1}{\Delta(\vec\alpha)}\int{\prod_{v=1}^Vp_v^{n^{(v)}+\alpha_v-1}\mathrm d\vec{p}} & | \mathrm{Dirichlet}\int\\
=&\frac{\Delta(\vec{n}+\vec\alpha)}{\Delta(\vec\alpha)},\vec{n}=\{n^{(v)}\}_{v=1}^V
\end{aligned}
p ( W ∣ α ) = = = = ∫ n = 1 ∏ N M u l t ( w = w n ∣ p , 1 ) ⋅ D i r ( p ∣ α ) d p ∫ v = 1 ∏ V p v n ( v ) ⋅ Δ ( α ) 1 v = 1 ∏ V p v α v − 1 d p Δ ( α ) 1 ∫ v = 1 ∏ V p v n ( v ) + α v − 1 d p Δ ( α ) Δ ( n + α ) , n = { n ( v ) } v = 1 V ∣ D i r i c h l e t ∫ M u l t ( w = w n ∣ p ⃗ , 1 ) \mathrm{Mult}(w=w_n|\vec{p},1) M u l t ( w = w n ∣ p , 1 )
上面這個推導的意義是，消去了多項式分佈的參數 p ⃗ \vec{p} p
那麼對於 LDA 模型來說，就是 K+M 個 Dirichlet-Multinomial 共軛結構。p ( w ⃗ ∣ z ⃗ , β ⃗ ) = ∫ p ( w ⃗ ∣ z ⃗ , Φ ) p ( Φ ∣ β ⃗ ) d Φ = ∏ k = 1 K ∫ 1 Δ ( β ⃗ ) ∏ t = 1 V φ k , t n k ( t ) + β t − 1 d φ ⃗ k = ∏ k = 1 K Δ ( n ⃗ k + β ⃗ ) Δ ( β ⃗ ) , n ⃗ k = { n k ( t ) } t = 1 V
\begin{aligned}
p(\vec w|\vec z,\vec \beta)=&\int p(\vec w|\vec z,\Phi)p(\Phi|\vec \beta)\mathrm d\Phi\\
=&\prod_{k=1}^K\int\frac{1}{\Delta(\vec\beta)}\prod_{t=1}^V\varphi_{k,t}^{n_k^{(t)}+\beta_t-1}\mathrm d\vec\varphi_k\\
=&\prod_{k=1}^K\frac{\Delta(\vec n_k+\vec\beta)}{\Delta(\vec\beta)},\vec n_k=\{n_k^{(t)}\}_{t=1}^V
\end{aligned}
p ( w ∣ z , β ) = = = ∫ p ( w ∣ z , Φ ) p ( Φ ∣ β ) d Φ k = 1 ∏ K ∫ Δ ( β ) 1 t = 1 ∏ V φ k , t n k ( t ) + β t − 1 d φ k k = 1 ∏ K Δ ( β ) Δ ( n k + β ) , n k = { n k ( t ) } t = 1 V p ( z ⃗ ∣ α ⃗ ) = ∫ p ( z ⃗ ∣ Θ ) p ( Θ ∣ α ⃗ ) d Θ = ∏ m = 1 M ∫ 1 Δ ( α ⃗ ) ∏ k = 1 K θ m , k n m ( k ) + α k − 1 d θ ⃗ m = ∏ m = 1 M Δ ( n ⃗ m + α ⃗ ) Δ ( α ⃗ ) , n ⃗ m = { n m ( k ) } k = 1 K
\begin{aligned}
p(\vec z|\vec \alpha)=&\int p(\vec z|\Theta)p(\Theta|\vec \alpha)\mathrm d\Theta\\
=&\prod_{m=1}^M\int\frac{1}{\Delta(\vec\alpha)}\prod_{k=1}^K\theta_{m,k}^{n_m^{(k)}+\alpha_k-1}\mathrm d\vec\theta_m\\
=&\prod_{m=1}^M\frac{\Delta(\vec n_m+\vec\alpha)}{\Delta(\vec\alpha)},\vec n_m=\{n_m^{(k)}\}_{k=1}^K
\end{aligned}
p ( z ∣ α ) = = = ∫ p ( z ∣ Θ ) p ( Θ ∣ α ) d Θ m = 1 ∏ M ∫ Δ ( α ) 1 k = 1 ∏ K θ m , k n m ( k ) + α k − 1 d θ m m = 1 ∏ M Δ ( α ) Δ ( n m + α ) , n m = { n m ( k ) } k = 1 K p ( z ⃗ , w ⃗ ∣ α ⃗ , β ⃗ ) = ∏ k = 1 K Δ ( n ⃗ k + β ⃗ ) Δ ( β ⃗ ) ⋅ ∏ m = 1 M Δ ( n ⃗ m + α ⃗ ) Δ ( α ⃗ ) p(\vec z,\vec w|\vec\alpha,\vec\beta)=\prod_{k=1}^K\frac{\Delta(\vec n_k+\vec\beta)}{\Delta(\vec\beta)}\cdot\prod_{m=1}^M\frac{\Delta(\vec n_m+\vec\alpha)}{\Delta(\vec\alpha)} p ( z , w ∣ α , β ) = k = 1 ∏ K Δ ( β ) Δ ( n k + β ) ⋅ m = 1 ∏ M Δ ( α ) Δ ( n m + α ) p ( z i = k ∣ z ⃗ ¬ i , w ⃗ ) = p ( w ⃗ , z ⃗ ) p ( w ⃗ , z ⃗ ¬ i ) = p ( w ⃗ ∣ z ⃗ ) p ( w ⃗ ¬ i ∣ z ⃗ ¬ i ) p ( w i ) ⋅ p ( z ⃗ ) p ( z ⃗ ¬ i ) ( 1 ) ∝ Δ ( n ⃗ k + β ⃗ ) Δ ( n ⃗ k , ¬ i + β ⃗ ) ⋅ Δ ( n ⃗ m + α ⃗ ) Δ ( n ⃗ m , ¬ i + α ⃗ ) ( 2 ) = Γ ( n k ( t ) + β t ) Γ ( ∑ t = 1 V n k , ¬ i ( t ) + β t ) Γ ( n k , ¬ i ( t ) + β t ) ) Γ ( ∑ t = 1 V n k ( t ) + β t ) ⋅ Γ ( n m ( k ) + α k ) Γ ( ∑ k = 1 K n m , ¬ i ( k ) + α k ) Γ ( n m , ¬ i ( k ) + α k ) Γ ( ∑ k = 1 K n m ( k ) + α k ) ( 3 ) = n k , ¬ i ( t ) + β t ∑ t = 1 V n k , ¬ i ( t ) + β t ⋅ n m , ¬ i ( k ) + α k [ ∑ k = 1 K n m ( k ) + α k ] − 1 ( 4 ) ∝ n k , ¬ i ( t ) + β t ∑ t = 1 V n k , ¬ i ( t ) + β t ⋅ ( n m , ¬ i ( k ) + α k ) ( 5 )
\begin{aligned}
p(z_i=k|\vec{z}_{\neg i},\vec{w})=& \frac{p(\vec{w},\vec{z})}{p(\vec{w},\vec{z}_{\neg i})}=\frac{p(\vec{w}|\vec{z})}{p(\vec{w}_{\neg i}|\vec{z}_{\neg i})p(w_i)}\cdot\frac{p(\vec{z})}{p(\vec{z}_{\neg i})} & (1)\\
\propto & \frac{\Delta(\vec{n}_k+\vec\beta)}{\Delta(\vec{n}_{k,\neg i}+\vec\beta)}\cdot \frac{\Delta(\vec{n}_m+\vec\alpha)}{\Delta(\vec{n}_{m,\neg i}+\vec\alpha)} &(2)\\
=& \frac{\Gamma(n_k^{(t)}+\beta_t)\Gamma(\sum_{t=1}^Vn_{k,\neg i}^{(t)}+\beta_t)}{\Gamma(n_{k,\neg i}^{(t)}+\beta_t))\Gamma(\sum_{t=1}^Vn_k^{(t)}+\beta_t)}\cdot\frac{\Gamma(n_m^{(k)}+\alpha_k)\Gamma(\sum_{k=1}^Kn_{m,\neg i}^{(k)}+\alpha_k)}{\Gamma(n_{m,\neg i}^{(k)}+\alpha_k)\Gamma(\sum_{k=1}^Kn_m^{(k)}+\alpha_k)}&(3)\\
=&\frac{n_{k,\neg i}^{(t)}+\beta_t}{\sum_{t=1}^Vn_{k,\neg i}^{(t)}+\beta_t}\cdot\frac{n_{m,\neg i}^{(k)}+\alpha_k}{[\sum_{k=1}^Kn_m^{(k)}+\alpha_k]-1}&(4)\\
\propto & \frac{n_{k,\neg i}^{(t)}+\beta_t}{\sum_{t=1}^Vn_{k,\neg i}^{(t)}+\beta_t}\cdot(n_{m,\neg i}^{(k)}+\alpha_k)&(5)
\end{aligned}
p ( z i = k ∣ z ¬ i , w ) = ∝ = = ∝ p ( w , z ¬ i ) p ( w , z ) = p ( w ¬ i ∣ z ¬ i ) p ( w i ) p ( w ∣ z ) ⋅ p ( z ¬ i ) p ( z ) Δ ( n k , ¬ i + β ) Δ ( n k + β ) ⋅ Δ ( n m , ¬ i + α ) Δ ( n m + α ) Γ ( n k , ¬ i ( t ) + β t ) ) Γ ( ∑ t = 1 V n k ( t ) + β t ) Γ ( n k ( t ) + β t ) Γ ( ∑ t = 1 V n k , ¬ i ( t ) + β t ) ⋅ Γ ( n m , ¬ i ( k ) + α k ) Γ ( ∑ k = 1 K n m ( k ) + α k ) Γ ( n m ( k ) + α k ) Γ ( ∑ k = 1 K n m , ¬ i ( k ) + α k ) ∑ t = 1 V n k , ¬ i ( t ) + β t n k , ¬ i ( t ) + β t ⋅ [ ∑ k = 1 K n m ( k ) + α k ] − 1 n m , ¬ i ( k ) + α k ∑ t = 1 V n k , ¬ i ( t ) + β t n k , ¬ i ( t ) + β t ⋅ ( n m , ¬ i ( k ) + α k ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 )
(1) 式和 (2) 式正比，是因爲 p ( w i ) p(w_i) p ( w i ) ¬ i \neg i ¬ i Δ \Delta Δ
(2) 式到 (3) 式，同樣是因爲 i 只與第 k 個主題和第 m 篇文檔有關，將 Δ \Delta Δ Γ \Gamma Γ Γ \Gamma Γ
(3) 式到 (4) 式利用了 Γ \Gamma Γ Γ ( x + 1 ) = x Γ ( x ) \Gamma(x+1)=x\Gamma(x) Γ ( x + 1 ) = x Γ ( x )
(4) 式裏有一個減 1，是因爲把第 m 篇文檔裏要去掉的那個計數分離了出來。
因爲第 m 篇文檔的單詞數是已知的，所以 (4) 式和 (5) 式是正比關係。
疑問：LDA 並沒有做任何詞語語義方面的工作，僅僅是做一些數量上的統計，它是怎麼挖掘出文檔具有語義含義的主題的呢？
LDA 本質上是對詞進行了聚類（依據某方面的相似度），聚的這個類就是主題。換句話說就是，按照主題給詞進行了聚類。
既然沒有做詞語義方面的工作，那詞之間的相似度是怎麼確定的呢？
