本系列通過如下兩部分來講解EM算法：
1. EM算法的原理
2. EM算法實際應用--HMM
EM算法原理
EM算法全稱是 最大期望算法（Expectation-Maximization algorithm, EM），是一類通過迭代進行極大似然估計（Maximum Likelihood Estimation, MLE）的優化算法 ，通常作爲牛頓迭代法（Newton-Raphson method）的替代用於對包含隱變量（latent variable）或缺失數據（incomplete-data）的概率模型進行參數估計 [2-3] 。
EM算法的標準計算框架由E步（Expectation-step）和M步（Maximization step）交替組成，算法的收斂性可以確保迭代至少逼近局部極大值 [4] 。EM算法是MM算法（Minorize-Maximization algorithm）的特例之一，有多個改進版本，包括使用了貝葉斯推斷的EM算法、EM梯度算法、廣義EM算法等 。
給定相互獨立的觀測數據
![logp(X|\theta) = log\prod_i{p(X_i|\theta)} = \sum_i{logp(X_i|\theta)} \\ = \sum_i{log[\sum_{Z_c}{p(X_i,Z_c|\theta)}]} \\ = \sum_i{log[\sum_{Z_c}{q(Z_c)\frac{p(X_i,Z_c|\theta)}{q(Z_c)}]}} \\](https://pic1.xuehuaimg.com/proxy/csdn/https://private.codecogs.com/gif.latex?logp%28X%7C%5Ctheta%29%20%3D%20log%5Cprod_i%7Bp%28X_i%7C%5Ctheta%29%7D%20%3D%20%5Csum_i%7Blogp%28X_i%7C%5Ctheta%29%7D%20%5C%5C%20%3D%20%5Csum_i%7Blog%5B%5Csum_%7BZ_c%7D%7Bp%28X_i%2CZ_c%7C%5Ctheta%29%7D%5D%7D%20%5C%5C%20%3D%20%5Csum_i%7Blog%5B%5Csum_%7BZ_c%7D%7Bq%28Z_c%29%5Cfrac%7Bp%28X_i%2CZ_c%7C%5Ctheta%29%7D%7Bq%28Z_c%29%7D%5D%7D%7D%20%5C%5C)
![logp(X|\theta) = \sum_i{log[\sum_{Z_c}{q(Z_c)\frac{p(X_i,Z_c|\theta)}{q(Z_c)}]}}\\ \geq \sum_i{\sum_{Z_c}{q(Z_c)log[\frac{p(X_i,Z_c|\theta)}{q(Z_c)}]}} = L(\theta, q)](https://pic1.xuehuaimg.com/proxy/csdn/https://private.codecogs.com/gif.latex?logp%28X%7C%5Ctheta%29%20%3D%20%5Csum_i%7Blog%5B%5Csum_%7BZ_c%7D%7Bq%28Z_c%29%5Cfrac%7Bp%28X_i%2CZ_c%7C%5Ctheta%29%7D%7Bq%28Z_c%29%7D%5D%7D%7D%5C%5C%20%5Cgeq%20%5Csum_i%7B%5Csum_%7BZ_c%7D%7Bq%28Z_c%29log%5B%5Cfrac%7Bp%28X_i%2CZ_c%7C%5Ctheta%29%7D%7Bq%28Z_c%29%7D%5D%7D%7D%20%3D%20L%28%5Ctheta%2C%20q%29)
![log[\frac{p(X,Z|\theta)}{q(Z)}]](https://pic1.xuehuaimg.com/proxy/csdn/https://private.codecogs.com/gif.latex?log%5B%5Cfrac%7Bp%28X%2CZ%7C%5Ctheta%29%7D%7Bq%28Z%29%7D%5D)
何時取等號呢？
![logp(X|\theta) - L(\theta, q) \\ = \sum_i{log[p(X_i |\theta)]\sum_{Z_c}{q(Z_c)}}- \sum_i{\sum_{Z_c}{q(Z_c)log[\frac{p(X_i,Z_c|\theta)}{q(Z_c)}]}} \\ = \sum_i\sum_{Z_c}q(Z_c)log[\frac{p(X_i |\theta)q(Z_c)}{p(X_i,Z_c|\theta)}]\\ = \sum_i\sum_{Z_c}q(Z_c)log[\frac{q(Z_c)}{p(Z_c|X_i,\theta)}] \\ = \sum_iKL(q(Z)|p(Z|X_i,\theta))](https://pic1.xuehuaimg.com/proxy/csdn/https://private.codecogs.com/gif.latex?logp%28X%7C%5Ctheta%29%20-%20L%28%5Ctheta%2C%20q%29%20%5C%5C%20%3D%20%5Csum_i%7Blog%5Bp%28X_i%20%7C%5Ctheta%29%5D%5Csum_%7BZ_c%7D%7Bq%28Z_c%29%7D%7D-%20%5Csum_i%7B%5Csum_%7BZ_c%7D%7Bq%28Z_c%29log%5B%5Cfrac%7Bp%28X_i%2CZ_c%7C%5Ctheta%29%7D%7Bq%28Z_c%29%7D%5D%7D%7D%20%5C%5C%20%3D%20%5Csum_i%5Csum_%7BZ_c%7Dq%28Z_c%29log%5B%5Cfrac%7Bp%28X_i%20%7C%5Ctheta%29q%28Z_c%29%7D%7Bp%28X_i%2CZ_c%7C%5Ctheta%29%7D%5D%5C%5C%20%3D%20%5Csum_i%5Csum_%7BZ_c%7Dq%28Z_c%29log%5B%5Cfrac%7Bq%28Z_c%29%7D%7Bp%28Z_c%7CX_i%2C%5Ctheta%29%7D%5D%20%5C%5C%20%3D%20%5Csum_iKL%28q%28Z%29%7Cp%28Z%7CX_i%2C%5Ctheta%29%29)
即當
假設
令
所以不斷迭代更新
總結下EM算法的流程：
1. 初始化參數，
2. E-step：構造隱期望函數
![L(\theta, q)= \sum_i{\sum_{Z_c}{q(Z_c)log[\frac{p(X_i,Z_c|\theta)}{q(Z_c)}]}} , \; q(Z_c) = p(Z_c|X_i,\theta^{t-1})](https://pic1.xuehuaimg.com/proxy/csdn/https://private.codecogs.com/gif.latex?L%28%5Ctheta%2C%20q%29%3D%20%5Csum_i%7B%5Csum_%7BZ_c%7D%7Bq%28Z_c%29log%5B%5Cfrac%7Bp%28X_i%2CZ_c%7C%5Ctheta%29%7D%7Bq%28Z_c%29%7D%5D%7D%7D%20%2C%20%5C%3B%20q%28Z_c%29%20%3D%20p%28Z_c%7CX_i%2C%5Ctheta%5E%7Bt-1%7D%29)
3. M-step：最大化隱期望函數
4. 重複2～3，直到似然函數幾乎不再增大