貝葉斯&python應用

貝葉斯

貝葉斯判定準則

爲最小化總體風險，只需要在每個樣本上選擇那個能使條件風險$R(c|x)$最小的類別標記，即：
$h^*{(x)}=\argmin\limits_{c\in{\mathcal Y}}R(c|x)\tag{式1}$
此時，$h^*(x)$稱爲貝葉斯最優分類器。
已知，條件風險$R(c|x)$的計算公式爲:
$R(c_i|x)=\sum_{j=1}^{N}\lambda_{ij}P(c_j|x)\tag{式2}$
如若目標是最小化分類錯誤率，則誤判損失$\lambda_{ij}$對應爲0/1損失，即:
$\begin{aligned} \lambda_{ij}= \begin{cases} 0,\qquad &{if\quad i=j}\\ 1,&otherwise \end{cases} \end{aligned}\tag{式3}$
那麼條件風險$R(c|x)$的計算公式進一步展開爲：
$\begin{aligned} R(c_i|x)&=1\cdot P(c_1|x)+\cdots +1\cdot P(c_{i-1}|x)+0\cdot P(c_i|x)\\ &+1\cdot P(c_{i+1}|x)+\cdots+1\cdot P(c_N|x)\\ &=P(c_1|x)+\cdots+P(c_{i-1}|x)+P(c_{i+1}|x)+\cdots +P(c_N|x)\tag{式4} \end{aligned}$
由於$\sum_{j=1}^{N}P(c_j|x)=1$,所以有:
$R(c_i|x)=1-P(c_i|x)\tag{式5}$
於是呢，最小化錯誤率的貝葉斯最優分類器就是:
$h^*(x)=\argmin\limits_{c\in{\mathcal{Y}}}R(c|x)=\argmin\limits_{c\in{\mathcal{Y}}}(1-P(c|x))=\argmax\limits_{c\in{\mathcal{Y}}}P(c|x)\tag{式6}$

多元正態度分佈參數的極大似然估計

已知對數似然函數爲:
$LL(\theta_c)=\sum_{x\in{D_c}}logP(x|\theta_c)\tag{式7}$
爲了便於計算，令$log$的底數爲$e$，則對數似然函數爲:
$LL(\theta_c)=\sum_{x\in{D_c}}lnP(x|\theta_c)\tag{式8}$
由於$P(x|\theta_c)=P(x|c)\sim\mathcal{N}(\mu_c,\sigma_c^2)$那麼：
$P(x|\theta_c)=\cfrac{1}{\sqrt{(2\pi)^d{|\Sigma_c|}}}exp(-\cfrac{1}{2}(x-\mu_c)^T\Sigma_c^{-1}(x-\mu_c))\tag{式9}$
其中，$d$表示$x$的維數，$\Sigma_c=\sigma_c^2$爲對稱正定協方差矩陣，$|\Sigma_c|$表示行列式，將上式代入對數似然函數可得:
$LL(\theta_c)=\sum_{x\in{D_c}}ln[\cfrac{1}{\sqrt{(2\pi)^d{|\Sigma_c|}}}exp(-\cfrac{1}{2}(x-\mu_c)^T\Sigma_c^{-1}(x-\mu_c))]\tag{式10}$
令$|D_c=N|$,則對數似然函數化爲:
$\begin{aligned} LL(\theta_c)&=\sum_{x=1}^{N}ln[\cfrac{1}{\sqrt{(2\pi)^d{|\Sigma_c|}}}exp(-\cfrac{1}{2}(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c))]\\ &=\sum_{i=1}^{N}ln[\cfrac{1}{\sqrt{(2\pi)^d}}\cdot \cfrac{1}{\sqrt{|\Sigma_c|}}exp(-\cfrac{1}{2}(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c))]\\ &=\sum_{i=1}^{N}\{ln\cfrac{1}{\sqrt{(2\pi)^d}}+ln\cfrac{1}{\sqrt{|\Sigma_c|}}+ln[exp(-\cfrac{1}{2}(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c))]\}\\ &=\sum_{i=1}^{N}\{-\cfrac{d}{2}ln(2\pi)-\cfrac{1}{2}ln|\Sigma_c|-\cfrac{1}{2}(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c)\}\\ &=-\cfrac{Nd}{2}ln(2\pi)-\cfrac{N}{2}ln|\Sigma_c|-\cfrac{1}{2}\Sigma_{i=1}^{N}(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c) \end{aligned}\tag{式11}$
由於參數$\theta_c$的極大似然估計$\hat{\theta}_c$爲;
$\hat{\theta}_c=\argmin\limits_{\theta_c}LL(\theta_c)\tag{式12}$
所以下面只需求出使得對數似然函數$LL(\theta_c)$取到最大值的$\hat{\mu}_c$和$\hat{\sum}_c$,就求出了$\hat{\theta}_c$。
對$LL(\theta_c)$關於$\mu_c$求偏導：
$\begin{aligned} \cfrac{\partial{LL(\theta_c)}}{\partial{\mu_c}}&=\cfrac{\partial}{\partial{\mu_c}}[-\cfrac{Nd}{2}ln(2\pi)-\cfrac{N}{2}ln|\Sigma_c|-\cfrac{1}{2}\sum_{i=1}^{N}(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c)]\\ &=\cfrac{\partial}{\partial{\mu_c}}[-\cfrac{1}{2}\sum_{i=1}^{N}(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c)]]\\ &=-\cfrac{1}{2}\sum_{i=1}^{N}\cfrac{\partial}{\partial{\mu_c}}[(x_i-\mu_c)^T\Sigma_c^{-1}(x_i-\mu_c)]\\ &=-\cfrac{1}{2}\sum_{i=1}^{N}\cfrac{\partial}{\partial{\mu_c}}[(x_i^T-\mu_c^T)\Sigma_c^{-1}(x_i-\mu_c)]\\ &=-\cfrac{1}{2}\sum_{i=1}^{N}\cfrac{\partial}{\partial{\mu_c}}[(x_i^T-\mu_c^T)(\Sigma_c^{-1}x_i-\Sigma_c^{-1}\mu_c)]\\ &=-\cfrac{1}{2}\sum_{i=1}^{N}\cfrac{\partial}{\partial{\mu_c}}[x_i^T\Sigma_c^{-1}x_i-x_i^T\Sigma_c^T\mu_c-\mu_c^T\Sigma_c^{-1}x_i+\mu_c^T\Sigma_c^{-1}\mu_c] \end{aligned}\tag{式13}$
由於$x_i^T\Sigma_c^{-1}\mu_c$的計算結果爲標量，所以有:
$x_i^T\Sigma_c^{-1}\mu_c=(x_i^T\Sigma_c^{-1}\mu_c)^T=\mu_c^T(\Sigma_c^{-1})^Tx_i=\mu_c^T(\Sigma_c^T)^{-1}x_i=\mu_c^T\Sigma_c^{-1}x_i\tag{式14}$
所以（式13）可以進一步化爲:
$\cfrac{\partial{LL(\theta_c)}}{\partial{\mu_c}}= -\cfrac{1}{2}\sum_{i=1}^{N}\cfrac{\partial}{\partial{\mu_c}}[x_i^T\Sigma_c^{-1}x_i-2x_i^T\Sigma_c^{-1}\mu_c+\mu_c^T\Sigma_c^{-1}\mu_c]\tag{式15}$
由矩陣微分公式：
$\cfrac{\partial a^T x}{\partial x}=a,\quad \cfrac{\partial x^T \beta x}{\partial x}=(\beta+\beta^T)x\tag{式16}$
可以得到;
$\begin{aligned} \cfrac{\partial LL(\theta_c)}{\partial \mu_c}&= -\cfrac{1}{2}\sum_{i=1}^{N}[0-(2x_i^T\Sigma_c^{-1})^T+(\Sigma_c^{-1}+{\Sigma_c^{-1})}^T\mu_c]\\ &=-\cfrac{1}{2}\sum_{i=1}^{N}[-(2(\Sigma_c^{-1})^T x_i)+(\Sigma_c^{-1}+{\Sigma_c^{-1})}^T\mu_c]\\ &=-\cfrac{1}{2}\sum_{i=1}^{N}[-(2\Sigma_c^{-1}x_i)+2\Sigma_c^{-1}\mu_c]\\ &=\sum_{i=1}^{N}\Sigma_c^{-1}x_i-N\Sigma_c^{-1}\mu_c \end{aligned}\tag{式17}$
令偏導數爲0，得到:
$\begin{aligned} \cfrac{\partial LL(\theta_c)}{\partial \mu_c}&=\sum_{i=1}^{N}\Sigma_c^{-1}x_i-N\Sigma_c^{-1}\mu_c=0\\ &\Longrightarrow\sum_{i=1}^{N}\Sigma_c^{-1}x_i=N\Sigma_c^{-1}\mu_c\\ &\Longrightarrow\Sigma_c^{-1}\sum_{i=1}^{N}x_i=N\Sigma_c^{-1}\mu_c\\ &\Longrightarrow N\mu_c = \sum_{i=1}^{N}x_i\\ &\Longrightarrow \mu_c = \cfrac{1}{N}\sum_{i=1}^{N}x_i \end{aligned}\tag{式18}$
同樣的，對$LL(\theta_c)$關於$\Sigma_c$求偏導得到:
$\Sigma_c = \cfrac{1}{N}\sum_{i=1}^{N}(x_i-\mu_c)(x_i-\mu_c)^T\tag{式19}$
最小化分類錯誤率的貝葉斯最優分類器爲:
$h^*(x)=\argmax\limits_{c\in\mathcal{Y}}P(c|x)\tag{式20}$
又由貝葉斯定理可以知道：
$P(c|x)=\cfrac{P(x,c)}{P(x)}=\cfrac{P(c)P(x|c)}{P(x)}\tag{式21}$
所以：
$h^*(x)=\argmax\limits_{c\in{\mathcal{Y}}}\cfrac{P(c)P(x|c)}{P(x)}=\argmax\limits_{c\in\mathcal{Y}}P(c)P(x|c)\tag{式22}$
又由屬性條件獨立性假設：
$P(x|c) = P(x_1,x_2,\cdots,x_d|c) = \prod_{i=1}^{d}P(x_i|c)\tag{式23}$
所以：
$h^*(x)=\argmax\limits_{c\in\mathcal{Y}}P(c)\prod_{i=1}^{d}P(x_i|c)\tag{式24}$
這個就是樸素貝葉斯分類器的表達式。
對於$P(c)$,表示的是樣本空間中各類樣本所佔的比例，根據大數定律，當訓練集包含充足的度量同分布樣本的時候，$P(c)$可以通過各類樣本的頻率來進行估計，即:
$P(c)=\cfrac{|D_c|}{|D|}\tag{式25}$
其中，$D$表示訓練集，$|D|$表示樣本數，$D_c$表示訓練集中第$c$類樣本的數量組成的集合，$|D_c|$表示集合$D_c$的樣本個數。

貝葉斯分類器python應用

# 導入乳腺腫瘤數據
from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
# 打印處數據的keys
print(cancer.keys())

dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename'])

# 打印數據集中的標註好的腫瘤分類
print("腫瘤的分類:",cancer['target_names'])
print("腫瘤的特徵:",cancer['feature_names'])

腫瘤的分類: ['malignant' 'benign']
腫瘤的特徵: ['mean radius' 'mean texture' 'mean perimeter' 'mean area'
 'mean smoothness' 'mean compactness' 'mean concavity'
 'mean concave points' 'mean symmetry' 'mean fractal dimension'
 'radius error' 'texture error' 'perimeter error' 'area error'
 'smoothness error' 'compactness error' 'concavity error'
 'concave points error' 'symmetry error' 'fractal dimension error'
 'worst radius' 'worst texture' 'worst perimeter' 'worst area'
 'worst smoothness' 'worst compactness' 'worst concavity'
 'worst concave points' 'worst symmetry' 'worst fractal dimension']

可見，腫瘤的分類分爲:惡性（Malignant），良性（benign）,特徵值有很多。

# 將數據集的數值和分類目標賦值給X,y
X, y = cancer.data, cancer.target
# 導入數據拆分工具
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=38)
# 查看數據形態
print("訓練集形態:", X_train.shape)
print("測試集形態:", X_test.shape)

訓練集形態: (426, 30)
測試集形態: (143, 30)

# 導入高斯樸素貝葉斯
from sklearn.naive_bayes import GaussianNB

# 進行擬合數據
gnb = GaussianNB()
gnb.fit(X_train, y_train)
# 打印模型得分
print("模型得分:{:.3f}".format(gnb.score(X_test, y_test)))

模型得分:0.944