垃圾郵件分類 jieba+bayes

簡介

本身我對文本方面的比如自然語言處理什麼的鑽研的不多，這裏是我之前寫的郵件分類，用的方法其實是很簡單的算法，同時這種處理方式可以說是最常用的文本處理技巧。

下下來一個是爲了自己記錄一下，當然如果您剛剛入門機器學習或者NLP，能給您一些幫助也最好不過了。

數據集

1. 垃圾郵件

2. 普通郵件

當然還有測試集：

這裏我使用的數據集中的部分截圖。

代碼

導入庫函數

import jieba
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

讀取數據

停用詞文檔。

stopWordFilePath = 'stopword.txt'
stopList = [word.strip() for word in open(stopWordFilePath, encoding='utf-8').readlines()]
stopList[:4]

ham = []
spam = []
with open('ham_100.utf8', 'r', encoding='utf-8') as f:
    for line in f.readlines():
        ham.append(line)
with open('spam_100.utf8', 'r', encoding='utf-8') as f:
    for line in f.readlines():
        spam.append(line)

ham[:2],spam[:2]

## 讀取測試集
test_data = []
with open('test.utf8', 'r', encoding='utf-8') as f:
    for line in f.readlines():
        test_data.append(line)
cut_test_data = [jieba.cut(item) for item in test_data]
test_data_fin = [' '.join(word) for word in cut_test_data]

分詞並去掉停用詞

cut_ham = [jieba.cut(sentence=str0) for str0 in ham]
train_ham = [' '.join(item) for item in cut_ham]
train_ham[:3]

cut_spam = [jieba.cut(str0) for str0 in spam]
train_spam = [' '.join(item) for item in cut_spam]
train_spam[:4]

這裏使用詞袋處理停用詞。

count = CountVectorizer(stop_words = stopList)
train_X_tmp = train_spam + train_ham
count.fit(train_spam + train_ham + test_data_fin)
train_X = count.transform(train_X_tmp).toarray()

train_X.shape

添加標記 1 ： 是垃圾郵件 0 ： 不是垃圾郵件

train_y_spam = [1 for i in range(len(train_spam))]
train_y_ham = [0 for i in range(len(train_ham))]

合併數據集

train_y_spam = [1 for i in range(len(train_spam))]
train_y_ham = [0 for i in range(len(train_ham))]

模型構建

mnb = MultinomialNB()
mnb.fit(train_X, train_y)
y_pred = mnb.predict(test_x)
y_pred

for lab, (index, i) in zip(y_pred, enumerate(test_data)):
    print(lab, "第", index+1, "封", i[:20])

算是做個筆記，大家共勉~~