NLP：用Senta做文本情感分析

筆者在做Web敏感信息檢測功能時，需要用到NLP領域的文本情感分析，這裏用的是百度的預訓練模型Senta。

一、下載預訓練模型

百度提供的預訓練模型在GitHub上Senta，該模型所需環境是PaddlePaddle，這裏就不得不吐槽一下在線安裝的速度簡直不要太慢。

二、修改代碼

筆者的需求是我將一個網頁中含有敏感信息的語句存入一個文件中，然後利用Senta對文件中的所有數據進行預測，並返回預測的概率（接近1表示積極情感，接近0表示消極情感）。代碼如下：

# encoding=utf-8
import os
import time
import io
from datetime import datetime
import random
import jieba

# def sentiment_pro(all_data):
if __name__ == "__main__":
    all_data = [
            ['敏感詞','含有敏感詞的語句'],
            ['敏感字','含有敏感詞的語句']]
    start = datetime.now()
#     #將含有敏感詞的句子保存到一個文件中，用於預測
    basedir = os.path.abspath(os.path.dirname(__file__))
    file_name = basedir + '\\WebSensitiveContent\\' + str(datetime.now()).replace(":","").replace(".",'').replace("-","").replace(" ","") + str(random.randint(0,100)) + '.txt'
    with io.open(file_name, "w", encoding='utf8') as fin:
            for data_sentence in all_data:
                    seg_list = jieba.cut(data_sentence[1])  # 默認是精確模式
                    data_string = " ".join(seg_list)
                    fin.write('1'+'\t'+data_string+'\n')#第一列中寫個固定值1是因爲模型的輸入格式要求，與結果無關，可以是任意整數
    #調用另一個python文件，使用深度模型對生成的文件進行情感預測
    dir = "python ./Senta-master/sentiment_classify.py --test_data_path ./Senta-master/data/test_data/corpus.test --word_dict_path ./Senta-master/C-API/fluid-senti-classify_config/config/train.vocab --mode infer --model_path ./Senta-master/C-API/fluid-senti-classify_config/config/Senta/"
    result = os.popen(dir)
    res = result.read()#讀取返回結果
    #將檢測的結果保存到reslut_pos_score中，返回的結果是0到1之間的小數，越接近1代表情感越偏向積極，反正亦然
    reslut_pos_score = []
    for line in res.splitlines():
        print(line)
    os.remove(file_name)#刪除臨時生成的敏感詞文檔

    #加將正向敏感詞和負向敏感分別加入到一個list中，用於後面對敏感次的檢測
    PositiveSensitiveWords = list()
    NegativeSensitiveWords = list()

    with open('./Senta-master/mydict/na/party-department.txt', 'r', encoding='utf8') as f:
        for line in f.readlines():
                PositiveSensitiveWords.append(line.replace("\n",""))
    with open('./Senta-master/mydict/na/party-leader.txt', 'r', encoding='utf8') as f:
        for line in f.readlines():
                PositiveSensitiveWords.append(line.replace("\n",""))
    with open('./Senta-master/mydict/na/party-meeting.txt', 'r', encoding='utf8') as f:
        for line in f.readlines():
                PositiveSensitiveWords.append(line.replace("\n",""))
    with open('./Senta-master/mydict/na/reaction.txt', 'r', encoding='utf8') as f:
        for line in f.readlines():
                PositiveSensitiveWords.append(line.replace("\n",""))
    with open('./Senta-master/mydict/po/brute.txt', 'r', encoding='utf8') as f:
        for line in f.readlines():
                NegativeSensitiveWords.append(line.replace("\n",""))
    with open('./Senta-master/mydict/po/drug.txt', 'r', encoding='utf8') as f:
        for line in f.readlines():
                NegativeSensitiveWords.append(line.replace("\n",""))
    with open('./Senta-master/mydict/po/superstition.txt', 'r', encoding='utf8') as f:
        for line in f.readlines():
                NegativeSensitiveWords.append(line.replace("\n",""))
        
    i = 0 #用於確定當前敏感詞的索引值，以便於找到對應的情感分析結果
    result_need_remove = []#用於存儲通過情感判斷後不需要報警的敏感句子
    for SensitiveWord in all_data:
        if SensitiveWord[0] in PositiveSensitiveWords:#判斷是否爲正向敏感詞
                if float(reslut_pos_score[i]) > 0.6:#正向敏感詞 並且情感是積極的 則移除
                        print(SensitiveWord[0]+"：是正向敏感次" + SensitiveWord[1] + "情感得分爲："+str(reslut_pos_score[i]))
                        result_need_remove.append(SensitiveWord)
        elif SensitiveWord[0] in NegativeSensitiveWords:#判斷是否爲負向敏感詞
                if float(reslut_pos_score[i]) < 0.35:#負向敏感詞 並且情感是消極的 則移除
                        print(SensitiveWord[0]+"：是負向敏感次" + SensitiveWord[1] + "情感得分爲："+str(reslut_pos_score[i]))
                        result_need_remove.append(SensitiveWord)
        else:#中性或者是絕對敏感詞
                print(SensitiveWord[0]+"：是中性或者絕對敏感詞" + SensitiveWord[1] + "情感得分爲："+str(reslut_pos_score[i]))
        i = i + 1
    for need_remove in result_need_remove:
            all_data.remove(need_remove)
    end = datetime.now()
    print('Running time: %s Seconds'%(end-start))

 

該代碼文件存放的位置是sentiment_classify.py的上級目錄。結構如下圖：

筆者沒有展示訓練的過程，而是直接先用Senta也有的訓練模型。我看了一下訓練的數據量是1萬條。測試結果只能說還行吧，畢竟也就一萬條數據。