https://github.com/facebookresearch/fastText
python版本
https://github.com/salestock/fastText.py
这个是非官方的版本 现在已经不在使用了
官方提供了Python版本
https://github.com/facebookresearch/fastText/tree/master/python
现在用的都是官方的版本
开始一直报错就是因为自己安装了官方的版本
却一直还在调用非官方的api
fasttext是facebook开源的一个词向量与文本分类工具，在学术上没有太多创新点，好处是模型简单，训练速度非常快。简单尝试可以发现，用起来还是非常顺手的，做出来的结果也不错，可以达到上线使用的标准。
简单说来，fastText做的事情，就是把文档中所有词通过lookup table变成向量，取平均后直接用线性分类器得到分类结果。fastText和ACL-15上的deep averaging network(DAN，如下图)比较相似，是一个简化的版本，去掉了中间的隐层。论文指出了对一些简单的分类任务，没有必要使用太复杂的网络结构就可以取得差不多的结果。
fastText结构
fastText论文中提到了两个tricks
- hierarchical softmax
- 类别数较多时，通过构建一个霍夫曼编码树来加速softmax layer的计算，和之前word2vec中的trick相同
- N-gram features
- 只用unigram的话会丢掉word order信息，所以通过加入N-gram features进行补充用hashing来减少N-gram的存储
fastText有监督学习(分类)示例
可以通过pip install fasttext安装包含fasttext python的接口的package
fastText做文本分类要求文本是如下的存储形式：
__label__2 , birchas chaim , yeshiva birchas chaim is a orthodox jewish mesivta high school in lakewood township new jersey . it was founded by rabbi shmuel zalmen stein in 2001 after his father rabbi chaim stein asked him to open a branch of telshe yeshiva in lakewood . as of the 2009-10 school year the school had an enrollment of 76 students and 6 . 6 classroom teachers ( on a fte basis ) for a student–teacher ratio of 11 . 5 1 .
__label__6 , motor torpedo boat pt-41 , motor torpedo boat pt-41 was a pt-20-class motor torpedo boat of the united states navy built by the electric launch company of bayonne new jersey . the boat was laid down as motor boat submarine chaser ptc-21 but was reclassified as pt-41 prior to its launch on 8 july 1941 and was completed on 23 july 1941 .
__label__11 , passiflora picturata , passiflora picturata is a species of passion flower in the passifloraceae family .
__label__13 , naya din nai raat , naya din nai raat is a 1974 bollywood drama film directed by a . bhimsingh . the film is famous as sanjeev kumar reprised the nine-role epic performance by sivaji ganesan in navarathri ( 1964 ) which was also previously reprised by akkineni nageswara rao in navarathri ( telugu 1966 ) . this film had enhanced his status and reputation as an actor in hindi cinema .
其中前面的__label__是前缀，也可以自己定义，__label__后接的为类别。
我们定义我们的5个类别分别为：
1:technology
2:car
3:entertainment
4:military
5:sports
生成文本格式
"""
https://github.com/facebookresearch/fastText
python版本
https://github.com/salestock/fastText.py
这个是非官方的版本 现在已经不在使用了
官方提供了Python版本
https://github.com/facebookresearch/fastText/tree/master/python
现在用的都是官方的版本
"""
import jieba
import pandas as pd
import random
cate_dic = {'technology':1, 'car':2, 'entertainment':3, 'military':4, 'sports':5}
df_technology = pd.read_csv("./data/technology_news.csv", encoding='utf-8')
df_technology = df_technology.dropna()
df_car = pd.read_csv("./data/car_news.csv", encoding='utf-8')
df_car = df_car.dropna()
df_entertainment = pd.read_csv("./data/entertainment_news.csv", encoding='utf-8')
df_entertainment = df_entertainment.dropna()
df_military = pd.read_csv("./data/military_news.csv", encoding='utf-8')
df_military = df_military.dropna()
df_sports = pd.read_csv("./data/sports_news.csv", encoding='utf-8')
df_sports = df_sports.dropna()
technology = df_technology.content.values.tolist()[1000:21000]
car = df_car.content.values.tolist()[1000:21000]
entertainment = df_entertainment.content.values.tolist()[:20000]
military = df_military.content.values.tolist()[:20000]
sports = df_sports.content.values.tolist()[:20000]
stopwords=pd.read_csv("data/stopwords.txt",index_col=False,quoting=3,sep="\t",names=['stopword'], encoding='utf-8')
stopwords=stopwords['stopword'].values
def preprocess_text(content_lines, sentences, category):
for line in content_lines:
try:
segs=jieba.lcut(line)
segs = filter(lambda x:len(x)>1, segs)
segs = filter(lambda x:x not in stopwords, segs)
sentences.append("__label__"+str(category)+" , "+" ".join(list(segs)))
except Exception:
print (line)
continue
"""
"""
#生成训练数据
sentences = []
preprocess_text(technology, sentences, cate_dic['technology'])
preprocess_text(car, sentences, cate_dic['car'])
preprocess_text(entertainment, sentences, cate_dic['entertainment'])
preprocess_text(military, sentences, cate_dic['military'])
preprocess_text(sports, sentences, cate_dic['sports'])
random.shuffle(sentences)
print ("writing data to fasttext format...")
out = open('train_data.txt', 'wb')
for sentence in sentences:
out.write(sentence.encode('utf8')+b"\n")
print("done!")
#将每个类别文档处理之后写入文件
开始训练 非常的快
"""
调用fastText训练生成模型
https://fasttext.cc/docs/en/python-module.html
意思以前官方没有提供python 现在官方提供了 所以２个要合并 那么我们现在安装的就是新的
https://github.com/facebookresearch/fastText/tree/master/python
"""
import fasttext
model = fasttext.train_supervised('train_data.txt')
"""
对模型效果进行评估
"""
def print_results(N, p, r):
print("N\t" + str(N))
print("P@{}\t{:.3f}".format(1, p))
print("R@{}\t{:.3f}".format(1, r))
print_results(*model.test('train_data.txt'))
"""
N 87584
P@1 0.974
R@1 0.974
"""
保存模型
"""
保存模型
"""
model.save_model("model_filename.bin")
加载模型
"""
加载模型
"""
m1= fasttext.load_model("model_filename.bin")
压缩，模型
"""
Compress model files with quantization
When you want to save a supervised model file,
fastText can compress it in order to have a much smaller model file by sacrificing only a little bit performance.
进行模型压缩 以适配更小的机器
"""
# with the previously trained `model` object, call :
m1.quantize(input='train_data.txt', retrain=True)
# then display results and save the new model :
#print_results(*model.test(valid_data))
model.save_model("model_filename.ftz")
预测
"""
实际预测
"""
label_to_cate = {'__label__1':'technology', '__label__2':'car','__label__3':'entertainment', '__label__4':'military', '__label__5':'sports'}
texts = ['中新网 日电 2018 预赛 亚洲区 强赛 中国队 韩国队 较量 比赛 上半场 分钟 主场 作战 中国队 率先 打破 场上 僵局 利用 角球 机会 大宝 前点 攻门 得手 中国队 领先']
labels = model.predict(texts)
print(labels)
#([['__label__5']], array([[0.99998939]]))