简单构建新闻数据对股票的情绪因子（大盘因子）

简单思路描述：根据前一天的新闻数据，预测后一天大盘涨跌，涨为1，跌为0.
构建数据集：

import tushare as ts
ts.set_token(' ')
#ts.set_token('your token here')
pro = ts.pro_api()
df1 = pro.cctv_news(date='20190916')#0
df2 = pro.cctv_news(date='20190917')#1

此步骤为测试版，真正使用需考虑星期五星期六星期天的新闻数据，以及节假日数据合并。并且标签使用大盘数据确定而不是手工敲定。
完整测试代码：

all = df1.append(df2, ignore_index=True)
all['words'] = all['content'].apply(lambda s: list(jieba.cut(s))) #调用结巴分词
import numpy as np
import pandas as pd
import jieba

maxlen = 100 #截断词数
min_count = 1 #出现次数少于该值的词扔掉。这是最简单的降维方法

content = []
for i in all_['words']:
	content.extend(i)

abc = pd.Series(content).value_counts()
abc = abc[abc >= min_count]
abc[:] = list(range(1, len(abc)+1))
abc[''] = 0 #添加空字符串用来补全
word_set = set(abc.index)

def doc2num(s, maxlen): 
    s = [i for i in s if i in word_set]
    s = s[:maxlen] + ['']*max(0, maxlen-len(s))
    return list(abc[s])

all['doc2num'] = all['words'].apply(lambda s: doc2num(s, maxlen))

#手动打乱数据
idx = list(range(len(all_)))
np.random.shuffle(idx)
all = all.loc[idx]

model = Sequential()
model.add(Embedding(len(abc), 256, input_length=maxlen))
model.add(Dropout(0.5))
model.add(Dense(128))
#model.add(Bidirectional(LSTM(128))
model.add(Bidirectional(LSTM(128, return_sequences=True),merge_mode='concat'))
#model.add(Bidirectional(LSTM(16))
#model.add(LSTM(128)) 
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(64))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

batch_size = 128
train_num = 15000

model.fit(x, y, batch_size = batch_size, nb_epoch=10)

model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_8 (Embedding)      (None, 100, 256)          598784    
_________________________________________________________________
dropout_12 (Dropout)         (None, 100, 256)          0         
_________________________________________________________________
dense_9 (Dense)              (None, 100, 128)          32896     
_________________________________________________________________
bidirectional_4 (Bidirection (None, 100, 256)          263168    
_________________________________________________________________
dropout_13 (Dropout)         (None, 100, 256)          0         
_________________________________________________________________
flatten_2 (Flatten)          (None, 25600)             0         
_________________________________________________________________
dense_10 (Dense)             (None, 64)                1638464   
_________________________________________________________________
dense_11 (Dense)             (None, 1)                 65        
_________________________________________________________________
activation_8 (Activation)    (None, 1)                 0         
=================================================================
Total params: 2,533,377
Trainable params: 2,533,377
Non-trainable params: 0
__________________________

之所以构建深度较大，是为了方便后续使用，如果简单使用可以直接lstm层就行，把bilstm去掉。

后续改进：情绪因子构建为0,1,-1三个，分别为大盘跌幅位于0.2到-0.2，高于0.2，低于-0.2等方面。并且可以在分词中,加入词向量或者tfidf权重等方法。

后续文章将陆续更新：事件主体抽取，金融词语发现等nlp文章