訓練模型填充空值(fill null)的幾種方法

我們在進行模型訓練時，不可避免的會遇到某些特徵出現空值的情況，下面整理了幾種填充空值的方法

1. 用固定值填充

對於特徵值缺失的一種常見的方法就是可以用固定值來填充，例如0，9999， -9999, 例如下面對灰度分這個特徵缺失值全部填充爲-99

data['灰度分'] = data['灰度分'].fillna('-99')

2. 用均值填充

對於數值型的特徵，其缺失值也可以用未缺失數據的均值填充，下面對灰度分這個特徵缺失值進行均值填充

data['灰度分'] = data['灰度分'].fillna(data['灰度分'].mean()))

3. 用衆數填充

與均值類似，可以用未缺失數據的衆數來填充缺失值

data['灰度分'] = data['灰度分'].fillna(data['灰度分'].mode()))

4. 用上下數據進行填充

用前一個數據進行填充

data['灰度分'] = data['灰度分'].fillna(method='pad')

用後一個數據進行填充

data['灰度分'] = data['灰度分'].fillna(method='bfill')

5. 用插值法填充

data['灰度分'] = data['灰度分'].interpolate()

6. 用KNN進行填充

from fancyimpute import BiScaler, KNN, NuclearNormMinimization, SoftImpute
dataset = KNN(k=3).complete(dataset)

7. random forest進行填充

from sklearn.ensemble import RandomForestRegressor
zero_columns_2 = ['機構查詢數量', '直接聯繫人數量', '直接聯繫人在黑名單數量', '間接聯繫人在黑名單數量',
                '引起黑名單的直接聯繫人數量', '引起黑名單的直接聯繫人佔比']
#將出現空值的除了預測的列全部取出來，不用於訓練                
dataset_list2 = [x for x in dataset if x not in zero_columns_2]
dataset_2 = dataset[dataset_list2]
# 取出灰度分不爲空的全部樣本進行訓練
know = dataset_2[dataset_2['灰度分'].notnull()]
print(know.shape) #26417, 54
# 取出灰度分爲空的樣本用於填充空值
unknow = dataset_2[dataset_2['灰度分'].isnull()]
print(unknow.shape) #2078, 54

y = ['灰度分']
x = [1]

know_x2 = know.copy()
know_y2 = know.copy()
print(know_y2.shape)
#
know_x2.drop(know_x2.columns[x], axis=1, inplace=True)
print(know_y2.shape)
print(know_x2.shape)
#
know_y2 = know[y]
# RandomForestRegressor
rfr = RandomForestRegressor(random_state=666, n_estimators=2000, n_jobs=-1)
rfr.fit(know_x2, know_y2)

# 填充爲空的樣本
unknow_x2 = unknow.copy()
unknow_x2.drop(unknow_x2.columns[x], axis=1, inplace=True)
print(unknow_x2.shape) #(2078, 53)
unknow_y2 = rfr.predict(unknow_x2)
unknow_y2 = pd.DataFrame(unknow_y2, columns=['灰度分'])                

使用fancyimpute包中的其他方法

# matrix completion using convex optimization to find low-rank solution
# that still matches observed values. Slow!
X_filled_nnm = NuclearNormMinimization().complete(X_incomplete)

# Instead of solving the nuclear norm objective directly, instead
# induce sparsity using singular value thresholding
X_filled_softimpute = SoftImpute().complete(X_incomplete_normalized)

# print mean squared error for the three imputation methods above
nnm_mse = ((X_filled_nnm[missing_mask] - X[missing_mask]) ** 2).mean()
# print mean squared error for the three imputation methods above
nnm_mse = ((X_filled_nnm[missing_mask] - X[missing_mask]) ** 2).mean()
print("Nuclear norm minimization MSE: %f" % nnm_mse)

softImpute_mse = ((X_filled_softimpute[missing_mask] - X[missing_mask]) ** 2).mean()
print("SoftImpute MSE: %f" % softImpute_mse)

knn_mse = ((X_filled_knn[missing_mask] - X[missing_mask]) ** 2).mean()
print("knnImpute MSE: %f" % knn_mse)

以上是我總結的幾種填充空值的方法，具體使用哪種方法可以根據具體的業務場景選用