決策樹、隨機森林、網格搜索

決策樹系列之決策樹知識點
決策樹及決策樹生成與剪枝
【機器學習】 隨機森林（Random Forest）
機器學習（四）——模型調參利器 gridSearchCV（網格搜索）

筆記

決策樹的思想：把區分樣本效果好的特徵放在靠近根節點處。如何衡量一個特徵區分樣本的效果？採用信息增益率或基尼係數。
決策樹的剪枝：爲了防止過擬合現象

本案例來自唐宇迪機器學習課程，利用決策樹來對加州不同地區房價預測：

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn import tree
import pydotplus
from IPython.display import Image
housing = fetch_california_housing()
dtr = tree.DecisionTreeRegressor(max_depth = 2)
dtr.fit(housing.data[:, [6, 7]], housing.target)
dot_data = \
    tree.export_graphviz(
        dtr,
        out_file = None,
        feature_names = housing.feature_names[6:8],
        filled = True,
        impurity = False,
        rounded = True
    )
graph = pydotplus.graph_from_dot_data(dot_data)
graph.get_nodes()[7].set_fillcolor("#FFF2DD")
Image(graph.create_png())

僅採用6、7兩列數據（經緯度）作爲預測標準，將結果可視化展示如下：

隨機森林

隨機森林構建多棵決策樹，每棵樹有放回地採取部分樣本（60%-80%）的部分特徵，然後將所有樹結合成預測模型。隨機森林的預測效果顯然好於單一棵決策樹：

from sklearn.model_selection import train_test_split
data_train, data_test, target_train, target_test = \
    train_test_split(housing.data, housing.target, test_size = 0.1, random_state = 42)
dtr = tree.DecisionTreeRegressor(random_state = 42)
dtr.fit(data_train, target_train)
dtr.score(data_test, target_test)
# 0.637355881715626

from sklearn.ensemble import RandomForestRegressor
rfr = RandomForestRegressor( random_state = 42)
rfr.fit(data_train, target_train)
rfr.score(data_test, target_test)
# 0.8097021394052101

網格搜索

在給定的參數中，選擇最優的參數

from sklearn.model_selection import GridSearchCV
tree_param_grid = { 'min_samples_split': list((3,6,9)),'n_estimators':list((10,50,100))}
grid = GridSearchCV(RandomForestRegressor(),param_grid=tree_param_grid, cv=5)
grid.fit(data_train, target_train)
# 最優模型，最優參數，最優結果
grid.best_estimator_, grid.best_params_, grid.best_score_
# (RandomForestRegressor(bootstrap=True, ccp_alpha=0.0, criterion='mse',
#                        max_depth=None, max_features='auto', max_leaf_nodes=None,
#                        max_samples=None, min_impurity_decrease=0.0,
#                        min_impurity_split=None, min_samples_leaf=1,
#                        min_samples_split=3, min_weight_fraction_leaf=0.0,
#                        n_estimators=100, n_jobs=None, oob_score=False,
#                        random_state=None, verbose=0, warm_start=False),
#  {'min_samples_split': 3, 'n_estimators': 100},
#  0.8076100725804507)

# 利用此參數做預測
rfr = RandomForestRegressor( min_samples_split=3,n_estimators = 100,random_state = 42)
rfr.fit(data_train, target_train)
rfr.score(data_test, target_test)
# 0.8088623476993486

# 查看此模型中每個特徵的重要性
pd.Series(rfr.feature_importances_, index = housing.feature_names).sort_values(ascending = False)
# MedInc        0.524257
# AveOccup      0.137947
# Latitude      0.090622
# Longitude     0.089414
# HouseAge      0.053970
# AveRooms      0.044443
# Population    0.030263
# AveBedrms     0.029084
# dtype: float64