目录:
* 项目工作流程
基本流程:
- 数据清洗与格式转换
- 探索性数据分析
- 特征工程
- 建立基础模型,尝试多种算法
- 模型调参
- 评估与测试
- 解释我们的模型
- 完成项目
一. 数据清洗与格式转换
import warnings
warning.filterwarnings('ignore')
import pandas as pd
import numpy as np
pd.set_option('display.max_columns', 60)
pd.options.mode.chained_assignment = None
# No warnings about setting value on copy of slice
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['font.size'] = 24
from IPython.core.pylabtools import figsize
import seaborn as sns
sns.set(font_scale = 2)
data = pd.read_csv('Energy_and_Water_Data_Disclosure_for_Local_Law_84_2017__Data_for_Calendar_Year_2016_.csv')
data.head()
1.1 数据类型与缺失值
data.info()
将Not Available
转换为np.nan
,再将部分数值型数据转换成float
data = data.replace({'Not Available': np.nan})
for col in list(data.columns):
if ('ft²' in col or 'kBtu' in col or 'Metric Tons CO2e' in col or 'kWh' in
col or 'therms' in col or 'gal' in col or 'Score' in col):
data[col] = data[col].astype(float)
data.describe()
1.2 缺失值处理
import missingno as msno
msno.matrix(data, figsize = (16, 5))
1.2.1 缺失值比例函数:
def missing_values_table(df):
mis_val = df.isnull().sum() # 总缺失值
mis_val_percent = 100 * df.isnull().sum() / len(df) # 缺失值比例
mis_val_table = pd.concat([mis_val, mis_val_percent], axis = 1) # 缺失值制成表格
mis_val_table_ren_columns = mis_val_table.rename(columns = {0:'Missing Values',
1:'% of Total Values'})
mis_val_table_ren_columns = mis_val_table_ren_columns[
mis_val_table_ren_columns.iloc[:,1] != 0].sort_values('% of Total Values',ascending=False).round(1)
# 缺失值比例列由大到小排序
print('Your selected dataframe has {} columns.\nThere are {} columns that have missing values.'.format(df.shape[1], mis_val_table_ren_columns.shape[0]))
# 打印缺失值信息
return mis_val_table_ren_columns
missing_values_table(data)
Your selected dataframe has 60 columns.
There are 46 columns that have missing values.
1.2.2 获取缺失值比例 > 50% 的列
missing_df = missing_values_table(data)
missing_columns = list(missing_df[missing_df['% of Total Values'] > 50].index)
print('We will remove %d columns.' % len(missing_columns))
Your selected dataframe has 60 columns.
There are 46 columns that have missing values.
We will remove 11 columns.
1.2.3 删除缺失值比例高于50%的列
data = data.drop(columns = list(missing_columns))
二. 探索性数据分析
Exploratory Data Analysis, 就是画图来理解数据。。。
2.1 单变量绘图
- 标签数据
data = data.rename(columns = {'ENERGY STAR Score': 'score'})
plt.figure(figsize = (8, 6))
plt.style.use('ggplot')
plt.hist(data['score'].dropna(), bins = 100, edgecolor = 'k')
plt.xlabel('Score'); plt.ylabel('Number of Buildings')
plt.title('Energy Star Score Distribution')
- Site EUI 特征
plt.style.use('ggplot')
plt.figure(figsize(8, 6))
plt.hist(data['Site EUI (kBtu/ft²)'].dropna(), bins = 20, edgecolor = 'black')
plt.xlabel('Site EUI'); plt.ylabel('Count'); plt.title('Site EUI Distribution')
data['Site EUI (kBtu/ft²)'].describe()
data['Site EUI (kBtu/ft²)'].dropna().sort_values().tail(10)
存在着一些特别大的值,这些可能是离群点或记录错误点,对我们结果会有一些影响的。
2.2 剔除离群点
离群点的选择可能需要再斟酌一些,这里选择的方法是extreme outlier。
- First Quartile − 3 ∗ Interquartile Range
- First Quartile + 3 ∗ Interquartile Range
first_quartile = data['Site EUI (kBtu/ft²)'].describe()['25%']
third_quartile = data['Site EUI (kBtu/ft²)'].describe()['75%']
iqr = third_quartile - first_quartile
data = data[(data['Site EUI (kBtu/ft²)'] > (first_quartile - 3 * iqr)) &
(data[['Site EUI (kBtu/ft²)'] < (third_quartile + 3 * iqr))]
plt.figure(figsize = (8, 6))
plt.hist(data['Site EUI (kBtu/ft²)'].dropna(), bins = 50, edgecolor = 'black')
plt.xlabel('Site EUI'); plt.ylabel('Count'); plt.title('Site EUI Distribution')
2.3 观察哪些变量会对结果产生影响
选择大于80条数据的
Lput = data.dropna(subset = ['score'])['Largest Property Use Type'].value_counts()
Lput = list(Lput[Lput.values > 80].index)
plt.figure(figsize = (12, 10))
for lput in Lput:
subset = data[data['Largest Property Use Type'] == lput]
sns.kdeplot(subset['score'].dropna(), label = lput, shade = False, alpha = 0.8)
plt.xlabel('Energy Star Score', fontsize = 18)
plt.ylabel('Density', fontsize = 18)
plt.title('Density Plot of Energy Star Scores by Building Type', size = 24)
不同类型的建筑看起来对结果的影响是不一样的,所以我们需要充分利用这个变量的!
boroughs = data.dropna(subset = ['score'])['Borough'].value_counts()
boroughs = list(boroughs[boroughs.values > 150].index)
plt.figure(figsize = (12, 10))
for borough in boroughs:
subset = data[data['Borough'] == borough]
sns.kdeplot(subset['score'].dropna(), label = borough)
plt.xlabel('Energy Star Score', fontsize = 18)
plt.ylabel('Density', fontsize = 18)
plt.title('Density Plot of Energy Star Scores by Borough', fontsize = 24)
对于镇区这个特征来说看起来影响就不大,因为这几条线都差不多。
2.4 特征和标签之间的相关性
Pearson相关系数,帮助我们来筛选特征
corr_data = data.corr()['score'].sort_values()
print(corr_data.head(15), '\n')
print(corr_data.tail(15))
Site EUI (kBtu/ft²)
和 Weather Normalized Site EUI (kBtu/ft²)
呈现出明显的负相关,单位用电量越多,能源利用得分越低。
还需要在考虑下非线性变换的特征,比如平方,log等等,都可以来试试,对于类别变量还可以用one-hot encode来转换下。
2.4.* 特征变换与 one-hot encode
numeric_subset = data.select_dtypes('number') # 选择数值型列
for col in numeric_subset.columns: # 对数值型列开平方根和对数, 创建新的列
if col == 'score':
next
else:
numeric_subset['sqrt_' + col] = np.sqrt(numeric_subset[col])
numeric_subset['log_' + col] = np.log(numeric_subset[col])
categorical_subset = data[['Borough', 'Largest Property Use Type']] # 选择类别型列
categorical_subset = pd.get_dummies(categorical_subset) # One hot encode
features = pd.concat([numeric_subset, categorical_subset], axis = 1) # concat两个类型数据
features = features.dropna(subset = ['score']) # 删除标签列中的缺失值行
correlations = features.corr()['score'].dropna().sort_values() # 标签的相关系数
correlations.head(15)
correlations.tail(15)
2.5 双变量绘图
plt.figure(figsize = (12, 10))
features['Largest Property Use Type'] = data.dropna(subset =['score'])['Largest Property Use Type']
# 提取建筑类型特征
features = features[features['Largest Property Use Type'].isin(Lput)]
# Limit to building types with more than 80 observations
sns.lmplot('Site EUI (kBtu/ft²)', 'score', hue = 'Largest Property Use Type',
data = features, scatter_kws = {'alpha':0.7, 's':50}, fit_reg = False,
height = 12, aspect = 1.2)
plt.xlabel('Site EUI', fontsize = 24)
plt.ylabel('Energy Star Score', fontsize = 24)
plt.title('Energy Star Score vs Site EUI', fontsize = 30)
2.6 Pairs Plot
plot_data = features[['score', 'Weather Normalized Source EUI (kBtu/ft²)',
'Site EUI (kBtu/ft²)', 'sqrt_Source EUI (kBtu/ft²)']]
plot_data = plot_data.replace({np.inf: np.nan, -np.inf: np.nan}) # 无穷大和负无穷大替换为nan
plot_data = plot_data.rename(columns = {'Site EUI (kBtu/ft²)': 'Site EUI',
'sqrt_Source EUI (kBtu/ft²)': 'sqrt Source EUI',
'Weather Normalized Source EUI (kBtu/ft²)': 'Weather Norm EUI'})
plot_data = plot_data.dropna()
def corr_func(x, y, **kwargs):
r = np.corrcoef(x, y)[0][1] # x和y的皮尔逊相关系数
ax = plt.gca()
ax.annotate('r = {:.2f}'.format(r), xy = (.2, .8), xycoords=ax.transAxes, size=30)
grid = sns.PairGrid(data = plot_data, height = 4)
grid.map_upper(plt.scatter, alpha = 0.6)
grid.map_diag(plt.hist, edgecolor = 'black')
grid.map_lower(corr_func)
grid.map_lower(sns.kdeplot, cmap = plt.cm.Reds)
plt.suptitle('Pairs Plot of Energe Data', fontsize = 28, y = 1.05)
三. 特征工程与特征筛选
一般情况下我们分两步走:特征工程与特征筛选:
- 特征工程:概括性来说就是尽可能的多在数据中提取特征,各种数值变换,特征组合,分解等各种手段齐上阵。
- 特征选择:就是找到最有价值的那些特征作为我们模型的输入,但是之前做了那么多,可能有些是多余的,有些还没被发现,所以这俩阶段都是一个反复在更新的过程。比如我在建模之后拿到了特征重要性,这就为特征选择做了参考,有些不重要的我可以去掉,那些比较重要的,我还可以再想办法让其做更多变换和组合来促进我的模型。所以特征工程并不是一次性就能解决的,需要通过各种结果来反复斟酌。
3.1 特征变换 与 One-hot encode
同2.4.* 特征变换与 one-hot encode
features = data.copy()
numeric_subset = data.select_dtypes('number')
for col in numeric_subset.columns:
if col == 'score':
next
else:
numeric_subset['log_' + col] = np.log(numeric_subset[col])
categorical_subset = data[['Borough', 'Largest Property Use Type']]
categorical_subset = pd.get_dummies(categorical_subset)
features = pd.concat([numeric_subset, categorical_subset], axis = 1)
features.shape
(11319, 110)
3.2 共线特征
在数据中Site EUI 和 Weather Norm EUI就是要考虑的目标,他俩描述的基本是同一个事
plot_data = data[['Weather Normalized Site EUI (kBtu/ft²)', 'Site EUI (kBtu/ft²)']].dropna()
plt.plot(plot_data['Site EUI (kBtu/ft²)'], plot_data['Weather Normalized Site EUI (kBtu/ft²)'], 'bo')
plt.xlabel('Site EUI'); plt.ylabel('Weather Norm EUI')
plt.title('Weather Norm EUI vs Site EUI, R = %.4f' % np.corrcoef(data[['Weather Normalized Site EUI (kBtu/ft²)', 'Site EUI (kBtu/ft²)']].dropna(), rowvar=False)[0][1])
3.3 剔除共线特征
def remove_collinear_features(x, threshold):
'''
Objective:
Remove collinear features in a dataframe with a correlation coefficient
greater than the threshold. Removing collinear features can help a model
to generalize and improves the interpretability of the model.
Inputs:
threshold: any features with correlations greater than this value are removed
Output:
dataframe that contains only the non-highly-collinear features
'''
y = x['score']
x = x.drop(columns = ['score'])
corr_matrix = x.corr()
iters = range(len(corr_matrix.columns) - 1)
drop_cols = []
for i in iters:
for j in range(i):
item = corr_matrix.iloc[j: (j+1), (i+1): (i+2)]
col = item.columns
row = item.index
val = abs(item.values)
if val >= threshold:
# print(col.values[0], "|", row.values[0], "|", round(val[0][0], 2))
drop_cols.append(col.values[0])
drops = set(drop_cols)
# print(drops)
x = x.drop(columns = drops)
x = x.drop(columns = ['Weather Normalized Site EUI (kBtu/ft²)',
'Water Use (All Water Sources) (kgal)',
'log_Water Use (All Water Sources) (kgal)',
'Largest Property Use Type - Gross Floor Area (ft²)'])
x['score'] = y
return x
features = remove_collinear_features(features, 0.6)
features = features.dropna(axis = 1, how = 'all')
print(features.shape)
features.head()
(11319, 65)
3.4 数据集划分
no_score = features[features['score'].isna()]
score = features[features['score'].notnull()]
print('no_score.shape: ', no_score.shape)
print('score.shape', score.shape)
from sklearn.model_selection import train_test_split
features = score.drop(columns = 'score')
labels = pd.DataFrame(score['score'])
features = features.replace({np.inf: np.nan, -np.inf: np.nan})
X, X_test, y, y_test = train_test_split(features, labels, test_size = 0.3, random_state = 42)
print(X.shape)
print(X_test.shape)
print(y.shape)
print(y_test.shape)
no_score.shape: (1858, 65)
score.shape: (9461, 65)
(6622, 64)
(2839, 64)
(6622, 1)
(2839, 1)
3.5 建立一个Baseline
在建模之前,我们得有一个最坏的打算,就是模型起码得有点作用才行。
# 衡量标准: Mean Absolute Error
def mae(y_true, y_pred):
return np.mean(abs(y_true - y_pred))
baseline_guess = np.median(y)
print('The baseline guess is a score of %.2f' % baseline_guess)
print('Baseline Performance on the test set: MAE = %.4f' % mae(y_test, baseline_guess))
The baseline guess is a score of 66.00
Baseline Performance on the test set: MAE = 24.5164
* 保存结果
no_score.to_csv('data/no_score.csv', index = False)
X.to_csv('data/training_features.csv', index = False)
X_test.to_csv('data/testing_features.csv', index = False)
y.to_csv('data/training_labels.csv', index = False)
y_test.to_csv('data/testing_labels.csv', index = False)
- 未完待续:
建模
与分析