零基礎入門數據挖掘 - 二手車交易價格預測--EDA

EDA背景

• 賽題鏈接

零基礎入門數據挖掘 - 二手車交易價格預測

• EDA目的

1. 熟悉數據集，瞭解數據集，對數據集進行驗證
2. 瞭解變量間的相互關係以及變量與預測值之間的存在關係
3. 爲下一步的數據處理以及特徵工程奠定基礎

代碼及講解

• 引入數據科學以及可視化庫

#coding:utf-8
#導入warnings包，利用過濾器來實現忽略警告語句。
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import missingno as msno

• 讀取數據

## 1) 載入訓練集和測試集；
path = './datalab/231784/'
Train_data = pd.read_csv(path+'used_car_train_20200313.csv', sep=' ')
Test_data = pd.read_csv(path+'used_car_testA_20200313.csv', sep=' ')

• 特徵列
  name - 汽車編碼
  regDate - 汽車註冊時間
  model - 車型編碼
  brand - 品牌
  bodyType - 車身類型
  fuelType - 燃油類型
  gearbox - 變速箱
  power - 汽車功率
  kilometer - 汽車行駛公里
  notRepairedDamage - 汽車有尚未修復的損壞
  regionCode - 看車地區編碼
  seller - 銷售方
  offerType - 報價類型
  creatDate - 廣告發布時間
  price - 汽車價格
  v_0’, ‘v_1’, ‘v_2’, ‘v_3’, ‘v_4’, ‘v_5’, ‘v_6’, ‘v_7’, ‘v_8’, ‘v_9’, ‘v_10’, ‘v_11’, ‘v_12’, ‘v_13’,‘v_14’ 匿名特徵，包含v0-14在內15個匿名特徵
• 觀察數據

Train_data.head().append(Train_data.tail())

Train_data.shape

• 使用describe進行查看數據

Train_data.describe()

Train_data.info()

• 查看異常值與缺失值

Train_data.isnull().sum()

# nan可視化
missing = Train_data.isnull().sum()
missing = missing[missing > 0]
missing.sort_values(inplace=True)
missing.plot.bar()

• 異常值

Train_data['notRepairedDamage'].value_counts()

Train_data['notRepairedDamage'].replace('-', np.nan, inplace=True)

• 兩個傾斜特別嚴重的特徵值

Train_data["seller"].value_counts()

Train_data["offerType"].value_counts()

• 預測值price的分佈

## 1) 總體分佈概況（無界約翰遜分佈等）
import scipy.stats as st
y = Train_data['price']
plt.figure(1); plt.title('Johnson SU')
sns.distplot(y, kde=False, fit=st.johnsonsu)
plt.figure(2); plt.title('Normal')
sns.distplot(y, kde=False, fit=st.norm)
plt.figure(3); plt.title('Log Normal')
sns.distplot(y, kde=False, fit=st.lognorm)

結論： 價格不服從正態分佈，所以在進行迴歸之前，它必須進行轉換。雖然對數變換做得很好，但最佳擬合是無界約翰遜分佈

• 查看預測值price的具體頻數

## 2) 查看預測值的具體頻數
plt.hist(Train_data['price'], orientation = 'vertical',histtype = 'bar', color ='red')
plt.show()

結論： 大於20000得值極少，其實這裏也可以把這些當作特殊得值（異常值）直接用填充或者刪掉

• 分類特徵與數值特徵
  這個區別方式適用於沒有直接label coding的數據，這裏不適用，需要人爲根據實際含義來區分

numeric_features = ['power', 'kilometer', 'v_0', 'v_1', 'v_2', 'v_3', 'v_4', 'v_5', 'v_6', 'v_7', 'v_8', 'v_9', 'v_10', 'v_11', 'v_12', 'v_13','v_14' ]

categorical_features = ['name', 'model', 'brand', 'bodyType', 'fuelType', 'gearbox', 'notRepairedDamage', 'regionCode',]

• 數值特徵之間的相關性

f , ax = plt.subplots(figsize = (7, 7))
plt.title('Correlation of Numeric Features with Price',y=1,size=16)
sns.heatmap(correlation,square = True,  vmax=0.8)

• 數值特徵的分佈

## 3) 每個數字特徵得分佈可視化
f = pd.melt(Train_data, value_vars=numeric_features)
g = sns.FacetGrid(f, col="variable",  col_wrap=2, sharex=False, sharey=False)
g = g.map(sns.distplot, "value")

結論： 可以看出匿名特徵相對分佈均勻

• 多變量之間迴歸關係

## 5) 多變量互相迴歸關係可視化
fig, ((ax1, ax2), (ax3, ax4), (ax5, ax6), (ax7, ax8), (ax9, ax10)) = plt.subplots(nrows=5, ncols=2, figsize=(24, 20))
# ['v_12', 'v_8' , 'v_0', 'power', 'v_5',  'v_2', 'v_6', 'v_1', 'v_14']
v_12_scatter_plot = pd.concat([Y_train,Train_data['v_12']],axis = 1)
sns.regplot(x='v_12',y = 'price', data = v_12_scatter_plot,scatter= True, fit_reg=True, ax=ax1)

v_8_scatter_plot = pd.concat([Y_train,Train_data['v_8']],axis = 1)
sns.regplot(x='v_8',y = 'price',data = v_8_scatter_plot,scatter= True, fit_reg=True, ax=ax2)

v_0_scatter_plot = pd.concat([Y_train,Train_data['v_0']],axis = 1)
sns.regplot(x='v_0',y = 'price',data = v_0_scatter_plot,scatter= True, fit_reg=True, ax=ax3)

power_scatter_plot = pd.concat([Y_train,Train_data['power']],axis = 1)
sns.regplot(x='power',y = 'price',data = power_scatter_plot,scatter= True, fit_reg=True, ax=ax4)

v_5_scatter_plot = pd.concat([Y_train,Train_data['v_5']],axis = 1)
sns.regplot(x='v_5',y = 'price',data = v_5_scatter_plot,scatter= True, fit_reg=True, ax=ax5)

v_2_scatter_plot = pd.concat([Y_train,Train_data['v_2']],axis = 1)
sns.regplot(x='v_2',y = 'price',data = v_2_scatter_plot,scatter= True, fit_reg=True, ax=ax6)

v_6_scatter_plot = pd.concat([Y_train,Train_data['v_6']],axis = 1)
sns.regplot(x='v_6',y = 'price',data = v_6_scatter_plot,scatter= True, fit_reg=True, ax=ax7)

v_1_scatter_plot = pd.concat([Y_train,Train_data['v_1']],axis = 1)
sns.regplot(x='v_1',y = 'price',data = v_1_scatter_plot,scatter= True, fit_reg=True, ax=ax8)

v_14_scatter_plot = pd.concat([Y_train,Train_data['v_14']],axis = 1)
sns.regplot(x='v_14',y = 'price',data = v_14_scatter_plot,scatter= True, fit_reg=True, ax=ax9)

v_13_scatter_plot = pd.concat([Y_train,Train_data['v_13']],axis = 1)
sns.regplot(x='v_13',y = 'price',data = v_13_scatter_plot,scatter= True, fit_reg=True, ax=ax10)

• 類別特徵箱型圖

類別特徵箱形圖可視化

# 因爲 name和 regionCode的類別太稀疏了，這裏我們把不稀疏的幾類畫一下
categorical_features = ['model',
 'brand',
 'bodyType',
 'fuelType',
 'gearbox',
 'notRepairedDamage']
for c in categorical_features:
    Train_data[c] = Train_data[c].astype('category')
    if Train_data[c].isnull().any():
        Train_data[c] = Train_data[c].cat.add_categories(['MISSING'])
        Train_data[c] = Train_data[c].fillna('MISSING')

def boxplot(x, y, **kwargs):
    sns.boxplot(x=x, y=y)
    x=plt.xticks(rotation=90)

f = pd.melt(Train_data, id_vars=['price'], value_vars=categorical_features)
g = sns.FacetGrid(f, col="variable",  col_wrap=2, sharex=False, sharey=False, size=5)
g = g.map(boxplot, "value", "price")

本文內容均來自 鏈接: Datawhale 零基礎入門數據挖掘-Task2 數據分析.