python數據清洗（缺失值與異常值處理）

1。 將本地sql文件寫入mysql數據庫

本文寫入的是python數據庫的taob表

source [本地文件]

其中總數據爲9616行，列分別爲title,link,price,comment

2。使用python鏈接並讀取數據

查看數據概括

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
                        user='root',
                        passwd='123456',
                        db='python')#鏈接本地數據庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數據
print(data.describe())

說明數據的導入是正確的,簡單的分析發現問題並不是這麼簡單，因爲comment均值562可能偏大，最大評論數454037也可能出現錯誤，price價格爲0也不太可能出現。

 price        comment
count  9616.00000    9616.000000
mean     64.49324     562.239601
std     176.10901    6078.909643
min       0.00000       0.000000
25%      20.00000      16.000000
50%      36.00000      58.000000
75%      66.00000     205.000000
max    7940.00000  454037.000000

3。缺失值處理

將價格爲0的值設置爲中位數36

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
                        user='root',
                        passwd='123456',
                        db='python')#鏈接本地數據庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數據

data['price'][data['price']==0]=None
x = 0
for i in data.columns:
    for j in range(len(data)):
        if (data[i].isnull()) [j]:
            data[i][j]='36'
            x+=1
print(x)
#44

結果顯示修改了44行的數據。

4。異常值處理

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
                        user='root',
                        passwd='123456',
                        db='python')#鏈接本地數據庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數據
#缺失值處理
data['price'][data['price']==0]=None
x = 0
for i in data.columns:
    for j in range(len(data)):
        if (data[i].isnull()) [j]:
            data[i][j]='36'
            x+=1
print(x)
#異常值處理
#繪製散點圖,價格爲橫軸
data1 = data.T#轉置
price = data1.values[2]
comment = data1.values[3]
plt.plot(price,comment,'o')
plt.show()
#print(price)

結果如下圖，價格爲0左右時comment很大可能爲異常值，comments爲0時，價格極大這個有可能的。

接下來處理評論數異常值，假設異常值分割線設置爲20w，

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
                        user='root',
                        passwd='123456',
                        db='python')#鏈接本地數據庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數據
#缺失值處理
data['price'][data['price']==0]=None
x = 0
for i in data.columns:
    for j in range(len(data)):
        if (data[i].isnull()) [j]:
            data[i][j]='36'
            x+=1
print(x)
#異常值處理
da = data.values#重新賦值data
#異常值處理，將commments大於200000的數據comments設置爲58
cont_clou = len(da)#獲取行數
#遍歷數據進行處理
for i in range(0,cont_clou):
    if(data.values[i][3]>200000):
        #print(data.values[i][3])
        da[i][3]='58'
        #print(da[i][3])

#繪製散點圖,價格爲橫軸
data1 = da.T#轉置
price = data1[2]
comment = data1[3]
plt.plot(price,comment,'o')
plt.xlabel('price')
plt.ylabel('comments')
plt.show()

處理後的輸出結果爲：