半小時拿下Python數據處理之Pandas篇

import pandas as pd

Pandas數據結構

Series

Series一維的數據結構

通過list構建Series

ser_obj =pd.Series(range(10,15))
print(type(ser_obj)) # <class 'pandas.core.series.Series'>
print(ser_obj)
<class 'pandas.core.series.Series'>
0    10
1    11
2    12
3    13
4    14
dtype: int32

獲取數據

print(type(ser_obj.values)) # <class 'numpy.ndarray'>
print(ser_obj.values) # [10 11 12 13 14]
<class 'numpy.ndarray'>
[10 11 12 13 14]

獲取索引

print(type(ser_obj.index)) # <class 'pandas.core.indexes.range.RangeIndex'>
print(ser_obj.index) # RangeIndex(start=0, stop=5, step=1)
<class 'pandas.core.indexes.range.RangeIndex'>
RangeIndex(start=0, stop=5, step=1)

注意索引對象不可變

# 索引對象不可變
ser_obj.index[0] = 2
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-53-ce46badf9dd7> in <module>()
----> 1 ser_obj.index[0] = 2


G:\Anaconda3\lib\site-packages\pandas\core\indexes\base.py in __setitem__(self, key, value)
   1668
   1669     def __setitem__(self, key, value):
-> 1670         raise TypeError("Index does not support mutable operations")
   1671
   1672     def __getitem__(self, key):


TypeError: Index does not support mutable operations

預覽數據

print(ser_obj.head(3))
0    10
1    11
2    12
dtype: int32

通過索引獲取數據

print(ser_obj[0]) # 10
10

索引與數據的對應關係仍保持在數組運算的結果中

print(ser_obj > 12)
print(ser_obj[ser_obj > 12])
0    False
1    False
2    False
3     True
4     True
dtype: bool
3    13
4    14
dtype: int32

整合代碼

# 通過list構建Series
ser_obj =pd.Series(range(10,15))
print(type(ser_obj)) # <class 'pandas.core.series.Series'>
print(ser_obj)

# 獲取數據
print(type(ser_obj.values)) # <class 'numpy.ndarray'>
print(ser_obj.values) # [10 11 12 13 14]

# 獲取索引
print(type(ser_obj.index)) # <class 'pandas.core.indexes.range.RangeIndex'>
print(ser_obj.index) # RangeIndex(start=0, stop=5, step=1)

# 預覽數據
print(ser_obj.head(3))

#通過索引獲取數據
print(ser_obj[0]) # 10

# 索引與數據的對應關係仍保持在數組運算的結果中
print(ser_obj > 12)
print(ser_obj[ser_obj > 12])
<class 'pandas.core.series.Series'>
0    10
1    11
2    12
3    13
4    14
dtype: int32
<class 'numpy.ndarray'>
[10 11 12 13 14]
<class 'pandas.core.indexes.range.RangeIndex'>
RangeIndex(start=0, stop=5, step=1)
0    10
1    11
2    12
dtype: int32
10
0    False
1    False
2    False
3     True
4     True
dtype: bool
3    13
4    14
dtype: int32

通過dict構建Series(注意:字典的key自動作爲索引)

year_data = {2001: 17.8, 2002: 20.1, 2003: 16.5}
ser_obj2 = pd.Series(year_data)
print(type(ser_obj2)) # <class 'pandas.core.series.Series'>
print(ser_obj2)
<class 'pandas.core.series.Series'>
2001    17.8
2002    20.1
2003    16.5
dtype: float64

獲取數據

print(type(ser_obj2.values)) # <class 'numpy.ndarray'>
print(ser_obj2.values) # [ 17.8  20.1  16.5]
<class 'numpy.ndarray'>
[ 17.8  20.1  16.5]

獲取索引

print(type(ser_obj2.index)) # <class 'pandas.core.indexes.numeric.Int64Index'>
print(ser_obj2.index) # Int64Index([2001, 2002, 2003], dtype='int64')
<class 'pandas.core.indexes.numeric.Int64Index'>
Int64Index([2001, 2002, 2003], dtype='int64')

預覽數據(head()不加參數則顯示全部

print(ser_obj2.head())
2001    17.8
2002    20.1
2003    16.5
dtype: float64

通過索引獲取數據

print(ser_obj2[2001]) # 17.8
17.8

整合代碼

# 通過dict構建Series(注意:字典的key自動作爲索引)
year_data = {2001: 17.8, 2002: 20.1, 2003: 16.5}
ser_obj2 = pd.Series(year_data)
print(type(ser_obj2)) # <class 'pandas.core.series.Series'>
print(ser_obj2)

# 獲取數據
print(type(ser_obj2.values)) # <class 'numpy.ndarray'>
print(ser_obj2.values) # [ 17.8  20.1  16.5]

# 獲取索引
print(type(ser_obj2.index)) # <class 'pandas.core.indexes.numeric.Int64Index'>
print(ser_obj2.index) # Int64Index([2001, 2002, 2003], dtype='int64')

# 預覽數據(head()不加參數則顯示全部)
print(ser_obj2.head())

#通過索引獲取數據
print(ser_obj2[2001]) # 17.8
<class 'pandas.core.series.Series'>
2001    17.8
2002    20.1
2003    16.5
dtype: float64
<class 'numpy.ndarray'>
[ 17.8  20.1  16.5]
<class 'pandas.core.indexes.numeric.Int64Index'>
Int64Index([2001, 2002, 2003], dtype='int64')
2001    17.8
2002    20.1
2003    16.5
dtype: float64
17.8

DataFrame

一個Dataframe就是一張表格,Series表示的是一維數組,Dataframe則是一個二維數組,可以類比成一張excelspreadsheet也可以把 Dataframe當做一組Series的集合

通過ndarray構建DataFrame

import numpy as np

# 通過ndarray構建DataFrame
array = np.random.randn(5,4)
print(array)

df_obj = pd.DataFrame(array)
print(df_obj.head())
[[ 0.7346628  -1.13733651  0.72853785  0.38743511]
 [ 0.49549724  3.96998008  1.13567695 -0.21425912]
 [ 0.22094222  0.7766603   0.46086182  0.33199643]
 [-0.46279419  0.85898771  0.41993259 -0.61997791]
 [-0.83296535  1.19450707 -1.45531366 -0.13990243]]
          0         1         2         3
0  0.734663 -1.137337  0.728538  0.387435
1  0.495497  3.969980  1.135677 -0.214259
2  0.220942  0.776660  0.460862  0.331996
3 -0.462794  0.858988  0.419933 -0.619978
4 -0.832965  1.194507 -1.455314 -0.139902

通過dict構建DataFrame

dict_data = {'A': 1.,
             'B': pd.Timestamp('20180316'),
             'C': pd.Series(1, index=list(range(4)),dtype='float32'),
             'D': np.array([3] * 4,dtype='int32'),
             'E' : pd.Categorical(["Python","Java","C++","C#"])
            }
print(dict_data)
df_obj2 = pd.DataFrame(dict_data)
print(df_obj2.head())
{'A': 1.0, 'B': Timestamp('2018-03-16 00:00:00'), 'C': 0    1.0
1    1.0
2    1.0
3    1.0
dtype: float32, 'D': array([3, 3, 3, 3]), 'E': [Python, Java, C++, C#]
Categories (4, object): [C#, C++, Java, Python]}
     A          B    C  D       E
0  1.0 2018-03-16  1.0  3  Python
1  1.0 2018-03-16  1.0  3    Java
2  1.0 2018-03-16  1.0  3     C++
3  1.0 2018-03-16  1.0  3      C#

通過列索引獲取列數據

print(df_obj2['A'])
print(type(df_obj2['A']))

print(df_obj2.A)
0    1.0
1    1.0
2    1.0
3    1.0
Name: A, dtype: float64
<class 'pandas.core.series.Series'>
0    1.0
1    1.0
2    1.0
3    1.0
Name: A, dtype: float64

通過行索引(.loc)獲取行數據

print(df_obj2.loc[0])
print(type(df_obj2.loc[0]))
A                      1
B    2018-03-16 00:00:00
C                      1
D                      3
E                 Python
Name: 0, dtype: object
<class 'pandas.core.series.Series'>

增加列

df_obj2['F'] = df_obj2['D'] + 4
print(df_obj2.head())
     A          B    C  D       E  F
0  1.0 2018-03-16  1.0  3  Python  7
1  1.0 2018-03-16  1.0  3    Java  7
2  1.0 2018-03-16  1.0  3     C++  7
3  1.0 2018-03-16  1.0  3      C#  7

刪除列

del(df_obj2['F'] )
print(df_obj2.head())
     A          B    C  D       E
0  1.0 2018-03-16  1.0  3  Python
1  1.0 2018-03-16  1.0  3    Java
2  1.0 2018-03-16  1.0  3     C++
3  1.0 2018-03-16  1.0  3      C#

整合代碼

import numpy as np

# 通過ndarray構建DataFrame
array = np.random.randn(5,4)
print(array)

# 通過dict構建DataFrame
df_obj = pd.DataFrame(array)
print(df_obj.head())

dict_data = {'A': 1.,
             'B': pd.Timestamp('20180316'),
             'C': pd.Series(1, index=list(range(4)),dtype='float32'),
             'D': np.array([3] * 4,dtype='int32'),
             'E' : pd.Categorical(["Python","Java","C++","C#"])
            }
print(dict_data)
df_obj2 = pd.DataFrame(dict_data)
print(df_obj2.head())

# 通過列索引獲取列數據
print(df_obj2['A'])
print(type(df_obj2['A']))

print(df_obj2.A)

# 通過行索引獲取行數據
print(df_obj2.loc[0])
print(type(df_obj2.loc[0]))

# 增加列
df_obj2['G'] = df_obj2['D'] + 4
print(df_obj2.head())

# 刪除列
del(df_obj2['G'] )
print(df_obj2.head())
[[ 0.23758715 -1.13751056 -0.0863061  -0.71309414]
 [ 0.08129935  1.32099551 -0.27057527  0.49270974]
 [ 0.96111551  1.08307556  1.5094844   0.96117055]
 [-0.31003598  1.33959047 -0.42150857 -1.20605423]
 [ 0.12655879 -1.01810288 -1.34025171  0.98758417]]
          0         1         2         3
0  0.237587 -1.137511 -0.086306 -0.713094
1  0.081299  1.320996 -0.270575  0.492710
2  0.961116  1.083076  1.509484  0.961171
3 -0.310036  1.339590 -0.421509 -1.206054
4  0.126559 -1.018103 -1.340252  0.987584
{'A': 1.0, 'B': Timestamp('2018-03-16 00:00:00'), 'C': 0    1.0
1    1.0
2    1.0
3    1.0
dtype: float32, 'D': array([3, 3, 3, 3]), 'E': [Python, Java, C++, C#]
Categories (4, object): [C#, C++, Java, Python]}
     A          B    C  D       E
0  1.0 2018-03-16  1.0  3  Python
1  1.0 2018-03-16  1.0  3    Java
2  1.0 2018-03-16  1.0  3     C++
3  1.0 2018-03-16  1.0  3      C#
0    1.0
1    1.0
2    1.0
3    1.0
Name: A, dtype: float64
<class 'pandas.core.series.Series'>
0    1.0
1    1.0
2    1.0
3    1.0
Name: A, dtype: float64
A                      1
B    2018-03-16 00:00:00
C                      1
D                      3
E                 Python
Name: 0, dtype: object
<class 'pandas.core.series.Series'>
     A          B    C  D       E  G
0  1.0 2018-03-16  1.0  3  Python  7
1  1.0 2018-03-16  1.0  3    Java  7
2  1.0 2018-03-16  1.0  3     C++  7
3  1.0 2018-03-16  1.0  3      C#  7
     A          B    C  D       E
0  1.0 2018-03-16  1.0  3  Python
1  1.0 2018-03-16  1.0  3    Java
2  1.0 2018-03-16  1.0  3     C++
3  1.0 2018-03-16  1.0  3      C#

Pandas 數據操作

import pandas as pd

Series索引

ser_obj = pd.Series(range(5), index = ['a', 'b', 'c', 'd', 'e'])
ser_obj.head()
a    0
b    1
c    2
d    3
e    4
dtype: int32

行索引

# 行索引
ser_obj['a'] #等同描述ser_obj[0]
0

切片索引可以按照默認索引號,也可以按照實際索引值

# 切片索引(按索引號)
ser_obj[1:3] #python索引默認是左閉右開
b    1
c    2
dtype: int32
# 切片索引(按索引值)
ser_obj['b':'d']
b    1
c    2
d    3
dtype: int32

不連續索引,同樣可以按照默認索引號,也可以按照實際索引值

# 不連續索引表達一(按索引號)
ser_obj[[0, 2, 4]]
a    0
c    2
e    4
dtype: int32
# 不連續索引表達二(按索引值)
ser_obj[['a', 'e']]
a    0
e    4
dtype: int32

布爾索引

# 布爾索引
ser_bool = ser_obj > 2
print(ser_bool)
print()
print(ser_obj[ser_bool])
print()
print(ser_obj[ser_obj > 2])
a    False
b    False
c    False
d     True
e     True
dtype: bool

d    3
e    4
dtype: int32

d    3
e    4
dtype: int32

DataFrame索引

import numpy as np

df_obj = pd.DataFrame(np.random.randn(5,4), columns = ['a', 'b', 'c', 'd'])
df_obj.head()
a b c d
0 0.983790 1.063804 0.854634 -1.269025
1 0.161653 -0.904602 -1.840041 0.138183
2 -1.256608 -1.740634 -1.653686 -0.412524
3 0.165782 1.116089 0.065008 -1.693706
4 1.313987 0.734437 -0.625647 -1.738446

列索引

# 列索引
print(type(df_obj['a'])) # 返回Series類型
df_obj['a'] # 返回對應列值
<class 'pandas.core.series.Series'>





0    0.983790
1    0.161653
2   -1.256608
3    0.165782
4    1.313987
Name: a, dtype: float64

行索引

# 行索引
print(type(df_obj.loc[0])) # 返回Series類型
df_obj.loc[0] # 返回對應行值
<class 'pandas.core.series.Series'>





a    0.983790
b    1.063804
c    0.854634
d   -1.269025
Name: 0, dtype: float64

不連續索引

#不連續列索引
df_obj[['a','c']]  #不連續列索引
a c
0 0.983790 0.854634
1 0.161653 -1.840041
2 -1.256608 -1.653686
3 0.165782 0.065008
4 1.313987 -0.625647
#不連續行索引
df_obj.loc[[1, 3]] #不連續行索引
a b c d
1 0.161653 -0.904602 -1.840041 0.138183
3 0.165782 1.116089 0.065008 -1.693706

混合索引

# 混合索引 loc
print(df_obj.loc[0:2, 'a']) # 連續行加列索引(這裏是從0-2)
print()
print(df_obj.loc[[0,2,4], 'a']) # 不連續行加列索引
0   -1.018941
1    0.089275
2   -2.210780
Name: a, dtype: float64

0   -1.018941
2   -2.210780
4    1.435787
Name: a, dtype: float64

運算與對齊

Series 對齊操作
s1 = pd.Series(range(10, 13), index = range(3))
s2 = pd.Series(range(20, 25), index = range(5))

print('s1: ' )
print(s1)

print('')

print('s2: ')
print(s2)
s1:
0    10
1    11
2    12
dtype: int32

s2:
0    20
1    21
2    22
3    23
4    24
dtype: int32
# Series 對齊運算
print(s1 + s2) # 沒有對應上的部分會顯示NaN
print()
print(s1.add(s2, fill_value = -1)) # 沒有對應上的部分會填充-1,然後運算
print()
s3 = s1 + s2
s3_filled = s3.fillna(-1)
print(s3_filled) ## 先運算,然後NaN填充爲-1
0    30.0
1    32.0
2    34.0
3     NaN
4     NaN
dtype: float64

0    30.0
1    32.0
2    34.0
3    22.0
4    23.0
dtype: float64

0    30.0
1    32.0
2    34.0
3    -1.0
4    -1.0
dtype: float64
DataFrame 對齊操作
import numpy as np

df1 = pd.DataFrame(np.ones((2,2)), columns = ['a', 'b'])
df2 = pd.DataFrame(np.ones((3,3)), columns = ['a', 'b', 'c'])

print('df1: ')
print(df1)

print('')
print('df2: ')
print(df2)
df1:
     a    b
0  1.0  1.0
1  1.0  1.0

df2:
     a    b    c
0  1.0  1.0  1.0
1  1.0  1.0  1.0
2  1.0  1.0  1.0
# DataFrame對齊操作
df1 + df2 # 沒有對應上的部分會顯示NaN
a b c
0 2.0 2.0 NaN
1 2.0 2.0 NaN
2 NaN NaN NaN
df1.add(df2, fill_value = 0) # 加法操作,沒有對應上的補零
a b c
0 2.0 2.0 1.0
1 2.0 2.0 1.0
2 1.0 1.0 1.0
df1 - df2 # 沒有對應上的部分會顯示NaN
a b c
0 0.0 0.0 NaN
1 0.0 0.0 NaN
2 NaN NaN NaN
df1.sub(df2, fill_value = 2) # 加法操作,沒有對應上的補2(先補充後運算)
a b c
0 0.0 0.0 1.0
1 0.0 0.0 1.0
2 1.0 1.0 1.0
df3 = df1 + df2
df3.fillna(100, inplace = True) # 先運行加法操作,沒有對應上的補2(先運算,後補充)
df3
a b c
0 2.0 2.0 100.0
1 2.0 2.0 100.0
2 100.0 100.0 100.0

函數應用

可以與NumPy中的ufunc函數結合操作

# Numpy ufunc 函數
df = pd.DataFrame(np.random.randn(5,4) - 1)
df
0 1 2 3
0 -0.938212 -2.487779 -1.805374 -1.130723
1 -0.533441 0.196536 -1.094895 -1.819312
2 -3.233318 0.255510 -1.560183 -2.404621
3 -1.956924 -2.947539 -1.640760 -0.757321
4 0.198618 0.344484 -0.893815 -0.498036
np.abs(df) #取絕對值(還有其他諸多NumPy中的函數可以操作)
0 1 2 3
0 0.938212 2.487779 1.805374 1.130723
1 0.533441 0.196536 1.094895 1.819312
2 3.233318 0.255510 1.560183 2.404621
3 1.956924 2.947539 1.640760 0.757321
4 0.198618 0.344484 0.893815 0.498036

使用apply應用行或列數據

# 使用apply應用行或列數據
# f = lambda x : x.max() # lambda存在意義就是對簡單函數的簡潔表示
def f(x):
    return x.max()

df.apply(f) # 默認按行比較(得到每列的最大值)
0    0.198618
1    0.344484
2   -0.893815
3   -0.498036
dtype: float64
df.apply(lambda x : x.max(), axis=1) # 按列比較(得到每行的最大值)
0   -0.938212
1    0.196536
2    0.255510
3   -0.757321
4    0.344484
dtype: float64
df.apply(lambda x : x.max(), axis=0) # # 按行比較(得到每列的最大值)
0    0.198618
1    0.344484
2   -0.893815
3   -0.498036
dtype: float64

使用applymap應用到每個數據

# 使用applymap應用到每個數據
f2 = lambda x : '%.2f' % x #每個數據顯示只保留兩位小數
df.applymap(f2)
0 1 2 3
0 -0.94 -2.49 -1.81 -1.13
1 -0.53 0.20 -1.09 -1.82
2 -3.23 0.26 -1.56 -2.40
3 -1.96 -2.95 -1.64 -0.76
4 0.20 0.34 -0.89 -0.50

排序

Series索引排序 & 值排序

#索引亂序生成
s4 = pd.Series([10,13,12,25,14], index = [2,1,5,3,4])
s4
2    10
1    13
5    12
3    25
4    14
dtype: int64
# 索引排序
s4.sort_index(ascending=False) #  索引倒序排列
5    12
4    14
3    25
2    10
1    13
dtype: int64
# 值排序
s4.sort_values()
2    10
5    12
1    13
4    14
3    25
dtype: int64

DataFrame 索引排序 & 值排序

df4 = pd.DataFrame(np.random.randn(3, 4),
                   index=[1,3,2],
                   columns=[1,4,2,3])
df4
1 4 2 3
1 0.948112 0.076323 0.089607 0.091737
3 -1.254556 1.483504 0.468995 0.286249
2 -0.806738 -0.842388 -1.127489 -0.020803
#按索引排序
df4.sort_index(ascending=False)# 對橫軸按倒序排列
1 4 2 3
3 -1.254556 1.483504 0.468995 0.286249
2 -0.806738 -0.842388 -1.127489 -0.020803
1 0.948112 0.076323 0.089607 0.091737
#按索引排序
df4.sort_index(axis=1) #列軸按序排列
1 2 3 4
1 0.948112 0.089607 0.091737 0.076323
3 -1.254556 0.468995 0.286249 1.483504
2 -0.806738 -1.127489 -0.020803 -0.842388
#按列排序
df4.sort_values(by=1) # by參數的作用是針對某一(些)列進行排序(不能對行使用 by 參數)
1 4 2 3
3 -1.254556 1.483504 0.468995 0.286249
2 -0.806738 -0.842388 -1.127489 -0.020803
1 0.948112 0.076323 0.089607 0.091737

處理缺失數據

生成數據

df_data = pd.DataFrame([np.random.randn(3), [1., np.nan, np.nan],
                       [4., np.nan, np.nan], [1., np.nan, 2.]])
df_data.head()
0 1 2
0 1.089477 -0.486706 -0.322284
1 1.000000 NaN NaN
2 4.000000 NaN NaN
3 1.000000 NaN 2.000000

二值化(NaN爲False,非NaN爲True)

# isnull
df_data.isnull()
0 1 2
0 False False False
1 False True True
2 False True True
3 False True False

丟掉有NaN的行或列

# dropna
print(df_data.dropna()) #默認丟掉有NaN的行
print()
print(df_data.dropna(axis=1)) #丟掉有NaN的列
          0         1         2
0  1.089477 -0.486706 -0.322284

          0
0  1.089477
1  1.000000
2  4.000000
3  1.000000

填充NaN值

# fillna
df_data.fillna(-100.) # NaN值填充爲-100
0 1 2
0 1.089477 -0.486706 -0.322284
1 1.000000 -100.000000 -100.000000
2 4.000000 -100.000000 -100.000000
3 1.000000 -100.000000 2.000000

數據統計計算和描述

常用的統計計算

df_obj = pd.DataFrame(np.random.randn(5,4), columns = ['a', 'b', 'c', 'd'])
df_obj
a b c d
0 0.145119 -2.398595 0.640806 0.696701
1 -0.877139 -0.261616 -2.211734 0.140729
2 -0.644545 0.523667 -1.460002 -0.341459
3 1.369260 1.039981 0.164075 0.380755
4 0.089507 -0.371051 1.348191 -0.828315
df_obj.sum()
a    0.082203
b   -1.467614
c   -1.518663
d    0.048410
dtype: float64
df_obj.max()
a    1.369260
b    1.039981
c    1.348191
d    0.696701
dtype: float64
df_obj.min(axis=1)
0   -2.398595
1   -2.211734
2   -1.460002
3    0.164075
4   -0.828315
dtype: float64

統計描述

df_obj.describe()
a b c d
count 5.000000 5.000000 5.000000 5.000000
mean 0.016441 -0.293523 -0.303733 0.009682
std 0.878550 1.311906 1.484695 0.602578
min -0.877139 -2.398595 -2.211734 -0.828315
25% -0.644545 -0.371051 -1.460002 -0.341459
50% 0.089507 -0.261616 0.164075 0.140729
75% 0.145119 0.523667 0.640806 0.380755
max 1.369260 1.039981 1.348191 0.696701

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章