Python必備技巧之Pandas數(shù)據(jù)合并函數(shù)

更新時間：2022年03月28日 16:02:39 作者：Python學習與數(shù)據(jù)挖掘

Pandas中一共有五個數(shù)據(jù)合并函數(shù)，分別為：concat、append、merge、join、combine，本文詳細講解這五個函數(shù)的使用方法，需要的可以參考一下

1. concat

concat是pandas中專門用于數(shù)據(jù)連接合并的函數(shù)，功能非常強大，支持縱向合并和橫向合并，默認情況下是縱向合并，具體可以通過參數(shù)進行設(shè)置。

pd.concat(
    objs: 'Iterable[NDFrame] | Mapping[Hashable, NDFrame]',
    axis=0,
    join='outer',
    ignore_index: 'bool' = False,
    keys=None,
    levels=None,
    names=None,
    verify_integrity: 'bool' = False,
    sort: 'bool' = False,
    copy: 'bool' = True,
) -> 'FrameOrSeriesUnion'

在函數(shù)方法中，各參數(shù)含義如下：

objs: 用于連接的數(shù)據(jù)，可以是DataFrame或Series組成的列表
axis=0 : 連接的方式，默認為0也就是縱向連接，可選 1 為橫向連接
join='outer'：合并方式，默認為 inner也就是交集，可選 outer 為并集
ignore_index: 是否保留原有的索引
keys=None：連接關(guān)系，使用傳遞的值作為一級索引
levels=None：用于構(gòu)造多級索引
names=None：索引的名稱
verify_integrity: 檢測索引是否重復，如果為True則有重復索引會報錯
sort: 并集合并方式下，對columns排序
copy: 是否深度拷貝

接下來，我們就對該函數(shù)功能進行演示

基礎(chǔ)連接

In [1]: import pandas as pd

In [2]: s1 = pd.Series(['a', 'b'])

In [3]: s2 = pd.Series(['c', 'd'])

In [4]: s1
Out[4]: 
0    a
1    b
dtype: object

In [5]: s2
Out[5]: 
0    c
1    d
dtype: object

In [6]: pd.concat([s1, s2])
Out[6]: 
0    a
1    b
0    c
1    d
dtype: object

In [7]: df1 = pd.DataFrame([['a', 1], ['b', 2]],
   ...:                     columns=['letter', 'number'])

In [8]: df2 = pd.DataFrame([['c', 3], ['d', 4]],
   ...:                     columns=['letter', 'number'])

In [9]: pd.concat([df1, df2])
Out[9]: 
  letter  number
0      a       1
1      b       2
0      c       3
1      d       4

橫向連接

In [10]: pd.concat([df1, df2], axis=1)
Out[10]: 
  letter  number letter  number
0      a       1      c       3
1      b       2      d       4

默認情況下，concat是取并集，如果兩個數(shù)據(jù)中有個數(shù)據(jù)沒有對應(yīng)行或列，則會填充為空值NaN。

合并交集

In [11]: df3 = pd.DataFrame([['c', 3, 'cat'], ['d', 4, 'dog']],
    ...:                     columns=['letter', 'number', 'animal'])

In [12]: df1
Out[12]: 
  letter  number
0      a       1
1      b       2

In [13]: df3
Out[13]: 
  letter  number animal
0      c       3    cat
1      d       4    dog

In [14]: pd.concat([df1, df3], join='inner')
Out[14]: 
  letter  number
0      a       1
1      b       2
0      c       3
1      d       4

索引重置（不保留原有索引）

In [15]: pd.concat([df1, df3], join='inner', ignore_index=True)
Out[15]: 
  letter  number
0      a       1
1      b       2
2      c       3
3      d       4
# 以下方式和上述的輸出結(jié)果等價
In [16]: pd.concat([df1, df3], join='inner').reset_index(drop=True)
Out[16]: 
  letter  number
0      a       1
1      b       2
2      c       3
3      d       4

指定索引

In [17]: pd.concat([df1, df3], keys=['df1','df3'])
Out[17]: 
      letter  number animal
df1 0      a       1    NaN
    1      b       2    NaN
df3 0      c       3    cat
    1      d       4    dog

In [18]: pd.concat([df1, df3], keys=['df1','df3'], names=['df名稱','行ID'])
Out[18]: 
         letter  number animal
df名稱 行ID                      
df1  0        a       1    NaN
     1        b       2    NaN
df3  0        c       3    cat
     1        d       4    dog

檢測重復

如果索引出現(xiàn)重復，則無法通過檢測，會報錯

In [19]: pd.concat([df1, df3], verify_integrity=True)
Traceback (most recent call last):
...
ValueError: Indexes have overlapping values: Int64Index([0, 1], dtype='int64')

合并并集下columns排序

In [21]: pd.concat([df1, df3], sort=True)
Out[21]: 
  animal letter  number
0    NaN      a       1
1    NaN      b       2
0    cat      c       3
1    dog      d       4

DataFrame與Series合并

In [22]: pd.concat([df1, s1])
Out[22]: 
  letter  number    0
0      a     1.0  NaN
1      b     2.0  NaN
0    NaN     NaN    a
1    NaN     NaN    b

In [23]: pd.concat([df1, s1], axis=1)
Out[23]: 
  letter  number  0
0      a       1  a
1      b       2  b
# 新增列一般可選以下兩種方式
In [24]: df1.assign(新增列=s1)
Out[24]: 
  letter  number 新增列
0      a       1   a
1      b       2   b

In [25]: df1['新增列'] = s1

In [26]: df1
Out[26]: 
  letter  number 新增列
0      a       1   a
1      b       2   b

以上就concat函數(shù)方法的一些功能，相比之下，另外一個函數(shù)append也可以用于數(shù)據(jù)追加（縱向合并）

2. append

append主要用于追加數(shù)據(jù)，是比較簡單直接的數(shù)據(jù)合并方式。

df.append(
    other,
    ignore_index: 'bool' = False,
    verify_integrity: 'bool' = False,
    sort: 'bool' = False,
) -> 'DataFrame'

在函數(shù)方法中，各參數(shù)含義如下：

other: 用于追加的數(shù)據(jù)，可以是DataFrame或Series或組成的列表
ignore_index: 是否保留原有的索引
verify_integrity: 檢測索引是否重復，如果為True則有重復索引會報錯
sort: 并集合并方式下，對columns排序

接下來，我們就對該函數(shù)功能進行演示

基礎(chǔ)追加

In [41]: df1.append(df2)
Out[41]: 
  letter  number
0      a       1
1      b       2
0      c       3
1      d       4

In [42]: df1.append([df1,df2,df3])
Out[42]: 
  letter  number animal
0      a       1    NaN
1      b       2    NaN
0      a       1    NaN
1      b       2    NaN
0      c       3    NaN
1      d       4    NaN
0      c       3    cat
1      d       4    dog

columns重置（不保留原有索引）

In [43]: df1.append([df1,df2,df3], ignore_index=True)
Out[43]: 
  letter  number animal
0      a       1    NaN
1      b       2    NaN
2      a       1    NaN
3      b       2    NaN
4      c       3    NaN
5      d       4    NaN
6      c       3    cat
7      d       4    dog

檢測重復

如果索引出現(xiàn)重復，則無法通過檢測，會報錯

In [44]: df1.append([df1,df2], verify_integrity=True)
Traceback (most recent call last):
...
ValueError: Indexes have overlapping values: Int64Index([0, 1], dtype='int64')

索引排序

In [46]: df1.append([df1,df2,df3], sort=True)
Out[46]: 
  animal letter  number
0    NaN      a       1
1    NaN      b       2
0    NaN      a       1
1    NaN      b       2
0    NaN      c       3
1    NaN      d       4
0    cat      c       3
1    dog      d       4

追加Series

In [49]: s = pd.Series({'letter':'s1','number':9})

In [50]: s
Out[50]: 
letter    s1
number     9
dtype: object

In [51]: df1.append(s)
Traceback (most recent call last):
...
TypeError: Can only append a Series if ignore_index=True or if the Series has a name

In [53]: df1.append(s, ignore_index=True)
Out[53]: 
  letter  number
0      a       1
1      b       2
2     s1       9

追加字典

這個在爬蟲的時候比較好使，每爬取一條數(shù)據(jù)就合并到DataFrame類似數(shù)據(jù)中存儲起來

In [54]: dic = {'letter':'s1','number':9}

In [55]: df1.append(dic, ignore_index=True)
Out[55]:?
? letter ?number
0 ? ? ?a ? ? ? 1
1 ? ? ?b ? ? ? 2
2 ? ? s1 ? ? ? 9

3. merge

merge函數(shù)方法類似SQL里的join，可以是pd.merge或者df.merge，區(qū)別就在于后者待合并的數(shù)據(jù)是

pd.merge(
    left: 'DataFrame | Series',
    right: 'DataFrame | Series',
    how: 'str' = 'inner',
    on: 'IndexLabel | None' = None,
    left_on: 'IndexLabel | None' = None,
    right_on: 'IndexLabel | None' = None,
    left_index: 'bool' = False,
    right_index: 'bool' = False,
    sort: 'bool' = False,
    suffixes: 'Suffixes' = ('_x', '_y'),
    copy: 'bool' = True,
    indicator: 'bool' = False,
    validate: 'str | None' = None,
) -> 'DataFrame'

在函數(shù)方法中，關(guān)鍵參數(shù)含義如下：

left: 用于連接的左側(cè)數(shù)據(jù)
right: 用于連接的右側(cè)數(shù)據(jù)
how: 數(shù)據(jù)連接方式，默認為 inner，可選outer、left和right
on: 連接關(guān)鍵字段，左右側(cè)數(shù)據(jù)中需要都存在，否則就用left_on和right_on
left_on: 左側(cè)數(shù)據(jù)用于連接的關(guān)鍵字段
right_on: 右側(cè)數(shù)據(jù)用于連接的關(guān)鍵字段
left_index: True表示左側(cè)索引為連接關(guān)鍵字段
right_index: True表示右側(cè)索引為連接關(guān)鍵字段
suffixes: ‘Suffixes’ = (’_x’, ‘_y’),可以自由指定，就是同列名合并后列名顯示后綴
indicator: 是否顯示合并后某行數(shù)據(jù)的歸屬來源

接下來，我們就對該函數(shù)功能進行演示

基礎(chǔ)合并

In [55]: df1 = pd.DataFrame({'key': ['foo', 'bar', 'bal'],
    ...:                     'value2': [1, 2, 3]})

In [56]: df2 = pd.DataFrame({'key': ['foo', 'bar', 'baz'],
    ...:                     'value1': [5, 6, 7]})

In [57]: df1.merge(df2)
Out[57]: 
   key  value2  value1
0  foo       1       5
1  bar       2       6

其他連接方式

In [58]: df1.merge(df2, how='left')
Out[58]: 
   key  value2  value1
0  foo       1     5.0
1  bar       2     6.0
2  bal       3     NaN

In [59]: df1.merge(df2, how='right')
Out[59]: 
   key  value2  value1
0  foo     1.0       5
1  bar     2.0       6
2  baz     NaN       7

In [60]: df1.merge(df2, how='outer')
Out[60]: 
   key  value2  value1
0  foo     1.0     5.0
1  bar     2.0     6.0
2  bal     3.0     NaN
3  baz     NaN     7.0

In [61]: df1.merge(df2, how='cross')
Out[61]: 
  key_x  value2 key_y  value1
0   foo       1   foo       5
1   foo       1   bar       6
2   foo       1   baz       7
3   bar       2   foo       5
4   bar       2   bar       6
5   bar       2   baz       7
6   bal       3   foo       5
7   bal       3   bar       6
8   bal       3   baz       7

指定連接鍵

可以指定單個連接鍵，也可以指定多個連接鍵

In [62]: df1 = pd.DataFrame({'lkey1': ['foo', 'bar', 'bal'],
    ...:                     'lkey2': ['a', 'b', 'c'],
    ...:                     'value2': [1, 2, 3]})

In [63]: df2 = pd.DataFrame({'rkey1': ['foo', 'bar', 'baz'],
    ...:                     'rkey2': ['a', 'b', 'c'],
    ...:                     'value2': [5, 6, 7]})
    
In [64]: df1
Out[64]: 
  lkey1 lkey2  value2
0   foo     a       1
1   bar     b       2
2   bal     c       3

In [65]: df2
Out[65]: 
  rkey1 rkey2  value2
0   foo     a       5
1   bar     b       6
2   baz     c       7

In [66]: df1.merge(df2, left_on='lkey1', right_on='rkey1')
Out[66]: 
  lkey1 lkey2  value2_x rkey1 rkey2  value2_y
0   foo     a         1   foo     a         5
1   bar     b         2   bar     b         6

In [67]: df1.merge(df2, left_on=['lkey1','lkey2'], right_on=['rkey1','rkey2'])
Out[67]: 
  lkey1 lkey2  value2_x rkey1 rkey2  value2_y
0   foo     a         1   foo     a         5
1   bar     b         2   bar     b         6

指定索引為鍵

Out[68]: df1.merge(df2, left_index=True, right_index=True)
Out[68]: 
  lkey1 lkey2  value2_x rkey1 rkey2  value2_y
0   foo     a         1   foo     a         5
1   bar     b         2   bar     b         6
2   bal     c         3   baz     c         7

設(shè)置重復列后綴

In [69]: df1.merge(df2, left_on='lkey1', right_on='rkey1', suffixes=['左','右'])
Out[69]: 
  lkey1 lkey2  value2左 rkey1 rkey2  value2右
0   foo     a        1   foo     a        5
1   bar     b        2   bar     b        6

連接指示

新增一列用于顯示數(shù)據(jù)來源

In [70]: df1.merge(df2, left_on='lkey1', right_on='rkey1', suffixes=['左','右'], how='outer',
    ...:           indicator=True
    ...:       )
Out[70]: 
  lkey1 lkey2  value2左 rkey1 rkey2  value2右      _merge
0   foo     a      1.0   foo     a      5.0        both
1   bar     b      2.0   bar     b      6.0        both
2   bal     c      3.0   NaN   NaN      NaN   left_only
3   NaN   NaN      NaN   baz     c      7.0  right_only

4. join

join就有點想append之于concat，用于數(shù)據(jù)合并

df.join(
    other: 'FrameOrSeriesUnion',
    on: 'IndexLabel | None' = None,
    how: 'str' = 'left',
    lsuffix: 'str' = '',
    rsuffix: 'str' = '',
    sort: 'bool' = False,
) -> 'DataFrame'

在函數(shù)方法中，關(guān)鍵參數(shù)含義如下：

other: 用于合并的右側(cè)數(shù)據(jù)
on: 連接關(guān)鍵字段，左右側(cè)數(shù)據(jù)中需要都存在，否則就用left_on和right_on
how: 數(shù)據(jù)連接方式，默認為 inner，可選outer、left和right
lsuffix: 左側(cè)同名列后綴
rsuffix：右側(cè)同名列后綴

接下來，我們就對該函數(shù)功能進行演示

In [71]: df = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3', 'K4', 'K5'],
    ...:                     'A': ['A0', 'A1', 'A2', 'A3', 'A4', 'A5']})

In [72]: other = pd.DataFrame({'key': ['K0', 'K1', 'K2'],
    ...:                        'B': ['B0', 'B1', 'B2']})

In [73]: df
Out[73]: 
  key   A
0  K0  A0
1  K1  A1
2  K2  A2
3  K3  A3
4  K4  A4
5  K5  A5

In [74]: other
Out[74]: 
  key   B
0  K0  B0
1  K1  B1
2  K2  B2

In [75]: df.join(other, on='key')
Traceback (most recent call last):
...
ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat

如果想用key關(guān)鍵字，則需要key是索引。。。

指定key

In [76]: df.set_index('key').join(other.set_index('key'))
Out[76]: 
      A    B
key         
K0   A0   B0
K1   A1   B1
K2   A2   B2
K3   A3  NaN
K4   A4  NaN
K5   A5  NaN

In [77]: df.join(other.set_index('key'), on='key')
Out[77]: 
  key   A    B
0  K0  A0   B0
1  K1  A1   B1
2  K2  A2   B2
3  K3  A3  NaN
4  K4  A4  NaN
5  K5  A5  NaN

指定重復列后綴

In [78]: df.join(other, lsuffix='_左', rsuffix='右')
Out[78]: 
  key_左   A key右    B
0    K0  A0   K0   B0
1    K1  A1   K1   B1
2    K2  A2   K2   B2
3    K3  A3  NaN  NaN
4    K4  A4  NaN  NaN
5    K5  A5  NaN  NaN

其他參數(shù)就不多做介紹了，和merge基本一樣。

5. combine

在數(shù)據(jù)合并的過程中，我們可能需要對對應(yīng)位置的值進行一定的計算，pandas提供了combine和combine_first函數(shù)方法來進行這方面的合作操作。

df.combine(
    other: 'DataFrame',
    func,
    fill_value=None,
    overwrite: 'bool' = True,
) -> 'DataFrame'

比如，數(shù)據(jù)合并的時候取單元格最小的值

In [79]: df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})

In [80]: df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})

In [81]: df1
Out[81]: 
   A  B
0  0  4
1  0  4

In [82]: df2
Out[82]: 
   A  B
0  1  3
1  1  3

In [83]: take_smaller = lambda s1, s2: s1 if s1.sum() < s2.sum() else s2

In [84]: df1.combine(df2, take_smaller)
Out[84]: 
   A  B
0  0  3
1  0  3

# 也可以調(diào)用numpy的函數(shù)
In [85]: import numpy as np

In [86]: df1.combine(df2, np.minimum)
Out[86]: 
   A  B
0  0  3
1  0  3

fill_value填充缺失值

In [87]: df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})

In [87]: df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})

In [88]: df1
Out[88]: 
   A    B
0  0  NaN
1  0  4.0

In [89]: df2
Out[89]: 
   A  B
0  1  3
1  1  3

In [90]: df1.combine(df2, take_smaller, fill_value=-88)
Out[90]: 
   A     B
0  0 -88.0
1  0   4.0

overwrite=False保留

In [91]: df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})

In [92]: df2 = pd.DataFrame({'B': [3, 3], 'C': [-10, 1], }, index=[1, 2])

In [93]: df1
Out[93]: 
   A  B
0  0  4
1  0  4

In [94]: df2
Out[94]: 
   B   C
1  3 -10
2  3   1

In [95]: df1.combine(df2, take_smaller)
Out[95]: 
    A    B     C
0 NaN  NaN   NaN
1 NaN  3.0 -10.0
2 NaN  3.0   1.0
# 保留A列原有的值
In [96]: df1.combine(df2, take_smaller, overwrite=False)
Out[96]: 
     A    B     C
0  0.0  NaN   NaN
1  0.0  3.0 -10.0
2  NaN  3.0   1.0

另外一個combine_first

df.combine_first(other: 'DataFrame') -> 'DataFrame'

當df中元素為空采用other里的進行替換，結(jié)果為并集合并

In [97]: df1 = pd.DataFrame({'A': [None, 0], 'B': [None, 4]})

In [98]: df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})

In [99]: df1
Out[99]: 
     A    B
0  NaN  NaN
1  0.0  4.0

In [100]: df2
Out[100]: 
   A  B
0  1  3
1  1  3

In [101]: df1.combine_first(df2)
Out[101]: 
     A    B
0  1.0  3.0
1  0.0  4.0

In [102]: df1 = pd.DataFrame({'A': [None, 0], 'B': [4, None]})

In [103]: df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1]}, index=[1, 2])

In [104]: df1
Out[104]: 
     A    B
0  NaN  4.0
1  0.0  NaN

In [105]: df2
Out[105]: 
   B  C
1  3  1
2  3  1

In [106]: df1.combine_first(df2)
Out[106]: 
     A    B    C
0  NaN  4.0  NaN
1  0.0  3.0  1.0
2  NaN  3.0  1.0