三個(gè)Python常用的數(shù)據(jù)清洗處理方式總結(jié)

更新時(shí)間：2022年12月20日 09:19:20 作者：Sir 老王

這篇文章主要為大家詳細(xì)介紹了python數(shù)據(jù)處理過程中三個(gè)主要的數(shù)據(jù)清洗說明，分別是缺失值/空格/重復(fù)值的數(shù)據(jù)清洗，感興趣的小伙伴可以了解一下

#???????????姓名????年齡????班級???成績?表現(xiàn)
#?0???Python?集中營??10??1210???99??A
#?1???Python?集中營??11??1211??100??A
#?2???Python?集中營??12??1212??101??A
#?3???Python?集中營??13??1213??102??A
#?4???Python?集中營??14??1214??103??A
#?5???Python?集中營??15??1215??104??A
#?6???Python?集中營??16??1216??105??A
#?7???Python?集中營??17??1217??106??A
#?8???Python?集中營??18??1218??107??A
#?9???Python?集中營??19??1219??108??A
#?10??Python?集中營??20??1220??109??A
#?11??Python?集中營??21??1221??110??A
#?12??Python?集中營??22??1222??111??A
#?13??Python?集中營??23??1223??112??A
#?14??Python?集中營??24??1224??113??A
#?15??Python?集中營??25??1225??114??A
#?16??Python?集中營??26??1226??115??A
#?17??Python?集中營??27??1227??116??A
#?18??Python?集中營??28??1228??117??A
#
#?Process?finished?with?exit?code?0

準(zhǔn)備好數(shù)據(jù)源之后，我們使用三個(gè)方式來完成對源數(shù)據(jù)的數(shù)據(jù)清洗。

1. strip函數(shù)清除空格

首先，將所有的列名稱提取出來，使用DataFrame對象的columns函數(shù)進(jìn)行提取。

#?Extracting?the?column?names?from?the?dataframe?and?storing?it?in?a?variable?called?`columns_`.
columns_?=?result_.columns.values

#?Printing?the?column?names?of?the?dataframe.
print(columns_)

#?['??姓名??'?'年齡'?'班級'?'成績'?'表現(xiàn)']

從列名稱的打印結(jié)果發(fā)現(xiàn)'姓名'這一列是存在空格的，我們直接查找列名稱是找不到的，因?yàn)樾枰獙α忻Q的空格進(jìn)行數(shù)據(jù)清洗。

為了減少代碼塊的使用，我們這里直接使用列表推導(dǎo)式的方式對列名稱的空格進(jìn)行清洗。

#?A?list?comprehension?that?is?iterating?over?the?`columns_`?list?and?stripping?the?whitespaces?from?each?element?of?the
#?list.
result_.columns?=?[column_name.strip()?for?column_name?in?columns_]

#?Printing?the?column?names?of?the?dataframe.
print(result_.columns.values)

#?['姓名'?'年齡'?'班級'?'成績'?'表現(xiàn)']

經(jīng)過數(shù)據(jù)清洗后，發(fā)現(xiàn)所有的列名稱空格情況已經(jīng)被全部清洗了。若是存在某個(gè)列中的值空格需要清洗也可以采用strip函數(shù)進(jìn)行清洗。

2. duplicated函數(shù)清除重復(fù)數(shù)據(jù)

關(guān)于重復(fù)數(shù)據(jù)的判斷有兩種情況，一種是兩行完全相同的數(shù)據(jù)即為重復(fù)數(shù)據(jù)。另外一種則是部分相同指的是某個(gè)列的數(shù)據(jù)是相同的需要清洗。

#?The?`duplicated()`?function?is?returning?a?boolean?series?that?is?True?if?the?row?is?a?duplicate?and?False?if?the?row?is
#?not?a?duplicate.
repeat_num?=?result_.duplicated().sum()

#?Printing?the?number?of?duplicate?rows?in?the?dataframe.
print(repeat_num)

#?1

通過上面的duplicated().sum()函數(shù)得到的是兩個(gè)完全相同的數(shù)據(jù)行是多少。

接著則可以對源數(shù)據(jù)進(jìn)行實(shí)際意義上的刪除，使用DataFrame對象的drop_duplicates函數(shù)進(jìn)行刪除。

#?The?`drop_duplicates()`?function?is?dropping?the?duplicate?rows?from?the?dataframe?and?the?`inplace=True`?is
#?modifying?the?dataframe?in?place.
result_.drop_duplicates(inplace=True)

#?Printing?the?dataframe.
print(result_)

#????????????姓名??年齡????班級???成績?表現(xiàn)
#?0???Python?集中營??10??1210???99??A
#?1???Python?集中營??11??1211??100??A
#?2???Python?集中營??12??1212??101??A
#?3???Python?集中營??13??1213??102??A
#?4???Python?集中營??14??1214??103??A
#?5???Python?集中營??15??1215??104??A
#?6???Python?集中營??16??1216??105??A
#?7???Python?集中營??17??1217??106??A
#?8???Python?集中營??18??1218??107??A
#?9???Python?集中營??19??1219??108??A
#?10??Python?集中營??20??1220??109??A
#?11??Python?集中營??21??1221??110??A
#?12??Python?集中營??22??1222??111??A
#?13??Python?集中營??23??1223??112??A
#?14??Python?集中營??24??1224??113??A
#?15??Python?集中營??25??1225??114??A
#?16??Python?集中營??26??1226??115??A
#?17??Python?集中營??27??1227??116??A

因?yàn)樽詈笠恍泻偷谝恍械臄?shù)據(jù)是完全相同的，因此最后一行的數(shù)據(jù)已經(jīng)被清洗掉了。

一般在數(shù)據(jù)清洗刪除重復(fù)值之后需要重置索引，避免索引產(chǎn)生不連續(xù)性。

#?The?`range(result_.shape[0])`?is?creating?a?list?of?numbers?from?0?to?the?number?of?rows?in?the?dataframe.
result_.index?=?range(result_.shape[0])

#?The?`print(result_.index)`?is?printing?the?index?of?the?dataframe.
print(result_.index)

#?RangeIndex(start=0,?stop=18,?step=1)

3. 數(shù)據(jù)缺失值補(bǔ)全

一般查看DataFrame數(shù)據(jù)對象的缺失值就是通過使用isnull函數(shù)來提取所有數(shù)據(jù)缺失的部分。

#?The?`isnull()`?function?is?returning?a?boolean?series?that?is?True?if?the?value?is?missing?and?False?if?the?value
#?is?not?missing.
sul_?=?result_.isnull()

#?The?`print(sul_)`?is?printing?the?boolean?series?that?is?True?if?the?value?is?missing?and?False?if?the?value?is?not
#?missing.
print(sul_)

#????????姓名?????年齡?????班級?????成績?????表現(xiàn)
#?0???False??False??False??False??False
#?1???False??False??False??False??False
#?2???False??False??False??False??False
#?3???False??False??False??False??False
#?4???False??False??False??False??False
#?5???False??False??False??False??False
#?6???False??False??False??False??False
#?7???False??False??False??False??False
#?8???False??False??False??False??False
#?9???False??False??False??False??False
#?10??False??False??False??False??False
#?11??False??False??False??False??False
#?12??False??False??False??False??False
#?13??False??False??False??False??False
#?14??False??False??False??False??False
#?15??False??False??False??False??False
#?16??False??False??False??False??False
#?17??False??False??False??False??False

返回的每一個(gè)單元格數(shù)據(jù)結(jié)果為False則代表這個(gè)單元格的數(shù)據(jù)是沒有缺失的，或者也可以使用notnull來反向查看。

使用isnull函數(shù)不想顯示很多的列表數(shù)據(jù)時(shí)，可以使用sum函數(shù)進(jìn)行統(tǒng)計(jì)。

#?The?`isnull_sum?=?result_.isnull().sum()`?is?returning?a?series?that?is?the?sum?of?the?boolean?series?that?is?True?if
#?the?value?is?missing?and?False?if?the?value?is?not?missing.
isnull_sum?=?result_.isnull().sum()

#?The?`isnull_sum?=?result_.isnull().sum()`?is?returning?a?series?that?is?the?sum?of?the?boolean?series?that?is?True?if
#?the?value?is?missing?and?False?if?the?value?is?not?missing.
print(isnull_sum)

#?姓名????0
#?年齡????0
#?班級????0
#?成績????0
#?表現(xiàn)????0
#?dtype:?int64

通過isnull函數(shù)處理后使用sum函數(shù)進(jìn)行統(tǒng)計(jì)，統(tǒng)計(jì)后會(huì)返回每一列的數(shù)據(jù)單元格為空的個(gè)數(shù)。

接下來就是數(shù)據(jù)值的填補(bǔ)過程，通?？梢院Y選每一列中的空值填補(bǔ)固定的數(shù)據(jù)。

#?The?`result_.loc[result_.姓名.isnull(),?'姓名']`?is?returning?a?series?that?is?the?values?of?the?column?`姓名`
#?where?the?values?are?missing.?The?`'Python?集中營'`?is?the?value?that?is?being?assigned?to?the?series.
result_.loc[result_.姓名.isnull(),?'姓名']?=?'Python?集中營'

#?Printing?the?dataframe.
print(result_)

#?????????????姓名??年齡????班級???成績?表現(xiàn)
#?0???Python?集中營??10??1210???99??A
#?1???Python?集中營??11??1211??100??A
#?2???Python?集中營??12??1212??101??A
#?3???Python?集中營??13??1213??102??A
#?4???Python?集中營??14??1214??103??A
#?5???Python?集中營??15??1215??104??A
#?6???Python?集中營??16??1216??105??A
#?7???Python?集中營??17??1217??106??A
#?8???Python?集中營??18??1218??107??A
#?9???Python?集中營??19??1219??108??A
#?10??Python?集中營??20??1220??109??A
#?11??Python?集中營??21??1221??110??A
#?12??Python?集中營??22??1222??111??A
#?13??Python?集中營??23??1223??112??A
#?14??Python?集中營??24??1224??113??A
#?15??Python?集中營??25??1225??114??A
#?16??Python?集中營??26??1226??115??A
#?17??Python?集中營??27??1227??116??A