快捷導(dǎo)航

Python multiprocess pool模塊報(bào)錯(cuò)pickling error問(wèn)題解決方法分析

更新時(shí)間：2019年03月20日 08:59:47 作者：Arkenstone

這篇文章主要介紹了Python multiprocess pool模塊報(bào)錯(cuò)pickling error問(wèn)題解決方法,結(jié)合實(shí)例形式分析了multiprocess pool模塊報(bào)錯(cuò)pickling error的原因與解決方法,需要的朋友可以參考下

本文實(shí)例講述了Python multiprocess pool模塊報(bào)錯(cuò)pickling error問(wèn)題解決方法。分享給大家供大家參考，具體如下：

問(wèn)題

之前在調(diào)用class內(nèi)的函數(shù)用multiprocessing模塊的pool函數(shù)進(jìn)行多線程處理的時(shí)候報(bào)了以下下錯(cuò)誤信息:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

查了下官方文檔發(fā)現(xiàn)python默認(rèn)只能pickle以下的類(lèi)型:

None, True, and False
integers, floating point numbers, complex numbers
strings, bytes, bytearrays
tuples, lists, sets, and dictionaries containing only picklable objects
functions defined at the top level of a module (using def, not lambda)
built-in functions defined at the top level of a module
classes that are defined at the top level of a module
instances of such classes whose dict or the result of calling getstate() is picklable (see section -
Pickling Class Instances for details).

函數(shù)只能pickle在頂層定義的函數(shù)，很明顯的class內(nèi)的函數(shù)無(wú)法被pickle因此會(huì)報(bào)錯(cuò)。

import multiprocessing
def work():  # top-level 函數(shù)
  print "work!"
class Foo():
  def work(self): # 非top-level函數(shù)
    print "work"
pool1 = multiprocessing.Pool(processes=4)
foo = Foo()
pool1.apply_async(foo.work)
pool1.close()
pool1.join()
# 此時(shí)報(bào)錯(cuò)
pool2 = multiprocessing.Pool(processes=4)
pool2.apply_async(work)
pool2.close()
pool2.join()
# 此時(shí)工作正常

解決方案

調(diào)用pathos包下的multiprocessing模塊代替原生的multiprocessing。pathos中multiprocessing是用dill包改寫(xiě)過(guò)的，dill包可以將幾乎所有python的類(lèi)型都serialize，因此都可以被pickle。或者也可以自己用dill寫(xiě)一個(gè)（有點(diǎn)重復(fù)造輪子之嫌?。?/p>

參考

1. https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function
2. https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled
3. https://github.com/uqfoundation/pathos