在某次開發中多進程間需要頻繁共享變量， 碰到了各種各樣的問題，總結一下。
一、multiprocessing中的Queue共享問題
1.先看一段代碼
import multiprocessing, time
q = multiprocessing.Queue()
def task(count):
for i in xrange(count):
q.put("%d mississippi" % i)
return "Done"
def main():
pool = multiprocessing.Pool()
result = pool.apply_async(task, (1,))
time.sleep(1)
while not q.empty():
print q.get()
print result.get()
if __name__ == "__main__":
main()運行之後，輸出結果：
0 mississippi
Done
通過以上代碼我們知道進程間通信可調用multiprocessing.Queue.但會發現此時的q是global。試想如果是局部變量情況會是怎麼樣的呢？看下面代碼：
import multiprocessing, time
def task(count):
for i in xrange(count):
q.put("%d mississippi" % i)
return "Done"
def main():
q = multiprocessing.Queue()
pool = multiprocessing.Pool()
result = pool.apply_async(task, (1, q))
time.sleep(1)
while not q.empty():
print q.get()
print result.get()
if __name__ == "__main__":
main()
RuntimeError: Queue objects should only be shared between processes through inheritance
解決方法1》： 在Pool初始化queue
import multiprocessing, time
def task(count):
for i in xrange(count):
task.q.put("%d mississippi" % i)
return "Done"
def task_init(q):
task.q = q
def main():
q = multiprocessing.Queue()
pool = multiprocessing.Pool(None, task_init, (q,))
result = pool.apply_async(task, (1,))
time.sleep(1)
while not q.empty():
print q.get()
print result.get()
if __name__ == "__main__":
main()解決方法2》：調用multiprocessing下的manager（）中的queue
import multiprocessing, time
def task(count, q):
for i in xrange(count):
q.put("%d mississippi" % i)
return "Done"
def main():
q = multiprocessing.Manager().Queue()
pool = multiprocessing.Pool()
result = pool.apply_async(task, (1,q,))
time.sleep(1)
while not q.empty():
print q.get()
print result.get()
if __name__ == "__main__":
main()
小結： 多進程間變量共享均可以調用Manager（）下的類型， A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore,Condition, Event, Queue, Value and Array.
二、調用Pool中的apply, apply_async區別
官方文檔上說的比較簡單:
-
apply(func[, args[, kwds]])
Equivalent of the apply() built-in function. It blocks until the result is ready, so apply_async() is better suited for performing work in parallel. Additionally, func is only executed in one of the workers of the pool.
apply_async(func[, args[, kwds[, callback]]])
A variant of the apply() method which returns a result object.
If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callbackis applied to it (unless the call failed). callback should complete immediately since otherwise the thread which handles the results will get blocked. -
- 我大概所理解的意思就是:
- apply, 是一個內建函數，當調用函數時會被堵住直到有返回結果。apply_async（）更適合用於平行工作。對於apply而言，func事實上只會在一個連接中工作。
- apply_async是apply函數的一個變種，當callback被指定時，它必須是能接受一個單獨的參數並且是可調用的。當結果生成時callback將立馬被執行除非是返回結果的函數被堵住了。
- 但是在使用的過程中還是要多注意apply_async所帶來的問題。
- 我所遇到的是異常捕獲的問題。看下面一段代碼：
from multiprocessing import Manager, Pool
from collections import namedtuple
import logging
def test():
A=namedtuple('A', ['x', 'y'])
return A(1, 2)
def f(q):
x = test()
try:
q.put(x)
except:
raise
try:
manager = Manager()
q = manager.Queue()
q.put(1)
p = Pool(processes = 2)
for i in range(3):
p.apply_async(f, args=(q,))
p.close()
p.join()
while q.empty() is False:
print q.get()
except:
logging.error('55555555', exc_info=True)我本來的意圖是想證明queue共享的問題。期望看到的結果是輸出1， A（1，2）.結果輸出就只有1.並且沒有任何異常拋出。
我一步一步跟進去發現函數f中是出現了異常的，由於namedtuple的也就是A的定義不是全局可見的，所以會拋出異常，但在父進程卻沒有捕獲到。
於是我寫了一個很簡單的例子來證明父進程是否能捕獲到子進程的異常。如下代碼：
import multiprocessing
def go():
print '---'
raise Exception()
if __name__ == '__main__':
p = multiprocessing.Pool(processes = 1)
p.apply_async(go)
p.close()
p.join()程序正常輸出---之後就結束了，未能捕獲到異常。但是將apply_aync換成apply之後就能正常捕獲到異常了。想了下apply是會block子進程的，捕獲到異常很正常。但是apply_async的異常爲什麼會報不出來呢？
查看官方文檔，看到下面一段話：
class multiprocessing.pool.AsyncResult
The class of the result returned by Pool.apply_async() and Pool.map_async().
get([timeout])
Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get().
大概意思就是apply_async的返回值是asyncresult類型，當子進程出現異常時，可以通過get（）函數來捕獲。
在網上搜到一段別人的代碼能詳細捕獲異常的代碼。
import traceback
from multiprocessing.pool import Pool
import multiprocessing
# Shortcut to multiprocessing's logger
def error(msg, *args):
return multiprocessing.get_logger().error(msg, *args)
class LogExceptions(object):
def __init__(self, callable):
self.__callable = callable
return
def __call__(self, *args, **kwargs):
try:
result = self.__callable(*args, **kwargs)
except Exception as e:
# Here we add some debugging help. If multiprocessing's
# debugging is on, it will arrange to log the traceback
error(traceback.format_exc())
# Re-raise the original exception so the Pool worker can
# clean up
raise
# It was fine, give a normal answer
return result
pass
class LoggingPool(Pool):
def apply_async(self, func, args=(), kwds={}, callback=None):
return Pool.apply_async(self, LogExceptions(func), args, kwds, callback)
def go():
print(1)
raise Exception()
print(2)
multiprocessing.log_to_stderr()
p = LoggingPool(processes=1)
p.apply_async(go)
p.close()
p.join()三、 PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed 錯誤
在調用multiprocessing的過程中經常性碰到這個錯誤，比如以下代碼：
import multiprocessing as mp
class Foo():
@staticmethod
def work(self):
pass
pool = mp.Pool()
foo = Foo()
pool.apply_async(foo.work)
pool.close()
pool.join()查找資料發現一個比較好的解釋：
The problem is that the pool methods all use a queue.Queue to pass tasks to the worker processes. Everything that goes through the queue.Queue must be pickable, and foo.work is not picklable since it is not defined
at the top level of the module.
解決方法：
def work(foo):
foo.work()
pool.apply_async(work,args=(foo,))