因爲GIL(全局解釋器鎖)的限制(GIL是用來保證在任意時刻只能有一個控制線程在執行),所以python中的多線程並非真正的多線程。只有python程序是I/O密集型應用時,多線程纔會對運行效率有顯著提高(因在等待I/O的時,會釋放GIL允許其他線程繼續執行),而在計算密集型應用中,多線程並沒有什麼用處。考慮到要充分利用多核CPU的資源,允許python可以並行處理一些任務,這裏就用到了python多進程編程了。multiprocessing是python中的多進程模塊,使用這個模塊可以方便地進行多進程應用程序開發。multiprocessing模塊中提供了:Process、Pool、Queue、Manager等組件。
1. Process類
1.1 構造方法
def __init__(self, group=None, target=None, name=None, args=(), kwargs={})
group:進程所屬組,基本不用
target:進程調用對象(可以是一個函數名,也可以是一個可調用的對象(實現了__call__方法的類))
args:調用對象的位置參數元組
name:別名
kwargs:調用對象的關鍵字參數字典
1.2 實例方法
is_alive():返回進程是否在運行
start():啓動進程,等待CPU調度
join([timeout]):阻塞當前上下文環境,直到調用此方法的進程終止或者到達指定timeout
terminate():不管任務是否完成,立即停止該進程
run():start()調用該方法,當實例進程沒有傳入target參數,stat()將執行默認的run()方法
1.3 屬性
authkey:
daemon:守護進程標識,在start()調用之前可以對其進行修改
exitcode:進程的退出狀態碼
name:進程名
pid:進程id
1.4 實例
# 實例一:傳入的target爲一個函數
def foo(i):
print time.ctime(), 'process the %d begin ......' % i
time.sleep(random.uniform(1, 3))
print time.ctime(), 'process the %d end !!!' % i
if __name__ == '__main__':
print time.ctime, 'process begin...'
p_lst = list()
for i in xrange(4):
p_lst.append(Process(target=foo, args=(i,)))
# 啓動子進程
for p in p_lst:
p.start()
# 等待子進程全部結束
for p in p_lst:
p.join()
print time.ctime, 'process end!!!'
運行結果:
<built-in function ctime> process begin...
Thu Apr 19 10:33:52 2018 process the 0 begin ......
Thu Apr 19 10:33:52 2018 process the 1 begin ......
Thu Apr 19 10:33:52 2018 process the 2 begin ......
Thu Apr 19 10:33:52 2018 process the 3 begin ......
Thu Apr 19 10:33:53 2018 process the 0 end !!!
Thu Apr 19 10:33:53 2018 process the 3 end !!!
Thu Apr 19 10:33:53 2018 process the 2 end !!!
Thu Apr 19 10:33:53 2018 process the 1 end !!!
<built-in function ctime> process end!!!
# 實例二:傳入的target爲一個可調用對象
class Foo(object):
# --- docstring for Foo ---
def __init__(self, arg):
super(Foo, self).__init__()
self.arg = arg
def __call__(self):
print time.ctime(), 'process the %d begin ......' % self.arg
time.sleep(random.uniform(1, 3))
print time.ctime(), 'process the %d end !!!' % self.arg
if __name__ == '__main__':
print time.ctime, 'process begin...'
p_lst = list()
for i in xrange(4):
p_lst.append(Process(target=Foo(i)))
# 啓動子進程
for p in p_lst:
p.start()
# 等待子進程全部結束
for p in p_lst:
p.join()
print time.ctime, 'process end!!!'
運行結果:
<built-in function ctime> process begin...
Thu Apr 19 10:47:05 2018 process the 0 begin ......
Thu Apr 19 10:47:05 2018 process the 1 begin ......
Thu Apr 19 10:47:05 2018 process the 3 begin ......
Thu Apr 19 10:47:05 2018 process the 2 begin ......
Thu Apr 19 10:47:06 2018 process the 2 end !!!
Thu Apr 19 10:47:06 2018 process the 0 end !!!
Thu Apr 19 10:47:07 2018 process the 1 end !!!
Thu Apr 19 10:47:07 2018 process the 3 end !!!
<built-in function ctime> process end!!!
# 實例三:派生Process子類,並創建子類的實例(繼承Process重寫run方法)
class Myprocess(Process):
def __init__(self, arg):
super(Myprocess, self).__init__()
self.arg = arg
# 重寫run方法
def run(self):
print time.ctime(), 'process the %d begin ......' % self.arg
time.sleep(random.uniform(1, 3))
print time.ctime(), 'process the %d end !!!' % self.arg
if __name__ == '__main__':
print time.ctime, 'process begin...'
p_lst = list()
for i in xrange(4):
p_lst.append(Myprocess(i))
# 啓動子進程
for p in p_lst:
p.daemen = True #加入daemon,設置爲後臺進程
p.start()
# 等待子進程全部結束
for p in p_lst:
p.join()
print time.ctime, 'process end!!!'
運行結果:
<built-in function ctime> process begin...
Thu Apr 19 10:47:16 2018 process the 0 begin ......
Thu Apr 19 10:47:16 2018 process the 1 begin ......
Thu Apr 19 10:47:16 2018 process the 2 begin ......
Thu Apr 19 10:47:16 2018 process the 3 begin ......
Thu Apr 19 10:47:17 2018 process the 0 end !!!
Thu Apr 19 10:47:17 2018 process the 2 end !!!
Thu Apr 19 10:47:18 2018 process the 3 end !!!
Thu Apr 19 10:47:18 2018 process the 1 end !!!
<built-in function ctime> process end!!!
2. Pool類
當使用Process類管理非常多(幾十上百個)的進程時,就會顯得比較繁瑣,這是就可以使用Pool(進程池)來對進程進行統一管理。當池中進程已滿時,有新進程請求執行時,就會被阻塞,直到池中有進程執行結束,新的進程請求才會被放入池中並執行。
2.1 構造方法
def __init__(self, processes=None, initializer=None, initargs=(), maxtasksperchild=None)
processes:池中可容納的工作進程數量,默認情況使用os.cpu_count()返回的數值,一般默認即可
2.2 實例方法
apply(self, func, args=(), kwds={}):阻塞型進程池,會阻塞主進程,直到工作進程全部退出,一般不用這個
apply_async(self, func, args=(), kwds={}, callback=None):非阻塞型進程池
map(self, func, iterable, chunksize=None):與內置map行爲一致,它會阻塞主進程,直到map運行結束
map_async(self, func, iterable, chunksize=None, callback=None):非阻塞版本的map
close():關閉進程池,不在接受新任務
terminate():結束工作進程
join():阻塞主進程等待子進程退出,該方法必須在close或terminate之後執行
2.3 實例
# 進程池Pool類實例
def foo(i):
print time.ctime(), 'process the %d begin ......' % i
time.sleep(random.uniform(1, 3))
print time.ctime(), 'process the %d end !!!' % i
if __name__ == '__main__':
print time.ctime, 'process begin...'
pool = Pool(processes=2) # 設置進程池中最大並行工作進程數爲2
for i in xrange(4):
pool.apply_async(foo, args=(i, )) #提交4個子進程任務,非阻塞型進程池
pool.close()
pool.join()
print time.ctime, 'process end!!!'
運行結果:
<built-in function ctime> process begin...
Thu Apr 19 10:54:47 2018 process the 0 begin ......
Thu Apr 19 10:54:47 2018 process the 1 begin ......
Thu Apr 19 10:54:48 2018 process the 0 end !!!
Thu Apr 19 10:54:48 2018 process the 2 begin ......
Thu Apr 19 10:54:49 2018 process the 1 end !!!
Thu Apr 19 10:54:49 2018 process the 3 begin ......
Thu Apr 19 10:54:50 2018 process the 2 end !!!
Thu Apr 19 10:54:51 2018 process the 3 end !!!
<built-in function ctime> process end!!!
3. Queue類
Queue主要提供進程間通信以及共享數據等功能。除Queue外還可以使用Pipes實現進程間通信(Pipes是兩個進程間進行通信)
3.1 構造方法
def __init__(self, maxsize=0)
maxsize:用於設置隊列最大長度,當爲maxsize<=0時,隊列的最大長度會被設置爲一個非常大的值(我的系統中隊列最大長度被設置爲2147483647)
3.2 實例方法
put(self, obj, block=True, timeout=None)
1、block爲True,若隊列已滿,並且timeout爲正值,該方法會阻塞timeout指定的時間,直到隊列中有出現剩餘空間,如果超時,會拋出Queue.Full異常
2、block爲False,若隊列已滿,立即拋出Queue.Full異常
get(self, block=True, timeout=None)
block爲True,若隊列爲空,並且timeout爲正值,該方法會阻塞timeout指定的時間,直到隊列中有出現新的數據,如果超時,會拋出Queue.Empty異常
block爲False,若隊列爲空,立即拋出Queue.Empty異常
3.3 實例
# 隊列Queue類實例
def write(q):
for val in 'abcd':
print time.ctime(), 'put %s to queue' % val
q.put(val)
time.sleep(random.random())
def read(q):
while True:
value = q.get()
print time.ctime(), 'get %s from queue' % value
if __name__ == '__main__':
#主進程創建Queue,並作爲參數傳遞給子進程
q = Queue()
pw = Process(target=write, args=(q, ))
pr = Process(target=read, args=(q, ))
#啓動子進程pw,往Queue中寫入
pw.start()
#啓動子進程pr,從Queue中讀取
pr.start()
#等待寫進程執行結束
pw.join()
#終止讀取進程
pr.terminate()
運行結果:
Thu Apr 19 11:14:31 2018 put a to queue
Thu Apr 19 11:14:31 2018 get a from queue
Thu Apr 19 11:14:31 2018 put b to queue
Thu Apr 19 11:14:31 2018 get b from queue
Thu Apr 19 11:14:31 2018 put c to queue
Thu Apr 19 11:14:31 2018 get c from queue
Thu Apr 19 11:14:32 2018 put d to queue
Thu Apr 19 11:14:32 2018 get d from queue
4 Manager類
Manager是進程間數據共享的高級接口。
Manager()返回的manager對象控制了一個server進程,此進程包含的python對象可以被其他的進程通過proxies來訪問。從而達到多進程間數據通信且安全。Manager支持的類型有list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, Value和Array。
4.1 實例
# 進程間數據共享高級接口Manager類實例
# 此例是使用Manager管理一個用於多進程共享的dict數據
def worker(d, key, val):
print time.ctime(), "insert the k-v pair to dict begin: {%d: %d}" %(key, val)
time.sleep(random.uniform(1, 2))
d[key] = val #訪問共享數據
print time.ctime(), "insert the k-v pair to dict end: {%d: %d}" %(key, val)
if __name__ == '__main__':
print time.ctime(), "process for manager begin"
mgr = Manager()
d = mgr.dict()
pool = Pool(processes=4)
for i in range(10):
pool.apply_async(worker, args=(d, i, i*i))
pool.close()
pool.join()
print'Result: %s' % d
print time.ctime(), "process for manager end"
運行結果:
Thu Apr 19 11:28:36 2018 process for manager begin
Thu Apr 19 11:28:36 2018 insert the k-v pair to dict begin: {0: 0}
Thu Apr 19 11:28:36 2018 insert the k-v pair to dict begin: {1: 1}
Thu Apr 19 11:28:36 2018 insert the k-v pair to dict begin: {2: 4}
Thu Apr 19 11:28:36 2018 insert the k-v pair to dict begin: {3: 9}
Thu Apr 19 11:28:37 2018 insert the k-v pair to dict end: {2: 4}
Thu Apr 19 11:28:37 2018 insert the k-v pair to dict begin: {4: 16}
Thu Apr 19 11:28:38 2018 insert the k-v pair to dict end: {3: 9}
Thu Apr 19 11:28:38 2018 insert the k-v pair to dict begin: {5: 25}
Thu Apr 19 11:28:38 2018 insert the k-v pair to dict end: {0: 0}
Thu Apr 19 11:28:38 2018 insert the k-v pair to dict begin: {6: 36}
Thu Apr 19 11:28:38 2018 insert the k-v pair to dict end: {1: 1}
Thu Apr 19 11:28:38 2018 insert the k-v pair to dict begin: {7: 49}
Thu Apr 19 11:28:39 2018 insert the k-v pair to dict end: {5: 25}
Thu Apr 19 11:28:39 2018 insert the k-v pair to dict begin: {8: 64}
Thu Apr 19 11:28:39 2018 insert the k-v pair to dict end: {4: 16}
Thu Apr 19 11:28:39 2018 insert the k-v pair to dict begin: {9: 81}
Thu Apr 19 11:28:40 2018 insert the k-v pair to dict end: {7: 49}
Thu Apr 19 11:28:40 2018 insert the k-v pair to dict end: {6: 36}
Thu Apr 19 11:28:41 2018 insert the k-v pair to dict end: {8: 64}
Thu Apr 19 11:28:41 2018 insert the k-v pair to dict end: {9: 81}
Result: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}
Thu Apr 19 11:28:41 2018 process for manager end
注意:以上內容是個人使用的隨手記錄, 就是介紹了下簡單的使用
歡迎大家來吐槽,準備好瓜子飲料礦泉水,開整!!!
---------------------------------------------------------------------------------------
搞笑一則:能動手儘量別吵吵