Python屏蔽GIL鎖實踐(調研+轉載+整理)

環境:

Ubuntu19.10

四核八線程.

題外話,八線程屬於超線程概念,程序員不可控制,屬於操作系統調度的工作.

基本概念:

什麼時候使用併發/並行?

資料中有兩種說法:

一種是:

涉及到多核就是並行,單核IO異步就是併發

另一種是:

整個系統服務於IO異步任務就是併發,涉及到CPU密集就是並行

本文以第一種爲準

--------------------------------

消除GIL鎖爲什麼會導致單線程變慢?

參考[6]的回答

Python多線程相當於單核多線程[7]

--------------------------------

網上資料的多線程包含:

單核多線程和多核多線程.

單核多線程就IO異步.

多核多線程大多數情況下指的是多核情況.

根據[4]

I'd welcome a set of patches into Py3k only if the performance for a single-threaded program (and for a multi-threaded but I/O-bound program) does not decrease”(這個意思是如果移除GIL鎖,那麼單線程性能會下降,也就是I/O異步性能會下降)

Each Python process gets its own Python interpreter and memory space so the GIL won't be a problem. (每個進程都有一個GIL鎖)

網上說的"Python的多線程是雞肋"的意思指的是多核多線程

有了多線程執行異步爲啥還要多協程執行IO異步?

根據[5]:

好處是協程的開銷成本更低.

GIL鎖影響的是IO密集還是CPU密集任務?

The GIL does not have much impact on the performance of I/O-bound multi-threaded programs as the lock is shared between threads while they are waiting for I/O.[4](影響的是CPU密集任務而不是IO密集任務)

GIL鎖在你調用開源庫的時候,依然存在嗎?

取決於底層是不是C寫的,如果是C寫的,屏蔽了GIL鎖,那就不存在,否則就存在

----------------------------------------------------------------------------------------------------------------------

網上關於屏蔽GIL鎖的資料如下:

[1]屏蔽了GIL鎖,沒有完整代碼

[3]講了原理,但是沒有實操

本篇博客重點解析[2].

GIL鎖每個進程都有一個,鎖的到底是什麼?

鎖的是多線程進行CPU密集計算時候的場景.案例如下[4]:

單線程:

# single_threaded.py
import time

COUNT = 50000000

def countdown(n):
    while n>0:
        n -= 1

start = time.time()
countdown(COUNT)
end = time.time()

print('Time taken in seconds -', end - start)

Time taken in seconds - 2.2484254837036133

多線程:

# multi_threaded.py
import time
from threading import Thread

COUNT = 50000000

def countdown(n):
    while n>0:
        n -= 1

t1 = Thread(target=countdown, args=(COUNT//2,))
t2 = Thread(target=countdown, args=(COUNT//2,))

start = time.time()
t1.start()
t2.start()
t1.join()
t2.join()
end = time.time()

print('Time taken in seconds -', end - start)

Time taken in seconds - 3.074549674987793

多進程:

from multiprocessing import Pool
import time

COUNT = 50000000
def countdown(n):
    while n>0:
        n -= 1

if __name__ == '__main__':
    pool = Pool(processes=2)
    start = time.time()
    r1 = pool.apply_async(countdown, [COUNT//2])
    r2 = pool.apply_async(countdown, [COUNT//2])
    pool.close()
    pool.join()
    end = time.time()
    print('Time taken in seconds -', end - start)

結論:

CPU密集任務速度上,

多核多進程>多核多線程>單線程

----------------------------------------------下面是屏蔽GIL鎖實踐------------------------------------------------------------------------------

回到前面一個話題,python的多線程真的是雞肋嗎?

並不是,請看下面實驗,下面實驗受到[2]的啓發,但是因爲[2]的代碼不完整,所以我自己想出了一個demo

我們把count提升100倍,下面是三個代碼文件:

pycall.c

#include <stdio.h>  
#include <stdlib.h>  
// #include <iostream.h>  
int foo(int a, int b)  
{  
	long long int count=5000000000;
	   while( count > 0 )
   {
       // cout << "a 的值：" << a << endl;
       count--;
       if(count==2500000000)
            printf("計數中%lld\n", count);
       if(count==0)
            printf("計數結束%lld\n", count);
   };
  printf("you input %d and %d\n", a, b);  
  return 1;  
}

因爲我們等下要進行多線程測試,

所以我們在上面代碼中故意加入一些if語句用來監控計數過程,如果多線程生效那麼就會打印兩次,

如果GIL鎖依然存在,那麼整個運行過程會比單線程還要慢一倍左右

pycall-1thread.py

import ctypes  
import threading
import time

if __name__ == '__main__':
    start=time.time()
    my_lib=ctypes.cdll.LoadLibrary("./libpycall.so")  
    t1=threading.Thread(target=my_lib.foo,args=(1,1))
    t1.start()
    t1.join()
    end=time.time()
    print('耗時=',end-start)

pycall-2thread.py

import ctypes  
import threading
import time

if __name__ == '__main__':
    start=time.time()
    my_lib=ctypes.cdll.LoadLibrary("./libpycall.so")  
    t1=threading.Thread(target=my_lib.foo,args=(1,1))
    t2=threading.Thread(target=my_lib.foo,args=(2,2))
    t1.start()
    t2.start()
    t1.join()
    t2.join()
    end=time.time()
    print('耗時=',end-start)

運行方法:

先生成.so文件

gcc -o libpycall.so -shared -fPIC pycall.c

然後:

命令	運行結果
python pycall-1thread.py	計數中2500000000 計數結束0 you input 1 and 1 耗時= 10.828569173812866
python pycall-2thread.py	計數中2500000000 計數中2500000000 計數結束0 you input 1 and 1 計數結束0 you input 2 and 2 耗時= 10.749252796173096