Python屏蔽GIL锁实践(调研+转载+整理)

环境:

Ubuntu19.10

四核八线程.

题外话,八线程属于超线程概念,程序员不可控制,属于操作系统调度的工作.

基本概念:

什么时候使用并发/并行?

资料中有两种说法:

一种是:

涉及到多核就是并行,单核IO异步就是并发

另一种是:

整个系统服务于IO异步任务就是并发,涉及到CPU密集就是并行

本文以第一种为准

--------------------------------

消除GIL锁为什么会导致单线程变慢?

参考[6]的回答

Python多线程相当於单核多线程[7]

--------------------------------

网上资料的多线程包含:

单核多线程和多核多线程.

单核多线程就IO异步.

多核多线程大多数情况下指的是多核情况.

根据[4]

I'd welcome a set of patches into Py3k only if the performance for a single-threaded program (and for a multi-threaded but I/O-bound program) does not decrease”(这个意思是如果移除GIL锁,那么单线程性能会下降,也就是I/O异步性能会下降)

Each Python process gets its own Python interpreter and memory space so the GIL won't be a problem. (每个进程都有一个GIL锁)

网上说的"Python的多线程是鸡肋"的意思指的是多核多线程

有了多线程执行异步为啥还要多协程执行IO异步?

根据[5]:

好处是协程的开销成本更低.

GIL锁影响的是IO密集还是CPU密集任务?

The GIL does not have much impact on the performance of I/O-bound multi-threaded programs as the lock is shared between threads while they are waiting for I/O.[4](影响的是CPU密集任务而不是IO密集任务)

GIL锁在你调用开源库的时候,依然存在吗?

取决于底层是不是C写的,如果是C写的,屏蔽了GIL锁,那就不存在,否则就存在

----------------------------------------------------------------------------------------------------------------------

网上关于屏蔽GIL锁的资料如下:

[1]屏蔽了GIL锁,没有完整代码

[3]讲了原理,但是没有实操

本篇博客重点解析[2].

GIL锁每个进程都有一个,锁的到底是什么?

锁的是多线程进行CPU密集计算时候的场景.案例如下[4]:

单线程:

# single_threaded.py
import time

COUNT = 50000000

def countdown(n):
    while n>0:
        n -= 1

start = time.time()
countdown(COUNT)
end = time.time()

print('Time taken in seconds -', end - start)

Time taken in seconds - 2.2484254837036133

多线程:

# multi_threaded.py
import time
from threading import Thread

COUNT = 50000000

def countdown(n):
    while n>0:
        n -= 1

t1 = Thread(target=countdown, args=(COUNT//2,))
t2 = Thread(target=countdown, args=(COUNT//2,))

start = time.time()
t1.start()
t2.start()
t1.join()
t2.join()
end = time.time()

print('Time taken in seconds -', end - start)

Time taken in seconds - 3.074549674987793

多进程:

from multiprocessing import Pool
import time

COUNT = 50000000
def countdown(n):
    while n>0:
        n -= 1

if __name__ == '__main__':
    pool = Pool(processes=2)
    start = time.time()
    r1 = pool.apply_async(countdown, [COUNT//2])
    r2 = pool.apply_async(countdown, [COUNT//2])
    pool.close()
    pool.join()
    end = time.time()
    print('Time taken in seconds -', end - start)

结论:

CPU密集任务速度上,

多核多进程>多核多线程>单线程

----------------------------------------------下面是屏蔽GIL锁实践------------------------------------------------------------------------------

回到前面一个话题,python的多线程真的是鸡肋吗?

并不是,请看下面实验,下面实验受到[2]的启发,但是因为[2]的代码不完整,所以我自己想出了一个demo

我们把count提升100倍,下面是三个代码文件:

pycall.c

#include <stdio.h>  
#include <stdlib.h>  
// #include <iostream.h>  
int foo(int a, int b)  
{  
	long long int count=5000000000;
	   while( count > 0 )
   {
       // cout << "a 的值：" << a << endl;
       count--;
       if(count==2500000000)
            printf("计数中%lld\n", count);
       if(count==0)
            printf("计数结束%lld\n", count);
   };
  printf("you input %d and %d\n", a, b);  
  return 1;  
}

因为我们等下要进行多线程测试,

所以我们在上面代码中故意加入一些if语句用来监控计数过程,如果多线程生效那么就会打印两次,

如果GIL锁依然存在,那么整个运行过程会比单线程还要慢一倍左右

pycall-1thread.py

import ctypes  
import threading
import time

if __name__ == '__main__':
    start=time.time()
    my_lib=ctypes.cdll.LoadLibrary("./libpycall.so")  
    t1=threading.Thread(target=my_lib.foo,args=(1,1))
    t1.start()
    t1.join()
    end=time.time()
    print('耗时=',end-start)

pycall-2thread.py

import ctypes  
import threading
import time

if __name__ == '__main__':
    start=time.time()
    my_lib=ctypes.cdll.LoadLibrary("./libpycall.so")  
    t1=threading.Thread(target=my_lib.foo,args=(1,1))
    t2=threading.Thread(target=my_lib.foo,args=(2,2))
    t1.start()
    t2.start()
    t1.join()
    t2.join()
    end=time.time()
    print('耗时=',end-start)

运行方法:

先生成.so文件

gcc -o libpycall.so -shared -fPIC pycall.c

然后:

命令	运行结果
python pycall-1thread.py	计数中2500000000 计数结束0 you input 1 and 1 耗时= 10.828569173812866
python pycall-2thread.py	计数中2500000000 计数中2500000000 计数结束0 you input 1 and 1 计数结束0 you input 2 and 2 耗时= 10.749252796173096