Python網絡爬蟲(十四)——threading

簡介

對於爬取圖片或者爬取章節數目過多的小說來說，採取同步的方式進行下載會導致效率的下降，這對於網絡爬蟲來說是一個很大的缺陷。而使用多線程則可以避免這個問題，提高整個爬取過程的效率。

多線程(multithreading)，是指從軟件或者硬件上實現多個線程併發執行的技術。具有多線程能力的計算機因有硬件支持而能夠在同一時間執行多於一個線程，進而提升整體處理性能。

threading

threading 是 python 中用來進行多線程的一個模塊。

基本使用

import threading
import time

def func1():
    for i in range(3):
        print('func1')
        time.sleep(1)

def func2():
    for i in range(3):
        print('func2')
        time.sleep(1)

if __name__ == '__main__':
    thread1 = threading.Thread(target=func1)
    thread2 = threading.Thread(target=func2)

    thread1.start()
    thread2.start()

結果爲：

func1
func2
func2
func1
func1
func2

上邊的結果顯示執行順序並不是先執行 func1 再執行 func2，而是 func1 和 func2 同時進行，這就是多線程。

派生 Thread 類

也可以將上邊提到的 Thread 類進行派生，並添加 run 方法，實現 run 方法的自動運行：

import threading
import time

class func1Thread(threading.Thread):
    def run(self):
        for i in range(3):
            print('func1',threading.current_thread())
            time.sleep(1)

class func2Thread(threading.Thread):
    def run(self):
        for i in range(3):
            print('func2',threading.current_thread())
            time.sleep(1)

if __name__ == '__main__':
    thread1 = func1Thread()
    thread2 = func2Thread()

    thread1.start()
    thread2.start()
    print(threading.enumerate())

結果爲：

func1 <func1Thread(Thread-1, started 7736)>
func2 <func2Thread(Thread-2, started 2972)>
[<_MainThread(MainThread, started 804)>, <func1Thread(Thread-1, started 7736)>, <func2Thread(Thread-2, started 2972)>]
func1 <func1Thread(Thread-1, started 7736)>
func2 <func2Thread(Thread-2, started 2972)>
func1 <func1Thread(Thread-1, started 7736)>
func2 <func2Thread(Thread-2, started 2972)>

上邊的程序中存在三個線程，主線程 MainThread 對應 main 函數，另外兩個線程 Thread-1 和 Thread-2 分別爲 func1Thread 和 func2Thread 構建的線程。

上邊使用到的兩個函數：

threading.enumerate() 可以獲得當前的所有線程信息
threading.current_thread() 可以獲得當前的線程信息

全局變量

在同一進程中，多線程能夠大大提升任務的執行效率，而因爲所有線程共享進程中的全局變量，因此也會造成某些問題。

import threading
import time

NUM = 0

def func():
    global NUM
    for i in range(3):
        NUM += 1
        time.sleep(1)
    print(NUM)

if __name__ == '__main__':
    for i in range(3):
        th = threading.Thread(target=func)
        th.start()

結果爲：

9
9
9

實際上我們想要的結果是 3，6，9，但是多線程忽略了各個線程之間的執行順序，因此最後的結果跟預想的結果不同。

Lock

而爲了解決多線程下共享變量的問題，threading 提供了 Lock 類，該類能夠在某個線程訪問某個變量的時候對變量加鎖，此時其它線程就不能訪問該變量，直到該 Lock 被釋放其它線程才能夠訪問該變量。

import threading
import time

NUM = 0
glock = threading.Lock()

def func():
    global NUM
    glock.acquire()
    for i in range(3):
        NUM += 1
        time.sleep(1)
    glock.release()
    print(NUM)

if __name__ == '__main__':
    for i in range(3):
        th = threading.Thread(target=func)
        th.start()

結果爲：

3
6
9

此時當某個線程獲得該全局變量的時候，別的線程就不能獲得該全局變量，因此保證了單個線程對全局變量的訪問。

生產者-消費者

生產者消費者模式是多線程開發中的一種常見模式。簡單的說，生產者用來產生數據，消費者用來處理數據，在生產者與消費者之間還存在緩衝區，圖示爲：

基本實現

import threading
import random
import time

TOTAL = 100
TIMES = 0

class Generant(threading.Thread):
    def run(self):
        global TIMES,TOTAL
        while True:
            produce = random.randint(0,100)
            if TIMES >= 10:
                break
            TOTAL += produce
            print('%s,TIMES = %d,produce = %d,TOTAL = %d' % (threading.current_thread(),TIMES,produce,TOTAL))
            TIMES += 1
            time.sleep(1)

class Consumer(threading.Thread):
    def run(self):
        global TIMES,TOTAL
        while True:
            consumption = random.randint(0,100)
            if TOTAL > consumption:
                TOTAL -= consumption
                print('%s,TIMES = %d,consumption = %d,TOTAL = %d' % (threading.current_thread(),TIMES,consumption,TOTAL))
                time.sleep(1)
            else:
                if TIMES >= 10:
                    break
                print('buffer error.')

if __name__ == '__main__':
    for i in range(3):
        Consumer(name='ConsumerThread%d' % i).start()

    for i in range(3):
        Generant(name='ConsumerThread%d' % i).start()

結果爲：

<Consumer(ConsumerThread0, started 15168)>,TIMES = 0,consumption = 43,TOTAL = 57
buffer error.
<Consumer(ConsumerThread1, started 15016)>,TIMES = 0,consumption = 12,TOTAL = 45
buffer error.
buffer error.
<Generant(ConsumerThread0, started 6100)>,TIMES = 0,produce = 49,TOTAL = 94
<Consumer(ConsumerThread2, started 12324)>,TIMES = 1,consumption = 90,TOTAL = 4
<Generant(ConsumerThread1, started 8116)>,TIMES = 1,produce = 54,TOTAL = 58
<Generant(ConsumerThread2, started 8032)>,TIMES = 2,produce = 7,TOTAL = 65
buffer error.
<Consumer(ConsumerThread1, started 15016)>,TIMES = 3,consumption = 9,TOTAL = 56
<Generant(ConsumerThread0, started 6100)>,TIMES = 3,produce = 86,TOTAL = 142
<Consumer(ConsumerThread2, started 12324)>,TIMES = 4,consumption = 88,TOTAL = 54
<Consumer(ConsumerThread0, started 15168)>,TIMES = 4,consumption = 10,TOTAL = 44
<Generant(ConsumerThread2, started 8032)>,TIMES = 4,produce = 9,TOTAL = 53
<Generant(ConsumerThread1, started 8116)>,TIMES = 5,produce = 77,TOTAL = 130
<Consumer(ConsumerThread1, started 15016)>,TIMES = 6,consumption = 54,TOTAL = 76
<Generant(ConsumerThread0, started 6100)>,TIMES = 6,produce = 39,TOTAL = 115
<Generant(ConsumerThread1, started 8116)>,TIMES = 7,produce = 7,TOTAL = 122
<Generant(ConsumerThread2, started 8032)>,TIMES = 7,produce = 37,TOTAL = 159
<Consumer(ConsumerThread2, started 12324)>,TIMES = 7,consumption = 6,TOTAL = 153
<Consumer(ConsumerThread0, started 15168)>,TIMES = 7,consumption = 63,TOTAL = 90
<Generant(ConsumerThread0, started 6100)>,TIMES = 9,produce = 21,TOTAL = 111
<Consumer(ConsumerThread1, started 15016)>,TIMES = 10,consumption = 17,TOTAL = 94
<Consumer(ConsumerThread2, started 12324)>,TIMES = 10,consumption = 29,TOTAL = 65
<Consumer(ConsumerThread0, started 15168)>,TIMES = 10,consumption = 20,TOTAL = 45
<Consumer(ConsumerThread1, started 15016)>,TIMES = 10,consumption = 41,TOTAL = 4

上邊的代碼段創建了三個生產者線程和三個消費者線程，這裏的 TOTAL 可以認爲是緩衝區，並用 TIMES 限制了生產者生產的次數，但並未對消費者的消費情況做出限制。雖然輸出的結果並沒有錯誤，但是涉及到全局變量 TIMES 和 TOTAL 的使用，因此爲了安全，最好還是使用 Lock。

Lock 實現

import threading
import random
import time

TOTAL = 100
TIMES = 0
LOCK = threading.Lock()

class Generant(threading.Thread):
    def run(self):
        global TIMES,TOTAL,LOCK
        while True:
            produce = random.randint(0,100)
            LOCK.acquire()
            if TIMES >= 10:
                LOCK.release()
                break
            TOTAL += produce
            print('%s,TIMES = %d,produce = %d,TOTAL = %d' % (threading.current_thread(),TIMES,produce,TOTAL))
            TIMES += 1
            LOCK.release()
            time.sleep(1)

class Consumer(threading.Thread):
    def run(self):
        global TIMES,TOTAL,LOCK
        while True:
            consumption = random.randint(0,100)
            LOCK.acquire()
            if TOTAL > consumption:
                TOTAL -= consumption
                print('%s,TIMES = %d,consumption = %d,TOTAL = %d' % (threading.current_thread(),TIMES,consumption,TOTAL))
                time.sleep(1)
            else:
                if TIMES >= 10:
                    LOCK.release()
                    break
                print('buffer error.')
            LOCK.release()

if __name__ == '__main__':
    for i in range(3):
        Consumer(name='ConsumerThread%d' % i).start()

    for i in range(3):
        Generant(name='ConsumerThread%d' % i).start()

結果爲：

<Consumer(ConsumerThread0, started 14616)>,TIMES = 0,consumption = 79,TOTAL = 21
buffer error.
buffer error.
<Generant(ConsumerThread0, started 11508)>,TIMES = 0,produce = 17,TOTAL = 38
<Generant(ConsumerThread1, started 9180)>,TIMES = 1,produce = 42,TOTAL = 80
<Generant(ConsumerThread2, started 12116)>,TIMES = 2,produce = 61,TOTAL = 141
<Consumer(ConsumerThread0, started 14616)>,TIMES = 3,consumption = 42,TOTAL = 99
<Consumer(ConsumerThread1, started 6692)>,TIMES = 3,consumption = 59,TOTAL = 40
buffer error.
buffer error.
buffer error.
<Consumer(ConsumerThread0, started 14616)>,TIMES = 3,consumption = 37,TOTAL = 3
<Generant(ConsumerThread0, started 11508)>,TIMES = 3,produce = 66,TOTAL = 69
buffer error.
<Consumer(ConsumerThread2, started 13116)>,TIMES = 4,consumption = 52,TOTAL = 17
<Generant(ConsumerThread2, started 12116)>,TIMES = 4,produce = 35,TOTAL = 52
<Generant(ConsumerThread1, started 9180)>,TIMES = 5,produce = 76,TOTAL = 128
<Consumer(ConsumerThread0, started 14616)>,TIMES = 6,consumption = 13,TOTAL = 115
<Consumer(ConsumerThread1, started 6692)>,TIMES = 6,consumption = 77,TOTAL = 38
buffer error.
buffer error.
buffer error.
buffer error.
buffer error.
buffer error.
buffer error.
<Consumer(ConsumerThread2, started 13116)>,TIMES = 6,consumption = 2,TOTAL = 36
<Generant(ConsumerThread0, started 11508)>,TIMES = 6,produce = 14,TOTAL = 50
<Generant(ConsumerThread1, started 9180)>,TIMES = 7,produce = 87,TOTAL = 137
<Consumer(ConsumerThread0, started 14616)>,TIMES = 8,consumption = 94,TOTAL = 43
<Generant(ConsumerThread2, started 12116)>,TIMES = 8,produce = 55,TOTAL = 98
<Consumer(ConsumerThread1, started 6692)>,TIMES = 9,consumption = 81,TOTAL = 17
buffer error.
<Generant(ConsumerThread1, started 9180)>,TIMES = 9,produce = 18,TOTAL = 35
<Consumer(ConsumerThread2, started 13116)>,TIMES = 10,consumption = 8,TOTAL = 27

雖然增加了 Lock 之後，輸出的結果並沒有發生什麼變化，但是這樣對全局變量的訪問形式無疑會安全很多。

Condition

上邊利用 Lock 雖然實現了對全局變量的安全訪問，但是在生產者和消費者中都是使用條件爲 True 的 while 循環來實現的，循環的結束則是通過 if 後 break 跳出的，這就導致了在每一次循環開始都要上鎖，在每一次跳出循環都要解鎖。雖然這樣的簡短程序的運行效率可能不會受到影響，但是對於一些比較大的項目來說，可能會耗費很多的 CPU 資源，因此可以使用其它方式。

threading.Condition 可以在沒有獲取到數據的時候一直處於阻塞等待狀態，一旦獲取數據的條件滿足就可以使用 notify 通知其它處於等待狀態的線程。這樣就可以不用想 Lock 一直地進行上鎖和解鎖地操作，從而能夠提高程序的性能。

threading.Condition 主要的相關函數爲：

acquire：和之前一樣，對資源上鎖
release：和之前一樣，對資源解鎖
wait：使線程處於等待狀態，並釋放鎖。可以被 notify 函數喚醒，喚醒之後繼續等待上鎖，上鎖後繼續執行
notify：通知某個正在等待的線程，默認爲第一個
notify_all：通知所有正在等待的線程

import threading
import random
import time

TOTAL = 100
TIMES = 0
TOTAL_TIMES = 5
CONDITION = threading.Condition()


class Generant(threading.Thread):
    def run(self):
        global TIMES,TOTAL,CONDITION
        while True:
            produce = random.randint(0,100)
            CONDITION.acquire()
            if TIMES >= TOTAL_TIMES:
                CONDITION.release()
                break
            TOTAL += produce
            print('%s,TIMES = %d,produce = %d,TOTAL = %d' % (threading.current_thread(),TIMES,produce,TOTAL))
            TIMES += 1
            time.sleep(0.5)
            CONDITION.notify_all()
            CONDITION.release()


class Consumer(threading.Thread):
    def run(self):
        global TOTAL,TIMES,TOTAL_TIMES,CONDITION
        while True:
            consumption = random.randint(0,100)
            CONDITION.acquire()
            # 這裏使用 while 進行判斷而不是 if，如果 TOTAL < consumption 就將當前線程掛起之後
            # 很可能下次的進程也是 TOTAL < consumption 的。
            while TOTAL < consumption:
                # 這裏設置 TIMES >= TOTAL_TIMES 作爲程序中止的關鍵，如果採用 Buffer Error 計數器作爲 flag
                # 則如果在 TIMES 達到最大值，計數器達到最大值之前所有的線程都陷入 wait，就會無限期的 wait 下去
                # 而這裏的 TIMES 作爲中止條件，剛好也符合生產者的停止生產條件
                if TIMES >= TOTAL_TIMES:
                    CONDITION.release()
                    return
                print('Buffer Error.')
                CONDITION.wait()
            TOTAL -= consumption
            print('%s,TIMES = %d,consumption = %d,TOTAL = %d' % (threading.current_thread(), TIMES, consumption, TOTAL))
            time.sleep(0.5)
            CONDITION.release()


if __name__ == '__main__':
    for i in range(3):
        Consumer(name='ConsumerThread%d' % i).start()

    for i in range(3):
        Generant(name='GenerantThread%d' % i).start()

結果爲：

<Consumer(ConsumerThread0, started 12300)>,TIMES = 0,consumption = 16,TOTAL = 84
<Consumer(ConsumerThread1, started 6488)>,TIMES = 0,consumption = 67,TOTAL = 17
Buffer Error.
<Generant(GenerantThread0, started 4132)>,TIMES = 0,produce = 75,TOTAL = 92
<Generant(GenerantThread1, started 13176)>,TIMES = 1,produce = 12,TOTAL = 104
<Generant(GenerantThread2, started 15288)>,TIMES = 2,produce = 10,TOTAL = 114
<Consumer(ConsumerThread0, started 12300)>,TIMES = 3,consumption = 76,TOTAL = 38
Buffer Error.
<Generant(GenerantThread0, started 4132)>,TIMES = 3,produce = 11,TOTAL = 49
<Consumer(ConsumerThread2, started 8716)>,TIMES = 4,consumption = 18,TOTAL = 31
<Generant(GenerantThread1, started 13176)>,TIMES = 4,produce = 32,TOTAL = 63
<Consumer(ConsumerThread0, started 12300)>,TIMES = 5,consumption = 43,TOTAL = 20

queue

之前我們提到過，多線程會忽視執行的順序，造成輸出次序的紊亂，而如果是跟順序有關的數據存儲是否也能夠使用多線程呢？

在 python 中內置了一個線程安全的隊列模塊 queue。queue 中包括了 FIFO queue，LIFO queue，這些 queue 的實現也都借用了 Lock 的操作，使之能夠直接在多線程中使用。常見函數爲：

Queue()：創建一個先進先出的 queue
qsize()：返回 queue 大小
empty()：queue 判空
full()：queue 判滿
get()：queue 出棧
put()：queue 入棧

import threading
import requests
from urllib import request
from lxml import etree
import re
import queue
import os
import time

headers = {
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36"
}

class Generrant(threading.Thread):
    def __init__(self,url_queue,img_queue,*arg,**argv):
        super(Generrant,self).__init__(*arg,**argv)
        self.url_queue = url_queue
        self.img_queue = img_queue

    def run(self):
        while True:
            if self.url_queue.empty():
                pass
            url = self.url_queue.get()
            self.parse_queue(url)

    def parse_queue(self,url):
        global headers
        response = requests.get(url,headers=headers)
        text = response.text
        html = etree.HTML(text)
        imgs = html.xpath("//div[@class='page-content text-center']//img")
        for img in imgs:
            if img.get('class') == 'gif':
                continue
            img_url = img.xpath('.//@data-original')[0]
            suffix = os.path.splitext(img_url)[1]
            alt = img.xpath('.//@alt')[0]
            alt = re.sub(r'[，。？?,/\\·]','',alt)
            img_name = alt+suffix
            self.img_queue.put((img_url,img_name))



class Consumer(threading.Thread):
    def __init__(self,url_queue,img_queue,*arg,**argv):
        super(Consumer,self).__init__(*arg,**argv)
        self.url_queue = url_queue
        self.img_queue = img_queue

    def run(self):
        while True:
            if self.img_queue.empty():
                if self.url_queue.empty():
                    return
            url, filename = self.img_queue.get(block=True)
            request.urlretrieve(url,'images/'+filename)
            time.sleep(1)


if __name__ == '__main__':
    url_queue = queue.Queue(100)
    img_queue = queue.Queue(500)
    for i in range(1,10):
        url = "http://www.doutula.com/photo/list/?page=%d" % i
        url_queue.put(url)

    for i in range(5):
        Thread_G = Generrant(url_queue,img_queue)
        Thread_G.start()

    for i in range(5):
        Thread_C = Consumer(url_queue,img_queue)
        Thread_C.start()

GIL

官網下載的 python 的解釋器爲 CPython。現在的計算機一般都是多核的，而 CPython 解釋器中的多線程實際上只利用了其中的一個核。
在同一時刻只有一個線程在運行，而爲了保證同一時刻只有一個線程執行，CPython 解釋器利用了全局解釋器鎖(Global Interpreter Lock,GIL)來實現。
因爲 CPython 解釋器的內存管理不是線程安全的，因此使用全局解釋器鎖也是必須的

當然不同的解釋器也不一定都存在 GIL。而 CPython 中的 GIL 雖然是並不是真正的多線程，但是在處理 IO 密集型的操作時，還是很能夠提高執行效率的。在對於 CPU 密集型的操作時，則最好使用多進程實現。

Python網絡爬蟲(十四)——threading

簡介

threading

基本使用

派生 Thread 類

全局變量

Lock

生產者-消費者

基本實現

Lock 實現

Condition

queue

GIL

Python網絡爬蟲(二十三)——Redis

Python網絡爬蟲(十九)——CrawlSpider

Python網絡爬蟲(二十四)——Scrapy-Redis

Python網絡爬蟲(二十二)——Downloader Middlewares

Python網絡爬蟲(二十一)——Request 和 Response

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結