python異步爬蟲

參考微信公衆號

event_loop：事件循環，相當於一個無限循環，我們可以把些函數註冊到這個事件循環上，當滿足條件發生的時候，就會調用對應的處理方法。
coroutine：中文翻譯叫協程，在 Python 中常指代爲協程對象類型，我們可以將協程對象註冊到時間循環中，它會被事件循環調用。我們可以使用 async 關鍵字來定義一個方法，這個方法在調用時不會立即被執行，而是返回一個協程對象。
task：任務，它是對協程對象的進一步封裝，包含了任務的各個狀態。
future：代表將來執行或沒有執行的任務的結果，實際上和task沒有本質區別。
參考文章鏈接： https://mp.weixin.qq.com/s?__biz=MzU4OTUwMDE1Mw==&mid=2247484614&idx=1&sn=9ed11ffd3658ad6d78755be18db51da8&chksm=fdcddf50caba5646b19bb32afa25e26769d97c6b45aec4a2f65edde406307c504d411cd07f43&mpshare=1&scene=1&srcid=0519URbZQ43kHZIRULUkhHv0&key=d0af80d42a2df10fba48d5e39e04e1fdf0f4e40f5339fedf6e90e9e45ce80b5e8db13beaff5447f3fb2a7604720dcd5ac15ee9ae3d8ea21cd9c6e8e4a34850cc3e77d482c9a2ffbe84b24cc367d7d3d6&ascene=1&uin=MjAzODMyMjAyOQ%3D%3D&devicetype=Windows+7&version=62060739&lang=zh_CN&pass_ticket=79DI0w8j6WY2Bw3np36cQk5diNkMK3KC0j9%2FdRLWWFnUcT2Oq5%2FAJVGB7oTXTZ%2BZ

1、定義協程

（1）使用async定義了一個方法
（2）調用該方法，返回一個coroutine協成對象
（3）使用get_event_loop()方法創建一個事件循環loop
（4）調用loop對象的run_until_complete()方法將協程註冊到事件循環loop中，然後啓動

import asyncio
async def execute(x):
    print('Number:', x)
coroutine = execute(1)
print('Coroutine:', coroutine)
print('After calling execute')
loop = asyncio.get_event_loop()
loop.run_until_complete(coroutine)
print('After calling loop')
# Coroutine: <coroutine object execute at 0x0000000002919E48>
# After calling execute
# Number: 1
# After calling loop

（5）定義task對象，調用loop對象的create_task()方法將coroutine對象轉化爲task對象，隨後我們打印輸出一下，發現它是 pending 狀態。接着我們將 task 對象添加到事件循環中得到執行，隨後我們再打印輸出一下 task 對象，發現它的狀態就變成了 finished。

import asyncio
async def execute(x):
    print('Number:', x)
    return x
coroutine = execute(1)
print('Coroutine:', coroutine)
print('After calling execute')
loop = asyncio.get_event_loop()
task = loop.create_task(coroutine)
print('Task:', task)
loop.run_until_complete(task)
print('Task:', task)
print('After calling loop')
# Coroutine: <coroutine object execute at 0x0000000002919E48>
# After calling execute
# Task: <Task pending coro=<execute() running at E:/project/python基礎.py:4>>
# Number: 1
# Task: <Task finished coro=<execute() done, defined at E:/project/python基礎.py:4> result=1>
# After calling loop

（6）定義task對象的另一種方式通過asyncio的ensure_future()方法，返回的也是task對象

import asyncio
async def execute(x):
    print('Number:', x)
    return x
coroutine = execute(1)
print('Coroutine:', coroutine)
print('After calling execute')
task = asyncio.ensure_future(coroutine)
print('Task:', task)
loop = asyncio.get_event_loop()
loop.run_until_complete(task)
print('Task:', task)
print('After calling loop')
# Coroutine: <coroutine object execute at 0x0000000002929E48>
# After calling execute
# Task: <Task pending coro=<execute() running at E:/project/python基礎.py:2>>
# Number: 1
# Task: <Task finished coro=<execute() done, defined at E:/project/python基礎.py:2> result=1>
# After calling loop

2、綁定回調

（1）調用add_done_callback()方法爲某個task綁定一個回調方法。我們將 callback() 方法傳遞給了封裝好的 task 對象，這樣當 task 執行完畢之後就可以調用 callback() 方法了，同時 task 對象還會作爲參數傳遞給 callback() 方法，調用 task 對象的 result() 方法就可以獲取返回結果了

import asyncio
import requests
async def request():
    url = 'https://www.baidu.com'
    status = requests.get(url)
    return status
def callback(task):
    print('Status:', task.result())
coroutine = request()
task = asyncio.ensure_future(coroutine)
task.add_done_callback(callback)
print('Task：', task)
loop = asyncio.get_event_loop()
loop.run_until_complete(task)
print('Task:', task)
# Task： <Task pending coro=<request() running at E:/project/python基礎.py:4> cb=[callback() at E:/project/python基礎.py:8]>
# Status: <Response [200]>
# Task: <Task finished coro=<request() done, defined at E:/project/python基礎.py:4> result=<Response [200]>>

（2）直接調用task運行完畢之後直接調用result()方法獲取結果

import asyncio
import requests
async def request():
    url = 'https://www.baidu.com'
    status = requests.get(url)
    return status
coroutine = request()
task = asyncio.ensure_future(coroutine)
print('Task:', task)
loop = asyncio.get_event_loop()
loop.run_until_complete(task)
print('Task:', task)
print('Task Result:', task.result())
# Task: <Task pending coro=<request() running at E:/project/python基礎.py:3>>
# Task: <Task finished coro=<request() done, defined at E:/project/python基礎.py:3> result=<Response [200]>>
# Task Result: <Response [200]>

3、多任務協程

定義一個task列表，然後使用asyncio的wait()方法即可執行；我們使用一個 for 循環創建了五個 task，組成了一個列表，然後把這個列表首先傳遞給了 asyncio 的 wait() 方法，然後再將其註冊到時間循環中，就可以發起五個任務了。最後我們再將任務的運行結果輸出出來

import asyncio
import requests
async def request():
    url = 'https://www.baidu.com'
    status = requests.get(url)
    return status
tasks = [asyncio.ensure_future(request()) for _ in range(5)]
print('Tasks:', tasks)
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
for task in tasks:
    print('Task Result:', task.result())
# Tasks: [<Task pending coro=<request() running at E:/project/python基礎.py:3>>, <Task pending coro=<request() running at E:/project/python基礎.py:3>>, <Task pending coro=<request() running at E:/project/python基礎.py:3>>, <Task pending coro=<request() running at E:/project/python基礎.py:3>>, <Task pending coro=<request() running at E:/project/python基礎.py:3>>]
# Task Result: <Response [200]>
# Task Result: <Response [200]>
# Task Result: <Response [200]>
# Task Result: <Response [200]>
# Task Result: <Response [200]>

4、協程實現

（1）使用 await 可以將耗時等待的操作掛起，讓出控制權。當協程執行的時候遇到 await，時間循環就會將本協程掛起，轉而去執行別的協程，直到其他的協程掛起或執行完畢。

import asyncio
import requests
import time
start = time.time()
async def get(url):
    return requests.get(url)
async def request():
    url = 'https://www.baidu.com'
    print('Waiting for', url)
    response = await get(url)
    print('Get response from', url, 'Result', response.status_code)
tasks = [asyncio.ensure_future(request()) for _ in range(5)]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
print('Cost time:', end - start)
# Waiting for https://www.baidu.com
# Get response from https://www.baidu.com Result 200
# Waiting for https://www.baidu.com
# Get response from https://www.baidu.com Result 200
# Waiting for https://www.baidu.com
# Get response from https://www.baidu.com Result 200
# Waiting for https://www.baidu.com
# Get response from https://www.baidu.com Result 200
# Waiting for https://www.baidu.com
# Get response from https://www.baidu.com Result 200
# Cost time: 0.45502614974975586

5、使用aiohttp

aiohttp是一個支持異步請求的庫，利用它和asyncio配合我們可以非常方便的實現異步請求操作。

import asyncio
import aiohttp
import time
start = time.time()
async def get(url):
    session = aiohttp.ClientSession()
    response = await session.get(url)
    result = await response.text()
    await session.close()
    return result
async def request():
    url = 'http://www.newsmth.net/nForum/#!mainpage'
    print('Waiting for', url)
    result = await get(url)
    print('Get response from', url, 'Result:', result)
tasks = [asyncio.ensure_future(request()) for _ in range(5)]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
print('Cost time:', end - start)

python異步爬蟲

參考微信公衆號

1、定義協程

2、綁定回調

3、多任務協程

4、協程實現

5、使用aiohttp

lightdb hash index的性能和限制

Python爬蟲bug_unable to decode value

Python爬蟲requests後的html亂碼解決(gzip, deflate, br)

Python_datetime模塊使用

python相關渲染庫Selenium、Puppeteer、Splash安裝

Python_深度學習環境配置

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結