Python爬蟲（三）— 深入瞭解Requests模塊

前言

以下關於Requests的內容講解，強烈推薦深入瞭解的查看官方文檔。

中文版：Requests、 http://cn.python-requests.org/zh_CN/latest/

Requests

Requests是用python語言基於urllib編寫的，採用的是Apache2 Licensed開源協議的HTTP庫。
默認安裝好python之後，是沒有安裝requests模塊的，需要單獨通過pip安裝。
requests是python實現的最簡單易用的HTTP庫，建議爬蟲使用requests庫。

Get請求

以下代碼使用了對 get請求的參數、請求的header、請求的超時設置、請求的返回參數進行示例。可自行根據需要進行刪減。

import requests

# 第一種設置請求參數的方法：get請求的參數寫成字典的形式，作爲get請求的params參數傳入
data = {
    "name":"luozheng",
    "age":22
}
# 第二種設置請求參數的方法  response = requests.get("http://httpbin.org/get?name=luozhengfan&age=23")

# 添加header
headers = {
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
}
response = requests.get("http://httpbin.org/get", params=data, headers=headers, timeout=5)
print(type(response))
print(response.status_code)
print(type(response.text))
print(response.text)
print(response.cookies)
# 二進制，用於解析視頻、圖片等
print(response.content)
print(response.content.decode("utf-8"))
print(response.history)
"""
<class 'requests.models.Response'>
200
<class 'str'>
{
  "args": {
    "age": "22", 
    "name": "luozheng"
  }, 
  ......
}
<RequestsCookieJar[]>
b'{\n  "args": {\n    "age": "22", \n    "name": "luozheng"\n  }, ......}'
{
  "args": {
    "age": "22", 
    "name": "luozheng"
  }, 
  ......
}
[]
"""

Post請求

Post請求的與上面的Get請求類似，唯一不同的是對data參數進行設置時，只能使用字典對參數進行設置，並將參數作爲data賦值傳入。

import requests

# 設置post請求的參數
data = {
    "name":"luozheng",
    "age":22
}
# 添加header
headers = {
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
}
response = requests.post("http://httpbin.org/post", data=data, headers=headers, timeout=5)
print(response.text)
# 二進制，用於解析視頻、圖片等
print(response.content)
print(response.content.decode("utf-8"))

其他用法

解析json
requests裏面集成的json其實就是執行了json.loads()方法，兩者的結果是一樣的。

import requests
import json

response = requests.get("http://httpbin.org/get?name=luozhengfan&age=23")
print(type(response.text))
print(response.json())
print(json.loads(response.text))
print(type(response.json()))
"""
<class 'str'>
{'args': {'age': '23', 'name': 'luozhengfan'}, ...}
{'args': {'age': '23', 'name': 'luozhengfan'}, ...}
<class 'dict'>
"""

獲取二進制數據
獲取響應的文本直接 response.text 。而上面提到了response.content，這樣獲取的數據是二進制數據，同樣的這個方法也可以用於下載圖片以及視頻資源
狀態碼判斷
Requests附帶了一個內置的狀態碼查詢對象，示例如下：

import requests

response= requests.get("http://www.baidu.com")
if response.status_code == requests.codes.ok:
    print("訪問成功")
"""
100: ('continue',),
101: ('switching_protocols',),
102: ('processing',),
103: ('checkpoint',),
122: ('uri_too_long', 'request_uri_too_long'),
200: ('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\o/', '✓'),
201: ('created',),
202: ('accepted',),
203: ('non_authoritative_info', 'non_authoritative_information'),
204: ('no_content',),
205: ('reset_content', 'reset'),
206: ('partial_content', 'partial'),
207: ('multi_status', 'multiple_status', 'multi_stati', 'multiple_stati'),
208: ('already_reported',),
226: ('im_used',),

Redirection.
300: ('multiple_choices',),
301: ('moved_permanently', 'moved', '\o-'),
302: ('found',),
303: ('see_other', 'other'),
304: ('not_modified',),
305: ('use_proxy',),
306: ('switch_proxy',),
307: ('temporary_redirect', 'temporary_moved', 'temporary'),
308: ('permanent_redirect',
'resume_incomplete', 'resume',), # These 2 to be removed in 3.0

Client Error.
400: ('bad_request', 'bad'),
401: ('unauthorized',),
402: ('payment_required', 'payment'),
403: ('forbidden',),
404: ('not_found', '-o-'),
405: ('method_not_allowed', 'not_allowed'),
406: ('not_acceptable',),
407: ('proxy_authentication_required', 'proxy_auth', 'proxy_authentication'),
408: ('request_timeout', 'timeout'),
409: ('conflict',),
410: ('gone',),
411: ('length_required',),
412: ('precondition_failed', 'precondition'),
413: ('request_entity_too_large',),
414: ('request_uri_too_large',),
415: ('unsupported_media_type', 'unsupported_media', 'media_type'),
416: ('requested_range_not_satisfiable', 'requested_range', 'range_not_satisfiable'),
417: ('expectation_failed',),
418: ('im_a_teapot', 'teapot', 'i_am_a_teapot'),
421: ('misdirected_request',),
422: ('unprocessable_entity', 'unprocessable'),
423: ('locked',),
424: ('failed_dependency', 'dependency'),
425: ('unordered_collection', 'unordered'),
426: ('upgrade_required', 'upgrade'),
428: ('precondition_required', 'precondition'),
429: ('too_many_requests', 'too_many'),
431: ('header_fields_too_large', 'fields_too_large'),
444: ('no_response', 'none'),
449: ('retry_with', 'retry'),
450: ('blocked_by_windows_parental_controls', 'parental_controls'),
451: ('unavailable_for_legal_reasons', 'legal_reasons'),
499: ('client_closed_request',),

Server Error.
500: ('internal_server_error', 'server_error', '/o\', '✗'),
501: ('not_implemented',),
502: ('bad_gateway',),
503: ('service_unavailable', 'unavailable'),
504: ('gateway_timeout',),
505: ('http_version_not_supported', 'http_version'),
506: ('variant_also_negotiates',),
507: ('insufficient_storage',),
509: ('bandwidth_limit_exceeded', 'bandwidth'),
510: ('not_extended',),
511: ('network_authentication_required', 'network_auth', 'network_authentication'),
"""

Requests高級用法

主要包括了文件上傳，證書驗證，代理設置，認證設置。

文件上傳
實現方法和其他參數類似，也是構造一個字典然後通過files參數傳遞

import requests

files = {"files":open("wechat.png","rb")}
response = requests.post("http://httpbin.org/post", files=files)
print(response.text)
"""
{
  "args": {}, 
  "data": "", 
  "files": {
    "files": "data:application/octet-stream;base64,iVBORw0KGg......YII="
  }, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "38735", 
    "Content-Type": "multipart/form-data; boundary=a18f2223063180433033132aee71a297", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.22.0"
  }, 
  "json": null, 
  "origin": "117.183.8.254, 117.183.8.254", 
  "url": "https://httpbin.org/post"
}
"""

證書驗證
對於https訪問的證書驗證，不設置可能會報錯 ssl.SSLError 錯誤。

import requests

# 消除警告InsecureRequestWarning
requests.packages.urllib3.disable_warnings()
response = requests.get("https://www.12306.cn",verify=False)
print(response.status_code)

代理設置
代理設置通常爲第一種情況，其他兩種情況請看註釋。

import requests

proxies= {
    "http":"http://127.0.0.1:9999",
    "https":"http://127.0.0.1:8888"
}
response  = requests.get("https://www.baidu.com",proxies=proxies)
print(response.text)
"""
如果代理需要設置賬戶名和密碼,只需要將字典更改爲如下：
proxies = {
"http":"http://user:[email protected]:9999"
}

如果你的代理是通過sokces這種方式則需要pip install "requests[socks]"
proxies= {
"http":"socks5://127.0.0.1:9999",
"https":"sockes5://127.0.0.1:8888"
}
"""

認證設置
如果碰到需要認證的網站可以通過requests.auth模塊實現。

import requests

from requests.auth import HTTPBasicAuth

response = requests.get("http://120.27.34.24:9001/",auth=HTTPBasicAuth("user","123"))
print(response.status_code)
"""
import requests

response = requests.get("http://120.27.34.24:9001/",auth=("user","123"))
print(response.status_code)
"""

requests cookie與會話維持session cookie

requests cookie
主要是對requests請求的cookie進行獲取和設置，請看以下代碼。

## 獲取cookie

import requests

response = requests.get("http://www.baidu.com")
print(response.cookies)

for key, value in response.cookies.items():
    print(key+"="+value)
"""
<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
BDORZ=27315
"""

##設置cookie

import requests

jar = requests.cookies.RequestsCookieJar()
jar.set('tasty_cookie', 'yum', domain='httpbin.org', path='/cookies')
jar.set('gross_cookie', 'blech', domain='httpbin.org', path='/cookies')

url = 'http://httpbin.org/cookies'
r = requests.get(url, cookies=jar)
print(r.text)
"""
{
  "cookies": {
    "gross_cookie": "blech", 
    "tasty_cookie": "yum"
  }
}
"""

會話維持session cookie
由於不同的requests請求之間是獨立的，因此維持回話狀態，只能創建一個session對象，請求都通過這個對象訪問。

以下介紹三種設置會話session的cookie的方法，第一種簡單，第二種可以設置cookie的作用域，第三種每個請求都要設置。

import requests
s = requests.Session()
s.get("http://httpbin.org/cookies/set/number/123456")
response = s.get("http://httpbin.org/cookies")
print(response.text)

add_dict_to_cookiejar對會話進行cookie設置

import requests

s = requests.session()
mycookie = {"tasty_cookie":"yum","gross_cookie":"blech"}

s.cookies = requests.utils.add_dict_to_cookiejar(s.cookies, mycookie)

r = s.get("http://httpbin.org/cookies")
print(r.text)
"""
{
  "cookies": {
    "gross_cookie": "blech", 
    "tasty_cookie": "yum"
  }
}
"""

session.cookies.set 對會話進行cookie設置

import requests

s = requests.session()

s.cookies.set('tasty_cookie', 'yum', domain='httpbin.org', path='/cookies')
s.cookies.set('gross_cookie', 'blech', domain='httpbin.org', path='/cookies')

r = s.get("http://httpbin.org/cookies")
print(r.text)

第三種…
這種每個requests請求都要加上，有點繁瑣。

import requests

s = requests.session()

mycookie = {"tasty_cookie":"yum","gross_cookie":"blech"}

r = s.get("http://127.0.0.1:80",cookies = mycookie)
print(r.text)

個人博客：Loak 正 - 關注人工智能及互聯網的個人博客
文章地址：Python爬蟲（二）— 深入瞭解Requests模塊

Python爬蟲（三）— 深入瞭解Requests模塊

前言

Requests

Get請求

Post請求

其他用法

Requests高級用法

requests cookie與會話維持session cookie

《Python進階》學習筆記

Leetcode 3161. 物塊放置查詢

leetcode 60 排列序列

一個docker容器暴露多個端口

微服務實踐之使用 Visual Studio 2022 調試Dapr 應用程序

wpf附加屬性理解 WPF附加屬性

Python基礎（二）—列表和元組

Python基礎（四）—日期類型

Python爬蟲（五）—正則表達式 re 的深入學習

Python爬蟲（三）— 深入瞭解Requests模塊

Python基礎（十三）—切片、yield、生成器、序列化JSON

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結