reqeusts模塊的學習

使用事前

• pip install reqeusts

發送get, post請求, 獲取響應

• response = requests.get(url) #發送get請求,請求url地址對應的響應
• response = requests.post(url,data={請求體的字典}) #發送post請求

response的方法

• response.text
  • 該方式往往會出現亂碼.出亂碼使用response.encoding=“utf-8”
• response.content.decode()
  • 把響應的二進制字節流轉化爲str類型

獲取網頁源碼的正確打開方式(移動)

• 1.response.content.decode()
• 2.response.content.decode(“gbk”)
• 3.response.text

發送帶header的請求

headers = {	
"User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"}
response = requests.get(url,headers=headers)

使用超時參數

• requests.get(url,headers=headers,timeout=3) #3秒內必須返回響應,否則報錯

retrying模塊的學習

• pip install retrying

import requests
from retrying import retry

'''
專門請求url地址的方法
'''
headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36'}

@retry(stop_max_attempt_number=3) #讓被裝飾的函數反覆執行三次,三次全部報錯纔會報錯,中間有一次正常,則會繼續執行
def _parse_url(url):
    print("*"*100)
    response = requests.get(url,headers=headers,timeout=5)
    return response.content.decode()

def parse_url(url):
    try:
        html_str = _parse_url(url)
    except:
        html_str = None
    return html_str

if __name__ == '__main__':
    url = "http://www.baidu.com"
    print(parse_url(url))