在我們訪問網頁的時候，經常會看到一種技術，當我們點擊了一個選項卡或者其他的鏈接，網頁數據發生了改變，但是網址卻沒有發生任何變化，那麼這種技術就是ajax異步加載數據，當我們點擊一個鏈接的時候，它只是發送了一個數據包過來，然後只是將這個數據包的數據進行了替換，這樣的好處就是我們進行網頁訪問的時候可以更加簡潔高效，也更加的節省資源，但是這對於我們爬取數據的要求就更高了，那麼我們接下來來看一下怎麼爬取異步加載的數據；
我們以豆瓣電影爲例https://movie.douban.com/，先打開網頁，調出Network：
我們把Network調出後，不要選擇All，選擇邊上的XHR，這個篩選的就是異步加載的數據；
我們點擊一下最新熱門電影 -> 熱門，看一下數據會有什麼樣的改變：
發現發送了兩個數據包過來，這兩個數據包究竟是什麼呢？
第一個數據包發送的是上面的選項卡信息，估計用來切換下面的內容信息的；
第二個就是下面改變的數據，也就是我們需要的數據了；
我們來着重分析下第二個數據包的信息：
- 請求地址是：https://movie.douban.com/j/search_subjects
- 請求方式是：GET請求
從中我們可以發現，變量就是tag，其他的量我們可以原樣直接上；
接下來我們以這個選項卡的內容爲例，來看看用代碼怎麼實現：
import requests
parmas = { # 傳參是數據
'type': 'movie',
'tag': '熱門',
'page_limit': 50,
'page_start': 0,
}
header = { # 請求頭
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36'
}
url = 'https://movie.douban.com/j/search_subjects' # 請求地址
result = requests.get(url,headers=header,params=parmas) # 傳參併發起請求
result.encoding = result.raise_for_status() # 設置編碼格式
html = result.text # 獲取數據的文本格式
print(html)
# 輸出結果：
# {"subjects":[{"rate":"8.7","cover_x":1024,"title":"82年生的金智英","url":"https:\/\/movie.douban.com\/subject\/30327842\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570137991.jpg","id":"30327842","cover_y":1459,"is_new":false},{"rate":"9.0","cover_x":2000,"title":"愛爾蘭人","url":"https:\/\/movie.douban.com\/subject\/6981153\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568902055.jpg","id":"6981153","cover_y":2929,"is_new":false},{"rate":"6.2","cover_x":1500,"title":"鬼影特攻：以暴制暴","url":"https:\/\/movie.douban.com\/subject\/30163500\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2574329643.jpg","id":"30163500","cover_y":2222,"is_new":true},{"rate":"6.5","cover_x":730,"title":"寶萊塢雙雄之戰","url":"https:\/\/movie.douban.com\/subject\/34607890\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2569921195.jpg","id":"34607890","cover_y":1000,"is_new":true},{"rate":"6.9","cover_x":1000,"title":"季節與季節之間","url":"https:\/\/movie.douban.com\/subject\/30318240\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570192512.jpg","id":"30318240","cover_y":1500,"is_new":true},{"rate":"8.8","cover_x":1800,"title":"婚姻故事","url":"https:\/\/movie.douban.com\/subject\/27202818\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2571760178.jpg","id":"27202818","cover_y":2666,"is_new":false},{"rate":"8.8","cover_x":4050,"title":"小丑","url":"https:\/\/movie.douban.com\/subject\/27119724\/","playable":false,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567198874.jpg","id":"27119724","cover_y":6000,"is_new":false},{"rate":"7.3","cover_x":1800,"title":"好萊塢往事","url":"https:\/\/movie.douban.com\/subject\/27087724\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2551119672.jpg","id":"27087724","cover_y":2668,"is_new":false},{"rate":"8.4","cover_x":5906,"title":"少年的你","url":"https:\/\/movie.douban.com\/subject\/30166972\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2572166063.jpg","id":"30166972","cover_y":8268,"is_new":false},{"rate":"7.5","cover_x":1385,"title":"紐約的一個雨天","url":"https:\/\/movie.douban.com\/subject\/27089612\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2556824333.jpg","id":"27089612","cover_y":2048,"is_new":false},{"rate":"7.9","cover_x":5906,"title":"我和我的祖國","url":"https:\/\/movie.douban.com\/subject\/32659890\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567998580.jpg","id":"32659890","cover_y":8268,"is_new":false},{"rate":"6.9","cover_x":1286,"title":"中國機長","url":"https:\/\/movie.douban.com\/subject\/30295905\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568258113.jpg","id":"30295905","cover_y":1800,"is_new":false},{"rate":"5.9","cover_x":5906,"title":"催眠·裁決","url":"https:\/\/movie.douban.com\/subject\/26752547\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2569044398.jpg","id":"26752547","cover_y":8268,"is_new":false},{"rate":"6.2","cover_x":3000,"title":"攀登者","url":"https:\/\/movie.douban.com\/subject\/30413052\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568577681.jpg","id":"30413052","cover_y":4200,"is_new":false},{"rate":"8.2","cover_x":1142,"title":"行騙天下JP：浪漫篇","url":"https:\/\/movie.douban.com\/subject\/30241102\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2549891899.jpg","id":"30241102","cover_y":1615,"is_new":false},{"rate":"8.7","cover_x":1500,"title":"寄生蟲","url":"https:\/\/movie.douban.com\/subject\/27010768\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2561439800.jpg","id":"27010768","cover_y":2138,"is_new":false},{"rate":"6.7","cover_x":3900,"title":"準備好了沒","url":"https:\/\/movie.douban.com\/subject\/27594938\/","playable":false,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2561370606.jpg","id":"27594938","cover_y":5813,"is_new":false},{"rate":"7.1","cover_x":2187,"title":"雙子殺手","url":"https:\/\/movie.douban.com\/subject\/3097572\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570906505.jpg","id":"3097572","cover_y":3000,"is_new":false},{"rate":"8.0","cover_x":5852,"title":"地久天長","url":"https:\/\/movie.douban.com\/subject\/26715636\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2550208359.jpg","id":"26715636","cover_y":8192,"is_new":false},{"rate":"7.5","cover_x":1125,"title":"第一滴血5：最後的血","url":"https:\/\/movie.douban.com\/subject\/10546436\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2564884269.jpg","id":"10546436","cover_y":1500,"is_new":false},{"rate":"8.6","cover_x":1500,"title":"克勞斯：聖誕節的祕密","url":"https:\/\/movie.douban.com\/subject\/26858510\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570825762.jpg","id":"26858510","cover_y":2222,"is_new":false},{"rate":"6.8","cover_x":2764,"title":"睡夢醫生","url":"https:\/\/movie.douban.com\/subject\/25932065\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2571039507.jpg","id":"25932065","cover_y":4096,"is_new":false},{"rate":"7.4","cover_x":2143,"title":"雪人奇緣","url":"https:\/\/movie.douban.com\/subject\/26929010\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567925521.jpg","id":"26929010","cover_y":3000,"is_new":false},{"rate":"8.5","cover_x":5594,"title":"哪吒之魔童降世","url":"https:\/\/movie.douban.com\/subject\/26794435\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2563780504.jpg","id":"26794435","cover_y":8268,"is_new":false},{"rate":"8.1","cover_x":1500,"title":"我失去了身體","url":"https:\/\/movie.douban.com\/subject\/33415943\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2553909340.jpg","id":"33415943","cover_y":2000,"is_new":false},{"rate":"7.4","cover_x":1080,"title":"別告訴她","url":"https:\/\/movie.douban.com\/subject\/30390144\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2574723117.jpg","id":"30390144","cover_y":1800,"is_new":false},{"rate":"6.4","cover_x":960,"title":"瘋狂的外星人","url":"https:\/\/movie.douban.com\/subject\/25986662\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2541901817.jpg","id":"25986662","cover_y":1359,"is_new":false},{"rate":"7.9","cover_x":1786,"title":"流浪地球","url":"https:\/\/movie.douban.com\/subject\/26266893\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2545472803.jpg","id":"26266893","cover_y":2500,"is_new":false},{"rate":"6.4","cover_x":8100,"title":"舞女大盜","url":"https:\/\/movie.douban.com\/subject\/30294313\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2565492557.jpg","id":"30294313","cover_y":12000,"is_new":false},{"rate":"6.4","cover_x":4050,"title":"小丑回魂2","url":"https:\/\/movie.douban.com\/subject\/27133569\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2562980418.jpg","id":"27133569","cover_y":6000,"is_new":false},{"rate":"8.2","cover_x":2000,"title":"羅小黑戰記","url":"https:\/\/movie.douban.com\/subject\/26709258\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568288336.jpg","id":"26709258","cover_y":3208,"is_new":false},{"rate":"7.7","cover_x":1500,"title":"惡人傳","url":"https:\/\/movie.douban.com\/subject\/30211551\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2555084871.jpg","id":"30211551","cover_y":2145,"is_new":false},{"rate":"8.6","cover_x":2740,"title":"痛苦與榮耀","url":"https:\/\/movie.douban.com\/subject\/30198539\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2554080075.jpg","id":"30198539","cover_y":3732,"is_new":false},{"rate":"7.1","cover_x":4371,"title":"送我上青雲","url":"https:\/\/movie.douban.com\/subject\/27180759\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2565751382.jpg","id":"27180759","cover_y":6215,"is_new":false},{"rate":"7.7","cover_x":3543,"title":"安娜","url":"https:\/\/movie.douban.com\/subject\/27166976\/","playable":false,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2553320254.jpg","id":"27166976","cover_y":4724,"is_new":false},{"rate":"6.9","cover_x":7142,"title":"飛馳人生","url":"https:\/\/movie.douban.com\/subject\/30163509\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2542973862.jpg","id":"30163509","cover_y":10000,"is_new":false},{"rate":"7.4","cover_x":2500,"title":"獅子王","url":"https:\/\/movie.douban.com\/subject\/26884354\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2559742751.jpg","id":"26884354","cover_y":3571,"is_new":false},{"rate":"5.6","cover_x":667,"title":"追兇十九年","url":"https:\/\/movie.douban.com\/subject\/26974334\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2572147897.jpg","id":"26974334","cover_y":1000,"is_new":false},{"rate":"6.4","cover_x":3000,"title":"犯罪現場","url":"https:\/\/movie.douban.com\/subject\/26235346\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570879785.jpg","id":"26235346","cover_y":4200,"is_new":false},{"rate":"8.1","cover_x":1500,"title":"續命之徒：絕命毒師電影","url":"https:\/\/movie.douban.com\/subject\/30372377\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2569548689.jpg","id":"30372377","cover_y":2222,"is_new":false},{"rate":"8.0","cover_x":2000,"title":"漫長的告別","url":"https:\/\/movie.douban.com\/subject\/30327897\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2546226109.jpg","id":"30327897","cover_y":2831,"is_new":false},{"rate":"7.3","cover_x":4050,"title":"伯納黛特你去了哪","url":"https:\/\/movie.douban.com\/subject\/21354767\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2555983860.jpg","id":"21354767","cover_y":6000,"is_new":false},{"rate":"8.2","cover_x":2025,"title":"我在雨中等你","url":"https:\/\/movie.douban.com\/subject\/3882427\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2557226052.jpg","id":"3882427","cover_y":3000,"is_new":false},{"rate":"7.5","cover_x":5000,"title":"阿麗塔：戰鬥天使","url":"https:\/\/movie.douban.com\/subject\/1652592\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2544987866.jpg","id":"1652592","cover_y":7413,"is_new":false},{"rate":"7.8","cover_x":1250,"title":"柳烈的音樂專輯","url":"https:\/\/movie.douban.com\/subject\/30225296\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2561996109.jpg","id":"30225296","cover_y":1782,"is_new":false},{"rate":"8.0","cover_x":750,"title":"最初的夢想","url":"https:\/\/movie.douban.com\/subject\/34787747\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570421055.jpg","id":"34787747","cover_y":1000,"is_new":false},{"rate":"8.7","cover_x":1968,"title":"玩具總動員4","url":"https:\/\/movie.douban.com\/subject\/6850547\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2557284230.jpg","id":"6850547","cover_y":2780,"is_new":false},{"rate":"7.9","cover_x":5315,"title":"白蛇：緣起","url":"https:\/\/movie.douban.com\/subject\/30331149\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2544313786.jpg","id":"30331149","cover_y":7441,"is_new":false},{"rate":"5.1","cover_x":1500,"title":"小小的願望","url":"https:\/\/movie.douban.com\/subject\/30235440\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567277578.jpg","id":"30235440","cover_y":2100,"is_new":false},{"rate":"6.5","cover_x":2835,"title":"古田軍號","url":"https:\/\/movie.douban.com\/subject\/30261964\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2564173381.jpg","id":"30261964","cover_y":4201,"is_new":false}]}
我們可以看見信息已經獲取到了，但是現在我們是以文本格式進行獲取的，數據的提取會很麻煩，在上面可以看到發送的數據包就是以鍵值對的形式發送的，那我們可以直接獲取json格式的數據，後面可以直接用鍵值對的方式進行提取：
