在我们访问网页的时候，经常会看到一种技术，当我们点击了一个选项卡或者其他的链接，网页数据发生了改变，但是网址却没有发生任何变化，那么这种技术就是ajax异步加载数据，当我们点击一个链接的时候，它只是发送了一个数据包过来，然后只是将这个数据包的数据进行了替换，这样的好处就是我们进行网页访问的时候可以更加简洁高效，也更加的节省资源，但是这对于我们爬取数据的要求就更高了，那么我们接下来来看一下怎么爬取异步加载的数据；
我们以豆瓣电影为例https://movie.douban.com/，先打开网页，调出Network：
我们把Network调出后，不要选择All，选择边上的XHR，这个筛选的就是异步加载的数据；
我们点击一下最新热门电影 -> 热门，看一下数据会有什么样的改变：
发现发送了两个数据包过来，这两个数据包究竟是什么呢？
第一个数据包发送的是上面的选项卡信息，估计用来切换下面的内容信息的；
第二个就是下面改变的数据，也就是我们需要的数据了；
我们来着重分析下第二个数据包的信息：
- 请求地址是：https://movie.douban.com/j/search_subjects
- 请求方式是：GET请求
从中我们可以发现，变量就是tag，其他的量我们可以原样直接上；
接下来我们以这个选项卡的内容为例，来看看用代码怎么实现：
import requests
parmas = { # 传参是数据
'type': 'movie',
'tag': '热门',
'page_limit': 50,
'page_start': 0,
}
header = { # 请求头
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36'
}
url = 'https://movie.douban.com/j/search_subjects' # 请求地址
result = requests.get(url,headers=header,params=parmas) # 传参并发起请求
result.encoding = result.raise_for_status() # 设置编码格式
html = result.text # 获取数据的文本格式
print(html)
# 输出结果：
# {"subjects":[{"rate":"8.7","cover_x":1024,"title":"82年生的金智英","url":"https:\/\/movie.douban.com\/subject\/30327842\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570137991.jpg","id":"30327842","cover_y":1459,"is_new":false},{"rate":"9.0","cover_x":2000,"title":"爱尔兰人","url":"https:\/\/movie.douban.com\/subject\/6981153\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568902055.jpg","id":"6981153","cover_y":2929,"is_new":false},{"rate":"6.2","cover_x":1500,"title":"鬼影特攻：以暴制暴","url":"https:\/\/movie.douban.com\/subject\/30163500\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2574329643.jpg","id":"30163500","cover_y":2222,"is_new":true},{"rate":"6.5","cover_x":730,"title":"宝莱坞双雄之战","url":"https:\/\/movie.douban.com\/subject\/34607890\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2569921195.jpg","id":"34607890","cover_y":1000,"is_new":true},{"rate":"6.9","cover_x":1000,"title":"季节与季节之间","url":"https:\/\/movie.douban.com\/subject\/30318240\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570192512.jpg","id":"30318240","cover_y":1500,"is_new":true},{"rate":"8.8","cover_x":1800,"title":"婚姻故事","url":"https:\/\/movie.douban.com\/subject\/27202818\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2571760178.jpg","id":"27202818","cover_y":2666,"is_new":false},{"rate":"8.8","cover_x":4050,"title":"小丑","url":"https:\/\/movie.douban.com\/subject\/27119724\/","playable":false,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567198874.jpg","id":"27119724","cover_y":6000,"is_new":false},{"rate":"7.3","cover_x":1800,"title":"好莱坞往事","url":"https:\/\/movie.douban.com\/subject\/27087724\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2551119672.jpg","id":"27087724","cover_y":2668,"is_new":false},{"rate":"8.4","cover_x":5906,"title":"少年的你","url":"https:\/\/movie.douban.com\/subject\/30166972\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2572166063.jpg","id":"30166972","cover_y":8268,"is_new":false},{"rate":"7.5","cover_x":1385,"title":"纽约的一个雨天","url":"https:\/\/movie.douban.com\/subject\/27089612\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2556824333.jpg","id":"27089612","cover_y":2048,"is_new":false},{"rate":"7.9","cover_x":5906,"title":"我和我的祖国","url":"https:\/\/movie.douban.com\/subject\/32659890\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567998580.jpg","id":"32659890","cover_y":8268,"is_new":false},{"rate":"6.9","cover_x":1286,"title":"中国机长","url":"https:\/\/movie.douban.com\/subject\/30295905\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568258113.jpg","id":"30295905","cover_y":1800,"is_new":false},{"rate":"5.9","cover_x":5906,"title":"催眠·裁决","url":"https:\/\/movie.douban.com\/subject\/26752547\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2569044398.jpg","id":"26752547","cover_y":8268,"is_new":false},{"rate":"6.2","cover_x":3000,"title":"攀登者","url":"https:\/\/movie.douban.com\/subject\/30413052\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568577681.jpg","id":"30413052","cover_y":4200,"is_new":false},{"rate":"8.2","cover_x":1142,"title":"行骗天下JP：浪漫篇","url":"https:\/\/movie.douban.com\/subject\/30241102\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2549891899.jpg","id":"30241102","cover_y":1615,"is_new":false},{"rate":"8.7","cover_x":1500,"title":"寄生虫","url":"https:\/\/movie.douban.com\/subject\/27010768\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2561439800.jpg","id":"27010768","cover_y":2138,"is_new":false},{"rate":"6.7","cover_x":3900,"title":"准备好了没","url":"https:\/\/movie.douban.com\/subject\/27594938\/","playable":false,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2561370606.jpg","id":"27594938","cover_y":5813,"is_new":false},{"rate":"7.1","cover_x":2187,"title":"双子杀手","url":"https:\/\/movie.douban.com\/subject\/3097572\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570906505.jpg","id":"3097572","cover_y":3000,"is_new":false},{"rate":"8.0","cover_x":5852,"title":"地久天长","url":"https:\/\/movie.douban.com\/subject\/26715636\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2550208359.jpg","id":"26715636","cover_y":8192,"is_new":false},{"rate":"7.5","cover_x":1125,"title":"第一滴血5：最后的血","url":"https:\/\/movie.douban.com\/subject\/10546436\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2564884269.jpg","id":"10546436","cover_y":1500,"is_new":false},{"rate":"8.6","cover_x":1500,"title":"克劳斯：圣诞节的秘密","url":"https:\/\/movie.douban.com\/subject\/26858510\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570825762.jpg","id":"26858510","cover_y":2222,"is_new":false},{"rate":"6.8","cover_x":2764,"title":"睡梦医生","url":"https:\/\/movie.douban.com\/subject\/25932065\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2571039507.jpg","id":"25932065","cover_y":4096,"is_new":false},{"rate":"7.4","cover_x":2143,"title":"雪人奇缘","url":"https:\/\/movie.douban.com\/subject\/26929010\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567925521.jpg","id":"26929010","cover_y":3000,"is_new":false},{"rate":"8.5","cover_x":5594,"title":"哪吒之魔童降世","url":"https:\/\/movie.douban.com\/subject\/26794435\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2563780504.jpg","id":"26794435","cover_y":8268,"is_new":false},{"rate":"8.1","cover_x":1500,"title":"我失去了身体","url":"https:\/\/movie.douban.com\/subject\/33415943\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2553909340.jpg","id":"33415943","cover_y":2000,"is_new":false},{"rate":"7.4","cover_x":1080,"title":"别告诉她","url":"https:\/\/movie.douban.com\/subject\/30390144\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2574723117.jpg","id":"30390144","cover_y":1800,"is_new":false},{"rate":"6.4","cover_x":960,"title":"疯狂的外星人","url":"https:\/\/movie.douban.com\/subject\/25986662\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2541901817.jpg","id":"25986662","cover_y":1359,"is_new":false},{"rate":"7.9","cover_x":1786,"title":"流浪地球","url":"https:\/\/movie.douban.com\/subject\/26266893\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2545472803.jpg","id":"26266893","cover_y":2500,"is_new":false},{"rate":"6.4","cover_x":8100,"title":"舞女大盗","url":"https:\/\/movie.douban.com\/subject\/30294313\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2565492557.jpg","id":"30294313","cover_y":12000,"is_new":false},{"rate":"6.4","cover_x":4050,"title":"小丑回魂2","url":"https:\/\/movie.douban.com\/subject\/27133569\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2562980418.jpg","id":"27133569","cover_y":6000,"is_new":false},{"rate":"8.2","cover_x":2000,"title":"罗小黑战记","url":"https:\/\/movie.douban.com\/subject\/26709258\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2568288336.jpg","id":"26709258","cover_y":3208,"is_new":false},{"rate":"7.7","cover_x":1500,"title":"恶人传","url":"https:\/\/movie.douban.com\/subject\/30211551\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2555084871.jpg","id":"30211551","cover_y":2145,"is_new":false},{"rate":"8.6","cover_x":2740,"title":"痛苦与荣耀","url":"https:\/\/movie.douban.com\/subject\/30198539\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2554080075.jpg","id":"30198539","cover_y":3732,"is_new":false},{"rate":"7.1","cover_x":4371,"title":"送我上青云","url":"https:\/\/movie.douban.com\/subject\/27180759\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2565751382.jpg","id":"27180759","cover_y":6215,"is_new":false},{"rate":"7.7","cover_x":3543,"title":"安娜","url":"https:\/\/movie.douban.com\/subject\/27166976\/","playable":false,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2553320254.jpg","id":"27166976","cover_y":4724,"is_new":false},{"rate":"6.9","cover_x":7142,"title":"飞驰人生","url":"https:\/\/movie.douban.com\/subject\/30163509\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2542973862.jpg","id":"30163509","cover_y":10000,"is_new":false},{"rate":"7.4","cover_x":2500,"title":"狮子王","url":"https:\/\/movie.douban.com\/subject\/26884354\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2559742751.jpg","id":"26884354","cover_y":3571,"is_new":false},{"rate":"5.6","cover_x":667,"title":"追凶十九年","url":"https:\/\/movie.douban.com\/subject\/26974334\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2572147897.jpg","id":"26974334","cover_y":1000,"is_new":false},{"rate":"6.4","cover_x":3000,"title":"犯罪现场","url":"https:\/\/movie.douban.com\/subject\/26235346\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570879785.jpg","id":"26235346","cover_y":4200,"is_new":false},{"rate":"8.1","cover_x":1500,"title":"续命之徒：绝命毒师电影","url":"https:\/\/movie.douban.com\/subject\/30372377\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2569548689.jpg","id":"30372377","cover_y":2222,"is_new":false},{"rate":"8.0","cover_x":2000,"title":"漫长的告别","url":"https:\/\/movie.douban.com\/subject\/30327897\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2546226109.jpg","id":"30327897","cover_y":2831,"is_new":false},{"rate":"7.3","cover_x":4050,"title":"伯纳黛特你去了哪","url":"https:\/\/movie.douban.com\/subject\/21354767\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2555983860.jpg","id":"21354767","cover_y":6000,"is_new":false},{"rate":"8.2","cover_x":2025,"title":"我在雨中等你","url":"https:\/\/movie.douban.com\/subject\/3882427\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2557226052.jpg","id":"3882427","cover_y":3000,"is_new":false},{"rate":"7.5","cover_x":5000,"title":"阿丽塔：战斗天使","url":"https:\/\/movie.douban.com\/subject\/1652592\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2544987866.jpg","id":"1652592","cover_y":7413,"is_new":false},{"rate":"7.8","cover_x":1250,"title":"柳烈的音乐专辑","url":"https:\/\/movie.douban.com\/subject\/30225296\/","playable":false,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2561996109.jpg","id":"30225296","cover_y":1782,"is_new":false},{"rate":"8.0","cover_x":750,"title":"最初的梦想","url":"https:\/\/movie.douban.com\/subject\/34787747\/","playable":false,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2570421055.jpg","id":"34787747","cover_y":1000,"is_new":false},{"rate":"8.7","cover_x":1968,"title":"玩具总动员4","url":"https:\/\/movie.douban.com\/subject\/6850547\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2557284230.jpg","id":"6850547","cover_y":2780,"is_new":false},{"rate":"7.9","cover_x":5315,"title":"白蛇：缘起","url":"https:\/\/movie.douban.com\/subject\/30331149\/","playable":true,"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2544313786.jpg","id":"30331149","cover_y":7441,"is_new":false},{"rate":"5.1","cover_x":1500,"title":"小小的愿望","url":"https:\/\/movie.douban.com\/subject\/30235440\/","playable":true,"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567277578.jpg","id":"30235440","cover_y":2100,"is_new":false},{"rate":"6.5","cover_x":2835,"title":"古田军号","url":"https:\/\/movie.douban.com\/subject\/30261964\/","playable":true,"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2564173381.jpg","id":"30261964","cover_y":4201,"is_new":false}]}
我们可以看见信息已经获取到了，但是现在我们是以文本格式进行获取的，数据的提取会很麻烦，在上面可以看到发送的数据包就是以键值对的形式发送的，那我们可以直接获取json格式的数据，后面可以直接用键值对的方式进行提取：
