本文用於學習交流使用，如有侵權，聯繫刪除
1 爬取需求
1.1 七麥網簡介
七麥網（https://www.qimai.cn/），該平臺支持提供iOS、Android應用市場、微信、小程序等數據查詢，是同時打通App數據、微信公衆號數據、小程序數據的數據分析平臺。
1.2 爬取需求
根據關鍵詞文件，在檢索欄內輸入檢索關鍵詞。以滴滴出行爲例，獲得如下界面，在這裏點擊第一個獲得相應的appID（554499054），在此基礎上獲得更詳細的信息。
2 url構造
2.1 analysis加密
相信來看的各位都是因爲七麥網的analysis的加密，參考了一系列的文章解決了這個問題，由於侵權問題，就不放破解的js代碼了。下面將參考用的博客放給大家
- https://mp.weixin.qq.com/s/1E_ONUnMwTFozd9-dB70Yw
- https://blog.csdn.net/wg5foc08/article/details/102831489
- https://blog.csdn.net/Herishwater/article/details/90733267
由於此文做的是研究某一個APP的ID，之前人做的JS是研究IOS的排行信息，所以在JS傳參方向有細節不同。可以通過申請網址來看（https://www.qimai.cn/search/index/country/cn/search/%E6%BB%B4%E6%BB%B4%E5%87%BA%E8%A1%8C），這裏傳參的param不一樣，除此以外，加密方式一樣。
2.2 構造鏈接代碼
我的代碼實在jupyter notebook裏面寫的，下面是圖片，把代碼集中貼在後面。
import requests
import execjs
import urllib
import os
import json
import random
import re
import csv
# 讀入整理的app名稱
search_list = []
with open("/Users/didi/Downloads/七麥網爬蟲/key_word.txt", "r") as f:
for line in f.readlines():
line = line.strip('\n')
search_list.append(line)
# 讀入解密js
with open('/Users/didi/Downloads/七麥網爬蟲/qimai_app.js', 'r', encoding='utf-8') as f:
get_analy_js = f.read()
jsData = execjs.compile(get_analy_js)
# 準備UA
user_agent = [
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko",
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
"Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)",
"Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999",
"Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25",
]
# 通過訪問七麥得到時間戳
def get_synct():
headers = {
"User-Agent": random.choice(user_agent),
'Origin': 'https://www.qimai.cn',
'Referer': 'https://www.qimai.cn/rank',
"Accept": "application/json,text/plain,*/*",
}
resp = requests.get('https://www.qimai.cn/rank', headers=headers, verify=False)
t = resp.cookies.get_dict()
synct = t.get('synct')
return synct
# 獲得加密後的url
def get_analysis_url(synct,app_name):
params = {
'country':'cn',
'search':app_name
}
analysis = jsData.call('getAnalysis',synct,params)
base_url = "https://api.qimai.cn/search/index?analysis="
params_str = "&country=cn&search=" + str(app_name)
url = str(base_url) + str(analysis) + str(params_str)
return url
def get_appID(url):
headers = {
"User-Agent": random.choice(user_agent),
'Origin': 'https://www.qimai.cn'
}
html = requests.get(url, headers=headers)
response = html.text
try:
pattern = r'"appId":"(.*?)","appName"'
appID = re.search(pattern, response, re.I)[1]
except:
appID = None
return appID
def write_csv(file,appInfo):
with open(file, 'a', encoding='utf-8',newline='') as csv_file:
writer = csv.weiter(csv_file)
writer.writerow(appInfo)
def main(app_name):
synct = get_synct()
url = get_analysis_url(synct,app_name)
appID = get_appID(url)
print(appID)
if __name__ == '__main__':
for i in range(len(search_list))
app_name = "滴滴出行"
main(app_name)
3 其他反爬
破解analysis後還有其他的反爬，下面是一張測試結果，這是對於訪問頻率的封禁，可以看出七麥網的反爬是非常厲害的。