一、簡單說明
熱點新聞等資訊信息繞不過今日頭條，今天講解下如何通過拿到_signature參數來獲取頭條新聞。
二、發現問題
今日頭條的熱點數據是ajax動態刷新得到的，通過篩選測試有數據的url爲https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature=Yu0DpgAAOZbIwkMmDCYgkGLtA7，其中_signature是需要得到的關鍵字參數，在js中搜尋中得到生成文件home_e14d2a0.js,生成流程
調試發現"refresh" === t ? 0 : r.params.max_behot_time_tmp可以直接賦值爲0；因爲生成_signature的js算法比較複雜，網上有很多這方面的博客，大多失效。我們直接通過selenium來獲取數據。
三、獲取數據
至於selenium的安裝和使用在這裏就不作過多描述，直接上代碼吧！
'''
今日頭條
js破解_signetrue
'''
from selenium import webdriver
import requests
import time
import json
from utils.get_proxy import func_proxy
# 進入瀏覽器設置
options = webdriver.ChromeOptions()
# 設置中文
options.add_argument('lang=zh_CN.UTF-8')
options.set_headless()
options.add_argument(
'user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"')
# 設置代理
service_args = ['--proxy={}'.format(func_proxy()['http']), '--proxy-type=socks5']
brower = webdriver.Chrome(chrome_options=options, service_args=service_args)
brower.get('https://www.toutiao.com/ch/news_hot/')
sinature = brower.execute_script('return TAC.sign(0)')
print(sinature)
"""獲取cookie"""
cookie = brower.get_cookies()
cookie = [item['name'] + "=" + item['value'] for item in cookie]
cookiestr = '; '.join(item for item in cookie)
brower.quit()
print(cookiestr)
header1 = {
'Host': 'www.toutiao.com',
'User-Agent': '"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"',
'Referer': 'https://www.toutiao.com/ch/news_hot/',
"Cookie": cookiestr
}
url = 'https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature={}'.format(
sinature)
print(url)
html = requests.get(url, headers=header1, proxies=func_proxy(), verify=False)
print(html.content.decode('unicode_escape'))
所獲取到的數據
{'http': 'http://124.230.0.20:42546'}
ZuU44QAAPZ4TmjxZv6zL.WblOP
UM_distinctid=1656b863a92f8-0e645b22313a8b-71272415-75300-1656b863a93f3; tt_webid=6593235575644423694; __tasessionId=lkyrlacv11535107283377; csrftoken=c50e03d22a62281c7ce646395bf32f17; uuid="w:f37f35cde0e24093943a530d5557d608"; CNZZDATA1259612802=222376684-1535103528-%7C1535103528
https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature=ZuU44QAAPZ4TmjxZv6zL.WblOP
C:\Anaconda\lib\site-packages\requests\packages\urllib3\connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
{"has_more": false, "message": "success", "data": [{"chinese_tag": "其它", "is_feed_ad": false, "tag_url": "search/?keyword=%E5%85%B6%E5%AE%83", "title": "颱風災害致山東13市500餘萬人受災 濰坊災情最重", "single_mode": true, "middle_mode": true, "tag": "news", "label": ["氣象災害", "颱風"], "behot_time": 1535107286, "source_url": "/api/pc/subject/6592925627836268808/", "source": "專題", "more_mode": false, "article_genre": "article", "image_url": "//p3.pstatp.com/list/190x124/aad60018f524a156bffb", "has_gallery": false, "group_source": 4, "item_id": "6592925627836268808", "comments_count": 34, "group_id": "6592925627836268808", "middle_image": {"url": "http://p3.pstatp.com/list/aad60018f524a156bffb", "width": 640, "url_list": [{"url": "http://p3.pstatp.com/list/aad60018f524a156bffb"}, {"url": "http://pb9.pstatp.com/list/aad60018f524a156bffb"}, {"url": "http://pb1.pstatp.com/list/aad60018f524a156bffb"}], "uri": "list/aad60018f524a156bffb", "height": 300}}, {"chinese_tag": "社會", "media_avatar_url": "//p3.pstatp.com/large/ca400072481685ad43b", "is_feed_ad": false, "tag_url": "news_society", "title": "高鐵“霸座”男被處治安罰款200元 記入鐵路徵信體系", "single_mode": false, "middle_mode": false, "abstract": "人民網濟南8月24日電近期在網上“火”了的高鐵“霸座”男孫某被處治安罰款200元,並在一定期限內被限制購票乘坐火車。", "tag": "news_society", "label": ["社會"], "behot_time": 1535107285, "source_url": "/group/6593171454974493192/", "source": "人民網", "more_mode": false, "article_genre": "article", "comments_count": 35450, "group_source": 2, "item_id": "6593171454974493192", "has_gallery": false, "group_id": "6593171454974493192", "media_url": "/c/user/50502346173/"}, {"chinese_tag": "財經", "media_avatar_url": "//p1.pstatp.com/large/4d00054b126ceaf920", "is_feed_ad": false, "tag_url": "news_finance", "title": "外交部談中美經貿磋商：沒覺得有必要時不時喊兩嗓子，替自己壯膽", "single_mode": false, "middle_mode": false, "abstract": "問：我們注意到，美國政府高層近日在談到中美經貿磋商時常常要強調，美國的經濟狀況很好，中國的情況不好。", "tag": "news_finance", "label": ["時政外交", "財經"], "behot_time": 1535107285, "source_url": "/group/6593194458169213448/", "source": "環球網", "more_mode": false, "article_genre": "article", "comments_count": 128, "group_source": 2, "item_id": "6593194458169213448", "has_gallery": false, "group_id": "6593194458169213448", "media_url": "/c/user/5954781019/"}, {"single_mode": true, "abstract": "京東物流西南區域分公司快運部城配運輸管理部機構負責人任兵、京東物流華中區域分公司大件物流部安裝運營部機構負責人黃川等多人已被公安機關刑事拘留。", "middle_mode": true, "more_mode": true, "tag": "news_finance", "label": ["電子商務", "京東", "刑法", "法律", "反腐倡廉"], "comments_count": 385, "tag_url": "news_finance", "title": "京東發佈反腐敗公告：多位員工被公安機關刑事拘留", "chinese_tag": "財經", "source": "雷帝觸網", "group_source": 2, "has_gallery": false, "media_url": "/c/user/6493820122/", "media_avatar_url": "//p6.pstatp.com/large/2c6800053c0c1b76bdfb", "image_list": [{"url": "//p99.pstatp.com/list/pgc-image/153508311899482775b997b"}, {"url": "//p99.pstatp.com/list/pgc-image/153508312799647d3847a69"}, {"url": "//p99.pstatp.com/list/pgc-image/1535083138646835745cbd7"}], "source_url": "/group/6593132015963341315/", "article_genre": "article", "item_id": "6593132015963341315", "is_feed_ad": false, "behot_time": 1535107284, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153508311899482775b997b", "group_id": "6593132015963341315", "middle_image": "http://p99.pstatp.com/list/pgc-image/153508311899482775b997b"}, {"single_mode": true, "abstract": "8月24日，網絡上流傳的一組圖片顯示，在上海江南造船廠製造的我國國產萬噸055型驅逐艦首艦已經開始試航。", "middle_mode": true, "more_mode": true, "tag": "news_military", "label": ["海軍", "驅逐艦", "武器", "江南造船廠", "中國軍情"], "comments_count": 2570, "tag_url": "news_military", "title": "055型驅逐艦首艦今日首次試航", "chinese_tag": "軍事", "source": "觀察者網", "group_source": 2, "has_gallery": false, "media_url": "/c/user/2928392215/", "media_avatar_url": "//p3.pstatp.com/large/d2a0015e51ed3a11c15", "image_list": [{"url": "//p9.pstatp.com/list/pgc-image/15350803549188193355432"}, {"url": "//p99.pstatp.com/list/pgc-image/1535080355549a3b71e372e"}, {"url": "//p99.pstatp.com/list/pgc-image/1535080355238e02c474b41"}], "source_url": "/group/6593119924883816973/", "article_genre": "article", "item_id": "6593119924883816973", "is_feed_ad": false, "behot_time": 1535107284, "image_url": "//p9.pstatp.com/list/190x124/pgc-image/15350803549188193355432", "group_id": "6593119924883816973", "middle_image": "http://p9.pstatp.com/list/pgc-image/15350803549188193355432"}, {"single_mode": true, "abstract": "近日，有網友在網上發佈了幾張王祖賢昔日舊照。其中有一張就是王祖賢和她爸爸媽媽，以及金牌經紀人陳自強的四人合影，她站在中間摟着爸爸和經紀人的胳膊，備受寵愛。", "middle_mode": true, "more_mode": true, "tag": "news_entertainment", "label": ["王祖賢", "陳自強", "籃球", "加拿大", "張曼玉"], "comments_count": 43, "tag_url": "news_entertainment", "title": "王祖賢全家照曝光，父母都是大高個，光看穿着就知道是有錢人家！", "chinese_tag": "娛樂", "source": "貓眼娛樂", "group_source": 2, "has_gallery": false, "media_url": "/c/user/64781639962/", "media_avatar_url": "//p3.pstatp.com/large/2c6b001dd55cf954a3f6", "image_list": [{"url": "//p99.pstatp.com/list/pgc-image/153507480604336e5031b95"}, {"url": "//p99.pstatp.com/list/pgc-image/1535074806030f559e9695a"}, {"url": "//p3.pstatp.com/list/pgc-image/1535074806129dbbd6d84d2"}], "source_url": "/group/6593099140371055112/", "article_genre": "article", "item_id": "6593099140371055112", "is_feed_ad": false, "behot_time": 1535107283, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153507480604336e5031b95", "group_id": "6593099140371055112", "middle_image": "http://p99.pstatp.com/list/pgc-image/153507480604336e5031b95"}], "next": {"max_behot_time": 1535107283}}
當再次將cookie添加發送請求可以刷新數據
sinature = "ZuU44QAAPZ4TmjxZv6zL.WblOP"
cookiestr = 'UM_distinctid=1656b863a92f8-0e645b22313a8b-71272415-75300-1656b863a93f3; tt_webid=6593235575644423694; __tasessionId=lkyrlacv11535107283377; csrftoken=c50e03d22a62281c7ce646395bf32f17; uuid="w:f37f35cde0e24093943a530d5557d608"; CNZZDATA1259612802=222376684-1535103528-%7C1535103528'
header1 = {
'Host': 'www.toutiao.com',
'User-Agent': '"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"',
'Referer': 'https://www.toutiao.com/ch/news_hot/',
"Cookie": cookiestr
}
url = 'https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature={}'.format(
sinature)
print(url)
html = requests.get(url, headers=header1, proxies=func_proxy(), verify=False)
print(html.content.decode('unicode_escape'))
刷新的數據爲
{'http': 'http://60.179.248.118:29128'}
https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature=ZuU44QAAPZ4TmjxZv6zL.WblOP
C:\Anaconda\lib\site-packages\requests\packages\urllib3\connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
{"has_more": false, "message": "success", "data": [{"single_mode": true, "abstract": "就在剛纔，一直對華很不友好的澳大利亞總理特恩布爾終於下臺了，可接替他的人卻同樣不令人放心…昨天，耿直哥曾撰文給大家講述了澳大利亞政壇這兩天發生了一場大地震：該國的公司的股權。", "middle_mode": true, "more_mode": false, "tag": "news_world", "label": ["中澳關係", "政治", "華爲", "時政外交", "國際"], "comments_count": 675, "tag_url": "news_world", "title": "澳洲總理突然下臺，新上臺的昨天才捅了我們一刀！", "chinese_tag": "國際", "source": "環球時報", "group_source": 2, "has_gallery": false, "media_url": "/c/user/5787290902/", "media_avatar_url": "//p2.pstatp.com/large/9935/2486304457", "source_url": "/group/6593151862864609799/", "article_genre": "article", "item_id": "6593151862864609799", "is_feed_ad": false, "behot_time": 1535107391, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/15350876575153d77867b24", "group_id": "6593151862864609799", "middle_image": "http://p99.pstatp.com/list/pgc-image/15350876575153d77867b24"}, {"single_mode": true, "abstract": "日前，外交部前發言人章啓月出任中國駐希臘大使。得知將擔任新聞發言人，章啓月覺得自己並不太適合做這項工作，丈夫劉結一鼓勵她說：“你還是試一試，應該去迎接點挑戰嘛”。", "middle_mode": true, "more_mode": false, "tag": "news_politics", "label": ["章啓月", "劉結一", "外交政策", "印度尼西亞", "時政"], "comments_count": 35, "tag_url": "search/?keyword=%E6%97%B6%E6%94%BF", "title": "外交部前發言人章啓月再任大使", "chinese_tag": "時政", "source": "長安街知事", "group_source": 2, "has_gallery": false, "media_url": "/c/user/4327876576/", "media_avatar_url": "//p3.pstatp.com/large/3538/9145332", "source_url": "/group/6593111799187374596/", "article_genre": "article", "item_id": "6593111799187374596", "is_feed_ad": false, "behot_time": 1535107376, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153507834923010ad6116e1", "group_id": "6593111799187374596", "middle_image": "http://p99.pstatp.com/list/pgc-image/153507834923010ad6116e1"}, {"single_mode": true, "abstract": "作爲中國新“四大發明”之一的共享單車，不僅改變了我們中國每個人的出行方式，它還走向國外，給外國帶去了互聯網+自行車交通的新方式。", "middle_mode": true, "more_mode": true, "tag": "news_tech", "label": ["自行車", "英國", "交通", "經濟", "科技"], "comments_count": 70, "tag_url": "news_tech", "title": "大量共享單車在英國遭偷盜破壞 摩拜公司憤怒發文警告", "chinese_tag": "科技", "source": "全球每日趣聞", "group_source": 2, "has_gallery": false, "media_url": "/c/user/100722784942/", "media_avatar_url": "//p7.pstatp.com/large/a14900002c28a48e0dcc", "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/1535008838168ab3417847e"}, {"url": "//p99.pstatp.com/list/pgc-image/1535010175592846a23e90b"}, {"url": "//p3.pstatp.com/list/pgc-image/153501044157296ceeb2798"}], "source_url": "/group/6592825874624545287/", "article_genre": "article", "item_id": "6592825874624545287", "is_feed_ad": false, "behot_time": 1535107361, "image_url": "//p3.pstatp.com/list/190x124/pgc-image/1535008838168ab3417847e", "group_id": "6592825874624545287", "middle_image": "http://p3.pstatp.com/list/pgc-image/1535008838168ab3417847e"}, {"single_mode": true, "abstract": "她忍痛送別的愛人，正是曾寫過《血染的風采》《黃土高坡》等經典歌曲的著名作曲家蘇越。蘇越和安雯也許對現在的“90後”“00後”來說，蘇越的名字和他的那些歌曲，過於陌生。", "middle_mode": true, "more_mode": false, "tag": "news_entertainment", "label": ["流行音樂", "音樂", "歌手", "黃格選", "日本"], "comments_count": 728, "tag_url": "news_entertainment", "title": "從內地樂壇教父到詐騙犯，爲了面子，他埋葬了自己的人生與愛情", "chinese_tag": "娛樂", "source": "環球人物雜誌", "group_source": 2, "has_gallery": false, "media_url": "/c/user/3994535116/", "media_avatar_url": "//p2.pstatp.com/large/2266/1953108337", "source_url": "/group/6592884788338622990/", "article_genre": "article", "item_id": "6592884788338622990", "is_feed_ad": false, "behot_time": 1535107346, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153502535842658f7bb37af", "group_id": "6592884788338622990", "middle_image": "http://p99.pstatp.com/list/pgc-image/153502535842658f7bb37af"}, {"single_mode": true, "abstract": "人民網北京8月24日電日前,福布斯公佈2018年全球收入最高的男演員榜。作爲演員、導演、製片和商人多重身份的喬治·克魯尼以2.39億美元登頂。", "middle_mode": true, "more_mode": true, "tag": "news_entertainment", "label": ["成龍", "海姆斯沃斯", "喬治·克魯尼", "小羅伯特·唐尼", "威爾·史密斯"], "comments_count": 25, "tag_url": "news_entertainment", "title": "全球收入最高男演員出爐：成龍第五系唯一上榜中國演員", "chinese_tag": "娛樂", "source": "人民網", "group_source": 2, "has_gallery": false, "media_url": "/c/user/50502346173/", "media_avatar_url": "//p3.pstatp.com/large/ca400072481685ad43b", "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/1535092968677eb1e2608f3"}, {"url": "//p99.pstatp.com/list/pgc-image/1535092969914fd64f5615a"}, {"url": "//p99.pstatp.com/list/pgc-image/1535092969609860dcf09ed"}], "source_url": "/group/6593174112720388616/", "article_genre": "article", "item_id": "6593174112720388616", "is_feed_ad": false, "behot_time": 1535107331, "image_url": "//p3.pstatp.com/list/190x124/pgc-image/1535092968677eb1e2608f3", "group_id": "6593174112720388616", "middle_image": "http://p3.pstatp.com/list/pgc-image/1535092968677eb1e2608f3"}, {"single_mode": true, "abstract": "說到“漂”，總避免不了“房租”這個話題。有報告顯示，目前中國租賃人口已接近2億人。根據中國房地產業協會的數據，北京、深圳、上海、杭州、廣州這5個城市房租水平居前，其中北京爲92.33元/月/㎡。", "middle_mode": true, "more_mode": true, "tag": "news_house", "label": ["租房", "經濟", "盤點", "房產"], "comments_count": 36, "tag_url": "search/?keyword=%E6%88%BF%E4%BA%A7", "title": "各大城市房租排行榜出爐！你一個月要交多少錢？", "chinese_tag": "房產", "source": "南方週末", "group_source": 2, "has_gallery": false, "media_url": "/c/user/50371413220/", "media_avatar_url": "//p3.pstatp.com/large/ca2000539631fc1d01d", "image_list": [{"url": "//p99.pstatp.com/list/pgc-image/15350346510653c6d876cd7"}, {"url": "//p3.pstatp.com/list/pgc-image/1535034651276d0f13bda31"}, {"url": "//p3.pstatp.com/list/pgc-image/1535034650096cd059c4f9f"}], "source_url": "/group/6592923629598540301/", "article_genre": "article", "item_id": "6592923629598540301", "is_feed_ad": false, "behot_time": 1535107316, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/15350346510653c6d876cd7", "group_id": "6592923629598540301", "middle_image": "http://p99.pstatp.com/list/pgc-image/15350346510653c6d876cd7"}, {"single_mode": true, "abstract": "雖然是參加婚禮，但是他與奧地利外長卡琳⋅克奈斯爾和奧地利總理塞巴斯蒂安⋅庫爾茨還成功進行了“商談”。", "middle_mode": true, "more_mode": true, "tag": "news_world", "label": ["普京", "婚禮", "外交政策", "政治", "柔道"], "comments_count": 6, "tag_url": "news_world", "title": "因爲對普京做出這個動作，奧地利外長被要求辭職", "chinese_tag": "國際", "source": "環球網", "group_source": 2, "has_gallery": false, "media_url": "/c/user/5954781019/", "media_avatar_url": "//p1.pstatp.com/large/4d00054b126ceaf920", "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/153507129660162fbe735e8"}, {"url": "//p99.pstatp.com/list/pgc-image/1535071296031f3dabca397"}, {"url": "//p9.pstatp.com/list/pgc-image/1535071296345e128c495f8"}], "source_url": "/group/6593081018062733837/", "article_genre": "article", "item_id": "6593081018062733837", "is_feed_ad": false, "behot_time": 1535107301, "image_url": "//p3.pstatp.com/list/190x124/pgc-image/153507129660162fbe735e8", "group_id": "6593081018062733837", "middle_image": "http://p3.pstatp.com/list/pgc-image/153507129660162fbe735e8"}], "next": {"max_behot_time": 1535107301}}
至於cookie過期時間自己測試吧！有點湊篇幅的嫌疑