爬蟲篇（2）使用pyexecjs破解js中cookies

首先下載pyexecjs：

pip install  PyExecJS

如果有需要，自行下載PyV8 , Node.js , PhantomJS等

使用參考：https://github.com/doloopwhile/PyExecJS

PyExecJS文檔：https://pypi.org/project/PyExecJS/

此次採集鏈接：http://www.landchina.com/default.aspx?tabid=226

直接請求：

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import requests
url = 'http://www.landchina.com/default.aspx?tabid=226'
headers = {
           "User-Agent": 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36',
           }

html = requests.get(url, headers=headers, verify=False)
print("txte:",html.text)

with open('數據2.html', 'w',encoding='utf8') as f:
    f.write(str(html.text) + '\n')
print("數據.html保存完畢！")

打開'數據2.html'，獲取js數據：

<!DOCTYPE html><html xmlns="http://www.w3.org/1999/xhtml"><head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <meta http-equiv="Cache-Control" content="no-store, no-cache, must-revalidate, post-check=0, pre-check=0"/><meta http-equiv="Connection" content="Close"/>
    <script type="text/javascript">
        function stringToHex(str){
            var val="";for(var i = 0; i < str.length; i++){if(val == "")val = str.charCodeAt(i).toString(16);else val += str.charCodeAt(i).toString(16);}return val;
        }
        function YunSuoAutoJump(){
            var width =screen.width; var height=screen.height; var screendate = width + "," + height;var curlocation = window.location.href;if(-1 == curlocation.indexOf("security_verify_")){ 
                document.cookie="srcurl=" + stringToHex(window.location.href) + ";path=/;";}self.location = "/default.aspx?tabid=226&security_verify_data=" + stringToHex(screendate);
        }
    </script><script>setTimeout("YunSuoAutoJump()", 50);
</script
</head><!--2019-06-20 14:08:07--></html>

可以看到document.cookie=******************，這個就是js設置cookies的代碼

使用PyExecJS執行stringToHex（）函數看看，代碼如下：

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import execjs
jstext = '''
function stringToHex(str){var val="";for(var i = 0; i < str.length; i++){if(val == "")
val = str.charCodeAt(i).toString(16);else val += str.charCodeAt(i).toString(16);}return val;}
'''
ctx = execjs.compile(jstext)# 編譯JS代碼
a = ctx.call("stringToHex","9999")
print(a)

或者使用nodejs

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import execjs
import execjs.runtime_names
jstext = '''
function stringToHex(str){var val="";for(var i = 0; i < str.length; i++){if(val == "")
val = str.charCodeAt(i).toString(16);else val += str.charCodeAt(i).toString(16);}return val;}
'''
node = execjs.get(execjs.runtime_names.Node)
ctx = node.compile(jstext)# 編譯JS代碼
a = ctx.call("stringToHex","9999")
print(a)

執行結果：

執行js成功，未完待續。。。。

繼續執行YunSuoAutoJump()函數：

node = execjs.get(execjs.runtime_names.Node)
ctx = node.compile(jstext)
a = ctx.call("YunSuoAutoJump",)

報錯：execjs._exceptions.ProgramError: ReferenceError: screen is not defined

screen未發現，screen是window的對象，發現未配置PhantomJS（參考：js逆向解密之網絡爬蟲+phantomjs 安裝教程+下載PhantomJS）

配置好PhantomJS編譯環境，並修改其js後執行代碼：

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import execjs,os
import execjs.runtime_names
# os.environ["EXECJS_RUNTIME"] = "Node"
print("當前環境:",execjs.get().name) # this value is depends on your environment.

jstext = '''
function stringToHex(str){var val="";for(var i = 0; i < str.length; i++){if(val == "")
    val = str.charCodeAt(i).toString(16);else val += str.charCodeAt(i).toString(16);}return val;}

function YunSuoAutoJump(){ var width =screen.width; var height=screen.height; var screendate = width + "," + height;
    var curlocation = window.location.href;if(-1 == curlocation.indexOf("security_verify_")){ 
    document.cookie="srcurl=" + stringToHex(window.location.href) + ";path=/;";
    fcookie="srcurl=" + stringToHex(window.location.href) + ";path=/;";    //加入一個變量記錄cookies
    }self.location = "/default.aspx?tabid=226&security_verify_data=" + stringToHex(screendate);
    return fcookie                                                         //返回cookies
;}'''
os.environ["EXECJS_RUNTIME"] = "PhantomJS"
print("修改環境:",execjs.get().name)
ctx = execjs.compile(jstext)
print(ctx.call("YunSuoAutoJump"))

注意：這裏的js中我加入了兩條代碼：

fcookie="srcurl=" + stringToHex(window.location.href) + ";path=/;";    //加入一個變量記錄cookies

return fcookie //返回cookies

執行結果：

修改代碼後，使用requests發起請求，代碼如下：

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import requests,re,execjs,os
url = 'http://www.landchina.com/default.aspx?tabid=226'
headers = {
           "User-Agent": 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36',
           }
html = requests.get(url, headers=headers, verify=False)
jstext = re.findall(r'<script type="text/javascript">(.+?)</script>',html.text)[0]
print("jstext:",jstext)
os.environ["EXECJS_RUNTIME"] = "PhantomJS"
jstext = jstext.replace('";path=/;";','";path=/;";fcookies="srcurl=" + stringToHex(window.location.href) + ";path=/;";')
jstext = jstext.replace('stringToHex(screendate);','stringToHex(screendate);return fcookies;')
print("jstext:",jstext)
ctx = execjs.compile(jstext)
print(ctx.call("YunSuoAutoJump"))

輸出結果：

jstext: function stringToHex(str){var val="";for(var i = 0; i < str.length; i++){if(val == "")val = str.charCodeAt(i).toString(16);else val += str.charCodeAt(i).toString(16);}return val;}function YunSuoAutoJump(){ var width =screen.width; var height=screen.height; var screendate = width + "," + height;var curlocation = window.location.href;if(-1 == curlocation.indexOf("security_verify_")){ document.cookie="srcurl=" + stringToHex(window.location.href) + ";path=/;";}self.location = "/default.aspx?tabid=226&security_verify_data=" + stringToHex(screendate);}
jstext: function stringToHex(str){var val="";for(var i = 0; i < str.length; i++){if(val == "")val = str.charCodeAt(i).toString(16);else val += str.charCodeAt(i).toString(16);}return val;}function YunSuoAutoJump(){ var width =screen.width; var height=screen.height; var screendate = width + "," + height;var curlocation = window.location.href;if(-1 == curlocation.indexOf("security_verify_")){ document.cookie="srcurl=" + stringToHex(window.location.href) + ";path=/;";fcookies="srcurl=" + stringToHex(window.location.href) + ";path=/;";}self.location = "/default.aspx?tabid=226&security_verify_data=" + stringToHex(screendate);return fcookies;}
srcurl=66696c653a2f2f2f433a2f55736572732f41444d494e497e312f417070446174612f4c6f63616c2f54656d702f657865636a7375753575783872652e6a73;path=/;

js中發現代碼段：self.location="*******************8"（不懂的朋友可以看看：js中location.href的用法）

self.location = "/default.aspx?tabid=226&security_verify_data=" + stringToHex(screendate);

通過fildder抓包分析，可以確定下一個請求鏈接http://www.landchina.com/default.aspx?tabid=226&security_verify_data=313533362c383634（注意，這個鏈接是js生成的）：

故，大概流程爲首先請求http://www.landchina.com/default.aspx?tabid=226，獲取js後解析獲得cookies和verify_url信息，隨後利用requests.session()訪問js中的鏈接verify_url（我的是http://www.landchina.com/default.aspx?tabid=226&security_verify_data=313533362c383634）,最後攜帶cookies信息再次請求http://www.landchina.com/default.aspx?tabid=226，最終拿到界面數據，整合代碼如下：

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import requests,re,execjs,os
session = requests.session()
url = 'http://www.landchina.com/default.aspx?tabid=226'
headers = {
           "User-Agent": 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36',
           }
html = session.get(url, headers=headers, verify=False)
jstext = re.findall(r'<script type="text/javascript">(.+?)</script>',html.text)[0]
print("正則獲取js:",jstext)
os.environ["EXECJS_RUNTIME"] = "PhantomJS"          #設置execjs使用PhantomJS編譯
jstext1 = jstext.replace('";path=/;";','";path=/;";fcookies="srcurl=" + stringToHex(window.location.href) + ";path=/;";')
jstext1 = jstext1.replace('stringToHex(screendate);','stringToHex(screendate);return fcookies;')
print("修改js:",jstext1)
ctx = execjs.compile(jstext1)           #編譯js
cookie1 = ctx.call("YunSuoAutoJump")    #執行js中 YunSuoAutoJump（）函數
print("js解析獲得cookie1:",cookie1)

jstext2 = jstext.replace('stringToHex(screendate);','stringToHex(screendate);verify_url= "/default.aspx?tabid=226&security_verify_data=" + stringToHex(screendate);return verify_url;')
print("再次修改js獲取以verify_url:",jstext2)
ctx = execjs.compile(jstext2)
verify_url = "http://www.landchina.com"+ctx.call("YunSuoAutoJump")
print("verify_url:",verify_url)

#分割cookies
cookie1= {item.split('=')[0]:item.split('=')[1] for item in cookie1.split('; ')}

html = session.get(verify_url)
# html = requests.get(next_url, headers=headers, verify=False,cookies=cookie1)
print("verify_url請求-html-2:",html.text)
html = session.get(url, headers=headers, verify=False)
print("最後獲得界面-長度:",len(html.text))
print("最後獲得界面-html:",html.text)

運行結果：

成功獲取界面，後面就是數據清洗，url提取，入庫等操作了。

參考文章：爬蟲--破解網站通過js加密生成cookie（一）

本博客屬技術研究，侵告刪。

爬蟲篇（2）使用pyexecjs破解js中cookies

.Net 8.0 下的新RPC，IceRPC之試試的新玩法"打洞"

關於遊戲付費的一點想法

我通過CKA和CKS啦！

《最新出爐》系列入門篇-Python+Playwright自動化測試-42-強大的可視化追蹤利器Trace Viewer

大數據怎麼學？對大數據開發領域及崗位的詳細解讀，完整理解大數據開發領域技術體系

Python 常用小技巧（持續更新）

爬蟲篇（3）scrapy 去重與 scrapy_redis 去重與布隆過濾器（轉）

爬蟲篇（2.1）selenium開啓開發者模式

爬蟲篇（2.3）scrapy通用爬蟲以及setting設置中一些提升效率的方式

爬蟲篇（2）使用pyexecjs破解js中cookies

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結