爬蟲指紋 JA3

JA3 指紋裏面，很大的一塊就是 Cipher Suits，也就是加密算法。而 requests 裏面默認的加密算法如下
參考文件:https://www.cnblogs.com/Eeyhan/p/15662849.html
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:!eNULL:!MD5

設計到代碼

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context
 
ORIGIN_CIPHERS = ('ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES')
 
 
class DESAdapter(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        """
        A TransportAdapter that re-enables 3DES support in Requests.
        """
        CIPHERS = ORIGIN_CIPHERS.split(':')
        random.shuffle(CIPHERS)
        CIPHERS = ':'.join(CIPHERS)
        self.CIPHERS = CIPHERS + ':!aNULL:!eNULL:!MD5'
        super().__init__(*args, **kwargs)
        
        
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=self.CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)
 
    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=self.CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)

常出現情況有時候scrapy沒法爬,而request可以爬

則需要在scrapy的配置文件中配置

DOWNLOADER_CLIENT_TLS_CIPHERS = 'DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:!eNULL:!MD5'

另外情況安全通過這個特效對你使用python,request進行檢測

這時候有幾個方法

1.訪問ip指定host繞過waf

2.代理中轉請求

在本地啓動代理服務器，如Burp Suite，發起http請求時指定代理服務器爲burp的地址，讓burp來進行TLS握手，算是一種曲線救國的方法

3.更換request工具庫

Requests其實是對urllib3的一個封裝，那python有沒有不用urllib的http request庫呢？

翻了翻aiohttp的源碼發現貌似並沒有用urllib3，抓包發現tls指紋和requests也有着明顯的差異

實際測試aiohttp確實沒有被攔截

4.魔改requests

從根本上解決問題，debug跟蹤到了幾處可能可以修改TLS握手特徵的代碼

/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py

https://cloud.tencent.com/developer/article/1875625魔改原文

另外魔改辦法改ssl中DEFAULT_CIPHERS