如何发出web请求来模拟浏览器(即TLS握手/客户端问候)

Question

我想知道如何发出一个典型的网络请求(例如使用python请求(，它将能够完美地模拟网络浏览器。我说的不是琐碎的用户代理，而是客户端Hello消息，它包括密码套件订单等。

例如，我向https://www.ssllabs.com/ssltest/viewMyClient.html，保存输出并将结果与常规chrome浏览器显示的结果进行比较

这是我到目前为止所拥有的(丑陋得可怕……我在玩多个适配器(

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context
import re
import json
import random
import subprocess
import os

from requests.packages.urllib3.poolmanager import PoolManager
from requests.packages.urllib3.util import ssl_
from urllib3.poolmanager import PoolManager
from requests.adapters import HTTPAdapter

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

# This is the 2.11 Requests cipher string, containing 3DES.
CIPHERS = (
'ECDH+AESGCM:ECDH+AES256:ECDH+AES128:'
'RSA+AESGCM'
)

class DESAdapter(HTTPAdapter):
"""
A TransportAdapter that re-enables 3DES support in Requests.
"""
def init_poolmanager(self, *args, **kwargs):
context = create_urllib3_context(ciphers=CIPHERS)
context.options |= ssl.OP_NO_SSLv3
context.options |= ssl.OP_NO_SSLv2
context.options |= ssl.OP_NO_SSLv3
context.options |= ssl.OP_NO_TLSv1
context.options |= ssl.OP_NO_TLSv1_1
kwargs['ssl_context'] = context
return super(DESAdapter, self).init_poolmanager(*args, **kwargs)
def proxy_manager_for(self, *args, **kwargs):
context = create_urllib3_context(ciphers=CIPHERS)
context.options |= ssl.OP_NO_SSLv3
kwargs['ssl_context'] = context
return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)


class Ssl3HttpAdapter(HTTPAdapter):
""""Transport adapter" that allows us to use SSLv3."""
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(
num_pools=connections, maxsize=maxsize,
block=block, ssl_version=ssl.OP_NO_SSLv3)


CIPHERS = (
'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:'
'ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:'
'AES256-GCM-SHA384:AES128-SHA:AES256-SHA:3DES'   #TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
)

class TlsAdapter(HTTPAdapter):
def __init__(self, ssl_options=0, **kwargs):
self.ssl_options = ssl_options
super(TlsAdapter, self).__init__(**kwargs)
def init_poolmanager(self, *pool_args, **pool_kwargs):
ctx = ssl_.create_urllib3_context(ciphers=CIPHERS,options=self.ssl_options)
self.poolmanager = PoolManager(*pool_args,
ssl_context=ctx,
**pool_kwargs)

s = requests.session()
adapter = TlsAdapter(ssl.OP_NO_SSLv3)
s.mount("https://", adapter)
r = s.get('https://www.ssllabs.com/ssltest/viewMyClient.html')

if 'Capabilities' in r.text:   
os.remove("ssl-test-python.html")
hs = open("ssl-test-python.html", "a")
hs.write(r.text)
hs.close()
os.startfile('ssl-test-python.html')

当将保存的输出与chrome显示的内容进行比较时，主要的不一致似乎是：

1(密码的顺序不正确

2(不应启用sslv3
<1(显示的"签名算法"太多>
<2br>4("命名组"不正确

5(可能缺少其他内容
>

有没有办法解决这些问题？我不想使用硒

Answer 1

您现在可以完美地模拟python中的浏览器了！使用curl_cffi。它是curl模拟项目的python绑定。

pip install curl_cffi

from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser

Answer 2

python和chrome的tls指纹不同的最大原因是python使用openssl，chrome使用boringssl。因此，您不能使用python设置来模拟chrome的tls指纹。您可以将openssl替换为boringssl来从源代码编译python。由于python不支持boringssl，因此需要更改一些源代码

如果你能用围棋。也许你可以试试这个，https://github.com/refraction-networking/utls.Go实现了tls本身。