Flask Proxy Server Error in HTTPS -适用于HTTP但不适用HTTPS

Question

我们已经在flask上设置了一个基本的代理服务器。它在HTTP中工作得很好，但是当在HTTPS中发出请求时，它会抛出一些难以调试的问题。

在HTTPS中，烧瓶似乎没有正确解析请求。

下面是更详细的描述

服务器代码

from flask import Flask, request
import requests
from requests import exceptions

application = Flask(__name__)
@application.route("/",methods=["GET","POST","DELETE","PATCH","PUT"])
def route():
try:
response = requests.request(method=request.method, url=request.url, headers=request.headers,timeout=60)
print(response.status_code, response.text)
return (response.content, response.status_code, response.headers.items())
except exceptions.ProxyError:
return (b"Error", 408, request.headers.items())
if __name__ == "__main__":
application.run(host="0.0.0.0", port=39100, debug=True, use_reloader=False,ssl_context=("certs/cert.pem","certs/key.pem"))

请求示例代码

import requests
proxies = {"http":"https://127.0.0.1:39100", "https":"https://127.0.0.1:39100"}
# does not work if the request is http://ipv4.icanhazip.com/
# works if the request is http://ipv4.icanhazip.com/
testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False) 
print(testing)
print(testing.content)

误差

/Users/a/PycharmProjects/invisiProxy/venv/bin/python /Users/a/PycharmProjects/invisiProxy/sudoRequest.py
Traceback (most recent call last):
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 994, in _prepare_proxy
conn.connect()
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connection.py", line 369, in connect
self._tunnel()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 924, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 404 NOT FOUND
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/a/PycharmProjects/invisiProxy/sudoRequest.py", line 6, in <module>
testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False)
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 559, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))
Process finished with exit code 1

Answer 1

在requests上使用proxies将导致客户端通过CONNECT方法使用HTTP隧道。为了在没有任何MITM代理的情况下建立端到端加密(参见urllib3对这种情况的描述——https://urllib3.readthedocs.io/en/stable/advanced-usage.html#http-and-https-proxies)，客户端有直接的TCP通道到目标是必要的。

您的flask服务器不支持CONNECT方法，因此它在请求时返回404。该行为可以在MVP服务器上观察到:

# client request http://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:08:12] "GET http://ipv4.icanhazip.com/ HTTP/1.1" 200 -
# client request https://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:03:48] "CONNECT ipv4.icanhazip.com:443 HTTP/1.0" 404

技术上，预期的行为是可能执行的…

# openssl s_client --connect 127.0.0.1:39100
CONNECTED(00000003)
---
Certificate chain
0 s:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
i:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
---
read R BLOCK
GET https://ipv4.icanhazip.com/ HTTP/1.1
Host: ipv4.icanhazip.com
HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.7.3
...
IP_REDACTED
read:errno=0

…这种情况是记录在案的，并且是可调整的。在urllib3级别(由requests用于实际连接)与use_forwarding_for_https的https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html urllib3.ProxyManager

,

• 一般不建议这样使用，因为它违反了TLS本身的设计(客户机和资源之间的端到端加密)。

• 在当前的requests实现中，似乎没有办法通过标准api从开发人员传递所需的标志。get_connection函数不允许向urllib3.ProxyManager传递任何额外的proxy_kwargs

https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L352
def get_connection(self, url, proxies=None):
...
if proxies:
...
proxy_manager = self.proxy_manager_for(proxy)
conn = proxy_manager.connection_from_url(url)

https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L201
def proxy_manager_for(self, proxy, **proxy_kwargs):
...
manager = self.proxy_manager[proxy] = proxy_from_url(
...
**proxy_kwargs,
)

我建议使用完整的http代理，而不是flask server。