带有重音字符 URL 的请求包生成 UnicodeDecodeError: 无效的延续字节

Question

我有以下网址链接，它有重音字符：

https://www.janes.com/...tamandaré...等。

当我尝试请求链接时，出现错误：

UnicodeDecodeError："utf-8"编解码器无法解码位置的字节0xe9 89：延续字节无效

这是我的代码：

import requests
def request_site(url):
    return requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0)'})
if __name__ == '__main__':
    url = 'https://www.janes.com/article/87665/laad-2019-united-kingdom-s-sea-signs-mou-with-brazilian-siatt-for-tamandaré-class-corvette-torpedo-tubes'
    print(request_site(url))

完整错误：

Traceback (most recent call last):
  File "D:/OneDrive/PhD/Web Crawler/playground.py", line 104, in <module>
    print(request_site(url))
  File "D:/OneDrive/PhD/Web Crawler/playground.py", line 73, in request_site
    return requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0)'})
  File "C:Python35libsite-packagesrequestsapi.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:Python35libsite-packagesrequestsapi.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:Python35libsite-packagesrequestssessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:Python35libsite-packagesrequestssessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "C:Python35libsite-packagesrequestssessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "C:Python35libsite-packagesrequestssessions.py", line 149, in resolve_redirects
    url = self.get_redirect_target(resp)
  File "C:Python35libsite-packagesrequestssessions.py", line 115, in get_redirect_target
    return to_native_string(location, 'utf8')
  File "C:Python35libsite-packagesrequests_internal_utils.py", line 25, in to_native_string
    out = string.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 89: invalid continuation byte

我发现许多类似的问题(如链接(，但没有一个为同一问题提出解决方案，以及之前的所有解决方案都是针对 python2 的。

Answer 1

只需要

一个快速编码，但你需要从url中删除http://，因为它也会对其进行编码：

import requests
def request_site(url):
    return requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0)'})
if __name__ == '__main__':
    url = 'www.janes.com/article/87665/laad-2019-united-kingdom-s-sea-signs-mou-with-brazilian-siatt-for-tamandaré-class-corvette-torpedo-tubes'
    url_encode = 'http://' + urllib.parse.quote(url.encode('latin-1'))
    print(request_site(url_encode))