使用urlopen时出现HTTP 406不可接受的客户端错误

Question

我使用urllib.request.urlopen来查询URLhttp://dblp.org/db/conf/lak/index.由于某些原因，我无法使用Python模块urlib访问该网站，因为我收到以下HTTP状态代码错误：

HTTPError：HTTP错误406：不可接受

这是我用来提出这个请求的代码：

from urllib.request import urlopen
from bs4 import BeautifulSoup
url = 'http://dblp.org/db'
html = urlopen(url).read()
soup = BeautifulSoup(html)
print(soup.prettify())

我不确定是什么导致了这个错误，我需要帮助解决这个错误。

以下是与此错误相关的堆栈跟踪：

HTTPError                                 Traceback (most recent call last)
<ipython-input-5-b158a1e893a0> in <module>
----> 1 html = urlopen("https://dblp.org/db").read()
2 #print(html)
3 soup = BeautifulSoup(html)
4 soup.prettify()
~Anaconda3liburllibrequest.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
220     else:
221         opener = _opener
--> 222     return opener.open(url, data, timeout)
223 
224 def install_opener(opener):
~Anaconda3liburllibrequest.py in open(self, fullurl, data, timeout)
529         for processor in self.process_response.get(protocol, []):
530             meth = getattr(processor, meth_name)
--> 531             response = meth(req, response)
532 
533         return response
~Anaconda3liburllibrequest.py in http_response(self, request, response)
639         if not (200 <= code < 300):
640             response = self.parent.error(
--> 641                 'http', request, response, code, msg, hdrs)
642 
643         return response
~Anaconda3liburllibrequest.py in error(self, proto, *args)
567         if http_err:
568             args = (dict, 'default', 'http_error_default') + orig_args
--> 569             return self._call_chain(*args)
570 
571 # XXX probably also want an abstract factory that knows when it makes
~Anaconda3liburllibrequest.py in _call_chain(self, chain, kind, meth_name, *args)
501         for handler in handlers:
502             func = getattr(handler, meth_name)
--> 503             result = func(*args)
504             if result is not None:
505                 return result
~Anaconda3liburllibrequest.py in http_error_default(self, req, fp, code, msg, hdrs)
647 class HTTPDefaultErrorHandler(BaseHandler):
648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
650 
651 class HTTPRedirectHandler(BaseHandler):
HTTPError: HTTP Error 406: Not Acceptable

Answer 1

我正在查看406错误代码，当服务器无法使用请求中指定的accept标头进行响应时，就会发生这种情况。如果我能让urlopen正常工作，我也会发布这个答案。

当使用Python请求时，我没有得到这个错误

import requests
from bs4 import BeautifulSoup
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
raw_html = requests.get('http://dblp.org/db/conf/lak/index')
soup = BeautifulSoup(raw_html.content, 'html.parser')
print(soup.prettify())

下面的答案使用urlopen，这不会产生406错误。

from urllib.request import Request
from urllib.request import urlopen
from bs4 import BeautifulSoup
raw_request = Request('https://dblp.org/db/conf/lak/index')
raw_request.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0')
raw_request.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8')
resp = urlopen(raw_request)
raw_html = resp.read()
soup = BeautifulSoup(raw_html, 'html.parser')
print(soup.prettify())