我正试图在AWS上部署我的django应用程序。我使用bs4和请求在我的应用程序中创建了一个网络爬虫。我用它来从电子商务网站获取数据。它在亚马逊上运行得很好，但当我试图从newegg中抓取时，它会抛出这个(见图1(。但是在localhost上也可以使用相同的代码。起初我认为这是用户代理的问题，我尝试了其他用户代理，但没有成功。
我真的不知道是什么问题，所以如果你知道是什么原因导致了这个错误，请告诉我这将非常有用。
下面的代码和图像：
图像1
headers = {"User-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"}
url = "https://www.newegg.com/p/pl?d="+query
data = re.get(url,headers=headers)
soup = bs4.BeautifulSoup(data.content, 'lxml')
items_container = soup.find("div", {"class":"list-wrap"})
items = items_container.find("div", {"class":"items-view"})
prdList = []
for item in items.find_all("div", {"class":"item-container"}):
header = item.find("a", {"class":"item-title"})
title = header.text
link = header["href"]
img = item.find("a", {"class":"item-img"})
imgsrc = img.find("img")["src"]
price = item.find("li", {"class":"price-current"})
if price == None:
price = '$--'
else:
price = price.find("strong").text
prdList.append(Scraper.prdTmplt(imgsrc, title, price, link, site))
allProduct = prdList
return allProduct
错误：
Environment:
Request Method: GET
Request URL: http://ec2-54-179-183-56.ap-southeast-1.compute.amazonaws.com:8000/search?sq=gaming+laptop&site_selected=newegg&sq-submit=Search
Django Version: 2.2.6
Python Version: 3.6.8
Installed Applications:
['groups.apps.GroupsConfig',
'product_details.apps.ProductDetailsConfig',
'scraper.apps.ScraperConfig',
'pages.apps.PagesConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "/home/ubuntu/django/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)
File "/home/ubuntu/django/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
115. response = self.process_exception_by_middleware(e, request)
File "/home/ubuntu/django/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
113. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/ubuntu/best-price-py/pages/views.py" in search
29. context['bb'] = Scraper.neweggList(search_query, site)
File "/home/ubuntu/best-price-py/scraper/scrapers.py" in neweggList
120. data = re.get(url)
File "/home/ubuntu/django/lib/python3.6/site-packages/requests/api.py" in get
75. return request('get', url, params=params, **kwargs)
File "/home/ubuntu/django/lib/python3.6/site-packages/requests/api.py" in request
60. return session.request(method=method, url=url, **kwargs)
File "/home/ubuntu/django/lib/python3.6/site-packages/requests/sessions.py" in request
519. prep = self.prepare_request(req)
File "/home/ubuntu/django/lib/python3.6/site-packages/requests/sessions.py" in prepare_request
462. hooks=merge_hooks(request.hooks, self.hooks),
File "/home/ubuntu/django/lib/python3.6/site-packages/requests/models.py" in prepare
314. self.prepare_headers(headers)
File "/home/ubuntu/django/lib/python3.6/site-packages/requests/models.py" in prepare_headers
446. for header in headers.items():
Exception Type: AttributeError at /search
Exception Value: 'set' object has no attribute 'items'
您传递的头dict被呈现为一个集合，而不是字典。看看图片中的垃圾场。