错误下载PDF文件

Question

我有以下(简化(代码：

import os
import scrapy
class TestSpider(scrapy.Spider):
    name = 'test_spider'
    start_urls = ['http://www.pdf995.com/samples/pdf.pdf', ]
    def parse(self, response):
        save_path = 'test'
        file_name = 'test.pdf'
        self.save_page(response, save_path, file_name)
    def save_page(self, response, save_dir, file_name):
        os.makedirs(save_dir, exist_ok=True)
        with open(os.path.join(save_dir, file_name), 'wb') as afile:
            afile.write(response.body)

当我运行它时，我会收到此错误：

[scrapy.core.scraper] ERROR: Error downloading <GET http://www.pdf995.com/samples/pdf.pdf>
Traceback (most recent call last):
File "C:Python36libsite-packagestwistedinternetdefer.py", line 1301, in _inlineCallbacks
    result = g.send(result)
File "C:Python36libsite-packagesscrapycoredownloadermiddleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
File "C:Python36libsite-packagestwistedinternetdefer.py", line 1278, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200 http://www.pdf995.com/samples/pdf.pdf>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):   
File "C:Python36libsite-packagestwistedinternetdefer.py", line 1301, in _inlineCallbacks
    result = g.send(result)   
File "C:Python36libsite-packagesscrapycoredownloadermiddleware.py", line 53, in process_response
    spider=spider)   
File "C:Python36libsite-packagesscrapy_beautifulsoupmiddleware.py", line 16, in process_response
    return response.replace(body=str(BeautifulSoup(response.body, self.parser)))   
File "C:Python36libsite-packagesscrapyhttpresponse__init__.py", line 79, in replace
    return cls(*args, **kwargs)   
File "C:Python36libsite-packagesscrapyhttpresponse__init__.py", line 20, in __init__
    self._set_body(body)   
File "C:Python36libsite-packagesscrapyhttpresponse__init__.py", line 55, in _set_body
    "Response body must be bytes. " 
TypeError: Response body must be bytes. If you want to pass unicode body use TextResponse or HtmlResponse.

我需要引入中间件或其他内容来处理此问题吗？至少在其他示例中，这看起来应该是有效的。

注意：目前我没有使用管道PDF查看是否已下载。如前所述，许多样本做了我在这里做的事情，所以我认为这会更容易且工作。

Answer 1

问题是因为您自己的scrapy_beautifulsoupmiddleware.py试图替换return response.replace(body=str(BeautifulSoup(response.body, self.parser)))。

您需要纠正这一点，这应该解决问题