发布请求不适用于scrapy，但适用于请求

Question

请求代码：

listings_url = "https://www.biltorvet.dk/Api/Search/Page"
form_data = {
"pageNumber": "1",
"searchOrigin": "1",
"searchValue": "22526899",
"sort": ""
}
response = requests.post(listings_url, json=form_data)
if response.status_code == 200:
data = response.json()
print(data)

报废代码：

class BiltorvetScraperSpider(scrapy.Spider):
name = 'biltorvet'
listings_url = "https://www.biltorvet.dk/Api/Search/Page"
form_data = {
"pageNumber": "1",
"searchOrigin": "1",
"searchValue": "22526899",
"sort": ""
}
def start_requests(self):
yield FormRequest(url=self.listings_url, callback=self.parse, body=json.dumps(self.form_data))
def parse(self, response):
print(response.text)

我收到了400英镑。我也尝试过头球，但结果是一样的。尝试将参数从body更改为json仍然没有影响。

Answer 1

这应该达到目的：

import json
import scrapy
from scrapy.http.request import Request
class BiltorvetScraperSpider(scrapy.Spider):
name = 'biltorvet'
start_url = "https://www.biltorvet.dk/Api/Search/Page"

form_data = {
"pageNumber": "1",
"searchOrigin": "1",
"searchValue": "22526899",
"sort": ""
}

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36',
'Content-Type': 'application/json; charset=UTF-8',
}
def start_requests(self):
yield Request(
self.start_url, 
headers=self.headers,
callback=self.parse, 
method='POST',
body=json.dumps(self.form_data)
)
def parse(self, response):
print(response.json())

或者，您可以根据文档尝试如下：

from scrapy.http import JsonRequest
def start_requests(self):
yield JsonRequest(
self.start_url, 
headers=self.headers,
callback=self.parse, 
data=self.form_data
)
def parse(self, response):
print(response.json())