使用项目加载程序scratchy获取键中的值

Question

我正试图从网页响应页面中的键中提取一些值。不幸的是，当我这样做时，它只返回键，而且我似乎无法获取值。因为每个键都是一个长列表，而且它们都有编号，所以我似乎无法想出如何获得所有键的值。

例如，这是我的工作代码：

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.loader import ItemLoader
from scrapy.item import Field
from itemloaders.processors import TakeFirst
class DepopItem(scrapy.Item):
brands = Field(output_processor=TakeFirst())
class DepopSpider(scrapy.Spider):
name = 'depop'
allowed_domains = ["depop.com"]
start_urls = ['https://webapi.depop.com/api/v2/search/filters/aggregates/?brands=1596&itemsPerPage=24&country=gb&currency=GBP&sort=relevance']

custom_settings = {
'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
}

def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(
url=url, 
callback=self.parse,
)
def parse(self, response):
resp= response.json()['brands']
for item in resp:
loader = ItemLoader(DepopItem(), selector=item)
loader.add_value('brands', item)

yield loader.load_item()

这将返回一个密钥列表：

{"brands": "1"}
{"brands": "2"}
{"brands": "3"}
{"brands": "4"}
{"brands": "5"}
{"brands": "7"}
{"brands": "9"}

相反，我想要与这些键对应的值：

{"brands": 946}
{"brands": 2376}
{"brands": 1286}
{"brands": 2774}
{"brands": 489}
{"brands": 11572}
{"brands": 1212}

Answer 1

使用values()或resp[item]。

示例：

import scrapy
from scrapy.loader import ItemLoader
from scrapy.item import Field
from itemloaders.processors import TakeFirst

class DepopItem(scrapy.Item):
brands = Field(output_processor=TakeFirst())

class DepopSpider(scrapy.Spider):
name = 'depop'
allowed_domains = ["depop.com"]
start_urls = ['https://webapi.depop.com/api/v2/search/filters/aggregates/?brands=1596&itemsPerPage=24&country=gb&currency=GBP&sort=relevance']
custom_settings = {
'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
}
def parse(self, response):
resp = response.json()['brands']
for item in resp.values():
loader = ItemLoader(DepopItem(), selector=item)
loader.add_value('brands', item['count'])
yield loader.load_item()

输出：

{'brands': 888}
{'brands': 1}
{'brands': 52}
{'brands': 138}
{'brands': 148}
...
...
...

Answer 2

我不确定scrapy怎么样，但你可以简单地做：

import requests
import json
from itertools import starmap
from requests.models import Response
from typing import Dict, List

url = "https://webapi.depop.com/api/v2/search/filters/aggregates/?brands=1596&itemsPerPage=24&country=gb&currency=GBP&sort=relevance"
resp: Response = requests.get(url)
data: Dict = json.loads(resp.text).get("brands")
values: List[Dict] = list(starmap(lambda k,v: {"brands": v["count"]}, data.items()))

输出：

[{'brands': 989},
{'brands': 1838},
{'brands': 2415},
{'brands': 1344},
...]