选择依赖的下拉阶段，并用砂纸切成薄片

Question

我正在尝试刮擦以下网站：https：//www.climatempo.com.br/climatologia/558/saopaulo-p。它有一个两个下拉菜单，第二个菜单取决于第一个菜单，因此我选择使用砂纸和飞溅。

。

我需要通过首先选择状态，然后选择城市来自动化位置的更改。我尝试了SplashFormRequest，但我无法更改城市列表。我的蜘蛛是（打印用于调试）：

import scrapy
from scrapy_splash import SplashRequest, SplashFormRequest

class ExampleSpider(scrapy.Spider):
    name = 'climatologia'
    def start_requests(self):
        urls = ['https://www.climatempo.com.br/climatologia/558/saopaulo-sp']
        for url in urls:
            yield SplashRequest(url=url, callback=self.parse,
                                endpoint='render.html',
                                args={'wait': 0.5},)
    def parse(self, response):
        print(response.url)
        state = response.css("select.slt-geo")[0].css("option::attr(value)").extract()
        print(state)
        return SplashFormRequest(response.url, method='POST',
                                 formdata={'sel-state-geo': 'SP'},
                                 callback=self.state_selected,
                                 args={'wait': 0.5})
    def state_selected(self, response):
        print('t:+)t:+)t:+)t:+)t:+)t:+)')
        print(response.css("select.slt-geo")[0].css("option::text").extract())
        print(response.css("select.slt-geo")[1].css("option::text").extract())

Answer 1

这是我建议的工作，如果您绝对必须使用网站菜单。脚本飞溅的唯一方法是通过LUA脚本。您将必须发送到执行终点并创建LUA脚本。我找到了您要选择的选项，但没有在哪里提交表单或网站上的功能。我确实必须翻译为英语。

我的建议是在浏览器检查员中查找这样的终点，这是看起来特别有趣的几个：https://www.climatempo.com.br/json/busca-estados

此端点给JSON提供如下

{"success":true,"message":"Resultados encontrados","time":"2017-11-30 16:05:20","totalRows":null,"totalPages":null,"page":null,"data":[{"idlocale":338,"idstate":31,"uf":"AC","state":"Acre","region":"N","latitude":null,"longitude":null},{"idlocale":339,"idstate":49,"uf":"AL","state":"Alagoas","region":"NE","latitude":null,"longitude":null},{"idlocale":340,"idstate":41,"uf":"AM","state":"Amazonas","region":"N","latitude":null,"longitude":null},{"idlocale":341,"idstate":30,"uf":"AP","state":"Amapu00e1","region":"N","latitude":null,"longitude":null},{"idlocale":342,"idstate":56,"uf":"BA","state":"Bahia","region":"NE","latitude":null,"longitude":null},{"idlocale":343,"idstate":44,"uf":"CE","state":"Cearu00e1","region":"NE","latitude":null,"longitude":null},{"idlocale":344,"idstate":47,"uf":"DF","state":"Distrito Federal","region":"CO","latitude":null,"longitude":null},{"idlocale":345,"idstate":45,"uf":"ES","state":"Espu00edrito Santo","region":"SE","latitude":null,"longitude":null},{"idlocale":346,"idstate":54,"uf":"GO","state":"Goiu00e1s","region":"CO","latitude":null,"longitude":null},{"idlocale":347,"idstate":52,"uf":"MA","state":"Maranhu00e3o","region":"NE","latitude":null,"longitude":null},{"idlocale":348,"idstate":53,"uf":"MG","state":"Minas Gerais","region":"SE","latitude":null,"longitude":null},{"idlocale":349,"idstate":39,"uf":"MS","state":"Mato Grosso do Sul","region":"CO","latitude":null,"longitude":null},{"idlocale":350,"idstate":40,"uf":"MT","state":"Mato Grosso","region":"CO","latitude":null,"longitude":null},{"idlocale":351,"idstate":50,"uf":"ND","state":"Nu00e3o Aplicu00e1vel","region":"ND","latitude":null,"longitude":null},{"idlocale":352,"idstate":55,"uf":"PA","state":"Paru00e1","region":"N","latitude":null,"longitude":null},{"idlocale":353,"idstate":37,"uf":"PB","state":"Parau00edba","region":"NE","latitude":null,"longitude":null},{"idlocale":354,"idstate":29,"uf":"PE","state":"Pernambuco","region":"NE","latitude":null,"longitude":null},{"idlocale":355,"idstate":33,"uf":"PI","state":"Piauu00ed","region":"NE","latitude":null,"longitude":null},{"idlocale":356,"idstate":32,"uf":"PR","state":"Paranu00e1","region":"S","latitude":null,"longitude":null},{"idlocale":357,"idstate":46,"uf":"RJ","state":"Rio de Janeiro","region":"SE","latitude":null,"longitude":null},{"idlocale":358,"idstate":35,"uf":"RN","state":"Rio Grande do Norte","region":"NE","latitude":null,"longitude":null},{"idlocale":359,"idstate":38,"uf":"RO","state":"Rondu00f4nia","region":"N","latitude":null,"longitude":null},{"idlocale":360,"idstate":43,"uf":"RR","state":"Roraima","region":"N","latitude":null,"longitude":null},{"idlocale":361,"idstate":48,"uf":"RS","state":"Rio Grande do Sul","region":"S","latitude":null,"longitude":null},{"idlocale":362,"idstate":36,"uf":"SC","state":"Santa Catarina","region":"S","latitude":null,"longitude":null},{"idlocale":363,"idstate":51,"uf":"SE","state":"Sergipe","region":"NE","latitude":null,"longitude":null},{"idlocale":364,"idstate":34,"uf":"SP","state":"Su00e3o Paulo","region":"SE","latitude":null,"longitude":null},{"idlocale":365,"idstate":42,"uf":"TO","state":"Tocantins","region":"N","latitude":null,"longitude":null}]}

希望这是获取您要查找的数据的另一种方法？

然后，您可以使用普通请求获取数据。您只需要同样的内容。通常，添加一个接受者，用户访问并要求使用标题足以通过。