我正试图在Scrapy上用硒刮一个网站。我用selenium更改了Scrapy响应URL，但当我试图返回start_urls时，函数如下：
Spider.py(start_urls(：
@property
def start_urls(self):
url = 'https://www.adana.bel.tr/home/hal_listesi' #The URL that script will scrape
opts = Options() #Set options for headless and user-agent etc.
#opts.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36")
opts.add_argument('headless')#It helps to work without opening browser
driver = webdriver.Chrome(options=opts,executable_path="chromedriver.exe")
driver.get(url) #URL starts
self.day = int(driver.find_element_by_xpath("/html/body/div/div[3]/div/div[2]/main/div/div/div/div/div/div/div[1]/div[1]/span[1]").text)
self.month = months[driver.find_element_by_xpath("/html/body/div/div[3]/div/div[2]/main/div/div/div/div/div/div/div[1]/div[1]/span[2]").text]
self.year = int(driver.find_element_by_xpath("/html/body/div/div[3]/div/div[2]/main/div/div/div/div/div/div/div[1]/div[1]/span[3]").text)
product_list =driver.find_element_by_xpath("/html/body/div/div[3]/div/div[2]/main/div/div/div/div/div/div/div[1]/div[3]/a/img")
product_list.click()
new_url = driver.current_url
driver.quit
return [new_url]
我在利用自我。日期，因为我必须从第一页中获取日期。
它开始返回3次，并给我以下错误3次。这花了太长时间，我不明白为什么它总是给我错误。
输出：
DevTools listening on ws://127.0.0.1:50639/devtools/browser/cd5830e4-5a11-4f28-a12d-cb605e96075d
[1103/153027.438:INFO:CONSOLE(54)] "Mixed Content: The page at 'https://www.adana.bel.tr/home/hal_listesi' was loaded over HTTPS, but requested an insecure stylesheet 'http://netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css'. This request has been blocked; the content must be served over HTTPS.", source: https://www.adana.bel.tr/home/hal_listesi (54)
[1103/153032.344:INFO:CONSOLE(54)] "Mixed Content: The page at 'https://www.adana.bel.tr/hal-detay/396' was loaded over HTTPS, but requested an insecure stylesheet 'http://netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css'. This request has been blocked; the content must
be served over HTTPS.", source: https://www.adana.bel.tr/hal-detay/396 (54)
[1103/153032.391:INFO:CONSOLE(1520)] "Uncaught ReferenceError: $ is not defined", source: https://www.adana.bel.tr/hal-detay/396 (1520)
DevTools listening on ws://127.0.0.1:50673/devtools/browser/e664e7e2-1c13-4128-bb20-a3df6437d2c7
[1103/153035.939:INFO:CONSOLE(54)] "Mixed Content: The page at 'https://www.adana.bel.tr/home/hal_listesi' was loaded over HTTPS, but requested an insecure stylesheet 'http://netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css'. This request has been blocked; the content must be served over HTTPS.", source: https://www.adana.bel.tr/home/hal_listesi (54)
[1103/153038.668:INFO:CONSOLE(54)] "Mixed Content: The page at 'https://www.adana.bel.tr/hal-detay/396' was loaded over HTTPS, but requested an insecure stylesheet 'http://netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css'. This request has been blocked; the content must
be served over HTTPS.", source: https://www.adana.bel.tr/hal-detay/396 (54)
[1103/153038.710:INFO:CONSOLE(1520)] "Uncaught ReferenceError: $ is not defined", source: https://www.adana.bel.tr/hal-detay/396 (1520)
DevTools listening on ws://127.0.0.1:50707/devtools/browser/5fcb91e4-a076-4aa2-9173-7fd3565f741f
[1103/153042.020:INFO:CONSOLE(54)] "Mixed Content: The page at 'https://www.adana.bel.tr/home/hal_listesi' was loaded over HTTPS, but requested an insecure stylesheet 'http://netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css'. This request has been blocked; the content must be served over HTTPS.", source: https://www.adana.bel.tr/home/hal_listesi (54)
[1103/153045.407:INFO:CONSOLE(54)] "Mixed Content: The page at 'https://www.adana.bel.tr/hal-detay/396' was loaded over HTTPS, but requested an insecure stylesheet 'http://netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css'. This request has been blocked; the content must
be served over HTTPS.", source: https://www.adana.bel.tr/hal-detay/396 (54)
[1103/153045.459:INFO:CONSOLE(1520)] "Uncaught ReferenceError: $ is not defined", source: https://www.adana.bel.tr/hal-detay/396 (1520)
设置.py:
BOT_NAME = 'first_bot'
SPIDER_MODULES = ['first_bot.spiders']
NEWSPIDER_MODULE = 'first_bot.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36"
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 3
ITEM_PIPELINES = {
'first_bot.pipelines.FirstBotPipeline': 300,
}
那么我该如何解决这个问题呢？它大约需要30秒，对于一个URL来说太长了。
我用解析请求更改起始URL：
def parse(self, response):
now_date = datetime.today()-timedelta(days=1)
self.day = response.xpath("//*[@class='day']/text()").extract()
self.month = response.xpath("//*[@class='month']/text()").extract()
self.year = response.xpath("//*[@class='year']/text()").extract()
count = 0
for check in self.day:
if now_date.day == int(check):
url =response.xpath("//*[@class='indir']/a/@href").extract()[count]
self.curt_day = self.day[count]
self.curt_month = self.month[count]
self.curt_year = self.year[count]
count +=1
absolute_url = response.urljoin(url)
request = scrapy.Request(
absolute_url, callback=self.parse_contractors)
yield request