为什么我不能将带有Scrapy或Selenium的cookie发送到网站？

Question

首先，如果我的问题有一个非常明显的解决方案，请原谅我。我是网页抓取和 Scrapy 的新手。这将是我第三个被废弃的网站(如果我能找到以下问题的解决方案(。

我想要实现的目标：

是从以下网站抓取产品数据：https://www.sanalmarket.com.tr/kweb/sclist/30011-tum-meyveler

但是，产品会根据您登录后选择的城区动态加载。

所以我想，也许我可以用自己的帐户登录，从请求标头中获取cookie，然后用抓取的请求发送它们。我想问题是，网站不接受我发送的cookie。

我也用硒尝试了同样的程序。

1. 打开页面

2. 已登录

3. 选择了城市

4. 得到了饼干(也用泡菜保存它们，以后在刮擦上使用，但没有工作(

5. 从网站中删除所有饼干

6. 刷新页面后发送步骤 4 中的 cookie

网站再次不接受饼干。

注意：由于我需要每天抓取网站中的所有类别，因此我需要像刮擦这样的快速抓取解决方案。所以用硒刮对我来说不是一个选择。

这里有一些日志和屏幕截图来支持我的问题。

请求网址和方法

请求标头和 Cookie 信息

登录并选择城市后的数据预览(请注意"sid：1885"，这是我要抓取的商店ID(

这是来自刮擦的视图(响应(行的输出

scrapy shell https://www.sanalmarket.com.tr/kweb/sclist/30011-tum-meyveler
from scrapy import Request
mycookie = {'JSESSIONID ': 'yndMqXswzQYeUw1CsLtp9A0GBI7ZZE0yI1W0zPk4u4JJxpZES8RF!-1577658491 ', 'NSC_wjq_dt_iuuq_lbohvsvn_lxfc    ': '756ca3c16479c6cdde0681fa2edb1040d4786c1c0a6b2f3116d5fc7f605b4631d4d0f199 ','_dc_gtm_UA-1547459-1  ':'1','_ga':'GA1.3.219867582.1525198968','_gat_UA-1547459-1 ':'1','_gid':'GA1.3.1499846526.1525198968','current-currency    ':'TRY','customer':'ggB2MTVRWi76tWJwj2ZvbDa896G27N3YaH','district':'ac00a4001701ce63cc30626def','first-permission-impression    ':'1','ins-gaSSId   ':'cbf3cd92-3c71-e321-30ac-b2d89dbf3826_1525528747  ','insIsUserLoggedIn    ':'1','insTotalCartAmount187    ':'194.96   ','insUserDetails   ':'%22muharrem.akkaya96%40gmail.com%22  ','insdrSV':'285','scs':'%7B%22t%22%3A1%7D  ','spUID':'15251989688268402d4dc11.7edd9701 ','total-cart-amount    ':'120.78   '}
req = Request('https://www.sanalmarket.com.tr/kweb/getProductList.do?shopCategoryId=30011',cookies = mycookie)
fetch(req)
view(response)

第一行的日志

2018-05-05 19:11:02 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: seleniumcrawler)
2018-05-05 19:11:03 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:19:30) [MSC v.1500 32 bit (Intel)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Windows-10-10.0.16299
2018-05-05 19:11:03 [scrapy.crawler] INFO: Overridden settings: {'COOKIES_DEBUG': True, 'NEWSPIDER_MODULE': 'seleniumcrawler.spiders', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'SPIDER_MODULES': ['seleniumcrawler.spiders'], 'BOT_NAME': 'seleniumcrawler', 'LOGSTATS_INTERVAL': 0, 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36', 'FEED_EXPORT_ENCODING': 'utf-8'}
2018-05-05 19:11:03 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2018-05-05 19:11:03 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'seleniumcrawler.middlewares.seleniumcrawlerDownloaderMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-05-05 19:11:03 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-05-05 19:11:03 [scrapy.middleware] INFO: Enabled item pipelines:
['seleniumcrawler.pipelines.JsonPipeline',
'seleniumcrawler.pipelines.CsvPipeline']
2018-05-05 19:11:03 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-05-05 19:11:03 [scrapy.core.engine] INFO: Spider opened
2018-05-05 19:11:03 [migros] INFO: Spider opened: migros
2018-05-05 19:11:04 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 https://www.sanalmarket.com.tr/kweb/sclist/30011-tum-meyveler>
Set-Cookie: JSESSIONID=cMTfOnFTK1dPSPF2Qdi0d1EqqCXP3HW0S00BwxOwljYjaOMcAOqE!1083904106; path=/; HttpOnly
Set-Cookie: NSC_wjq_dt_iuuq_lbohvsvn_lxfc=0933a3df2cf252c6b4bd9a5784157b04f2a0c6e4b29bff73d54a79d474fdc48e85bdc9ec;path=/;secure;httponly

其余行的日志

2018-05-05 19:19:32 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://www.sanalmarket.com.tr/kweb/getProductList.do?shopCategoryId=30011>
Cookie: customer=ggB2MTVRWi76tWJwj2ZvbDa896G27N3YaH; insIsUserLoggedIn=1; insUserDetails=%22muharrem.akkaya96%40gmail.com%22; district=ac00a4001701ce63cc30626def; spUID=15251989688268402d4dc11.7edd9701; ins-gaSSId=cbf3cd92-3c71-e321-30ac-b2d89dbf3826_1525528747; insTotalCartAmount187=194.96; _ga=GA1.3.219867582.1525198968; JSESSIONID=yndMqXswzQYeUw1CsLtp9A0GBI7ZZE0yI1W0zPk4u4JJxpZES8RF!-1577658491; current-currency=TRY; first-permission-impression=1; insdrSV=285; _gid=GA1.3.1499846526.1525198968; _gat_UA-1547459-1=1; total-cart-amount=120.78; _dc_gtm_UA-1547459-1=1; scs=%7B%22t%22%3A1%7D; NSC_wjq_dt_iuuq_lbohvsvn_lxfc=756ca3c16479c6cdde0681fa2edb1040d4786c1c0a6b2f3116d5fc7f605b4631d4d0f199; NSC_wjq_dt_iuuq_lbohvsvn_lxfc=0933a3df2cf252c6b4bd9a5784157b04f2a0c6e4b29bff73d54a79d474fdc48e85bdc9ec; JSESSIONID=cMTfOnFTK1dPSPF2Qdi0d1EqqCXP3HW0S00BwxOwljYjaOMcAOqE!1083904106
2018-05-05 19:19:32 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 https://www.sanalmarket.com.tr/kweb/getProductList.do?shopCategoryId=30011>
Set-Cookie: JSESSIONID=ZvvfjByrDdrOTwmJX7QaaU0jWBv5nxKgfXvPVfvwSsCn63bkGH3m!-1577658491; path=/; HttpOnly
2018-05-05 19:19:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sanalmarket.com.tr/kweb/getProductList.do?shopCategoryId=30011> (referer: None)

那么我该如何克服这种饼干情况呢？

Answer 1

Cookie 似乎通过您的 Scrapy 代码正确发送，据我所知，问题出在您的密钥JSESSIONID的 cookie 值上。

当我创建自己的会话，将我的城市设置为"AFYON-Akmescit"并获取该会话ID时，我按预期获得了AFYON-Akmescit的sid1885，但是当我尝试您的或任何其他损坏的会话ID(通过随机更改一个字符而损坏(时，我收到sid193。因此，不知何故，城市 ID193是默认的，它不接受您的JSESSIONID值，不接受 cookie 信息本身。

无论如何，作为回答您问题的另一个方面，您当然不应该在抓取时使用会话 ID 作为可靠的识别源，您可能还希望自动化身份验证过程。