Scrapy-通过谷歌服务认证

Question

我是一个完全的Python新手，但我需要在谷歌分析中抓取一个特定的页面，谷歌的登录过程分为两个页面，我不知道如何使用scrapy的FormRequest。

我尝试登录gmail作为测试，代码如下：

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import FormRequest, Request
class LoginSpider(BaseSpider):
name = 'super'
start_urls = ['https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=fr#identifier']
def parse(self, response):
    return [FormRequest.from_response(response,
                formdata={'Email': '[e-mail]', 'Passwd': 'password example'},
                callback=self.after_login)]
def after_login(self, response):
  if "authentication failed" in response.body:
    self.log("Login failed", level=log.ERROR)
    return
# We've successfully authenticated, let's have some fun!
  else:
    return Request(url="https://mail.google.com/mail/u/0/#inbox",
           callback=self.parse_tastypage)

def parse_tastypage(self, response):
  sel = Selector(response)
  item = Item()
  item ["Test"] = sel.xpath("//h1/text()").extract()
  yield item

但它没有工作，这是我的日志文件：

2016-03-27 10:30:19 [scrapy] INFO: Spider opened
2016-03-27 10:30:19 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 
items (at 0 items/min)
2016-03-27 10:30:19 [scrapy] DEBUG: Telnet console listening on 148.0.0.1:6023
2016-03-27 10:30:19 [scrapy] DEBUG: Crawled (200) <GET https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=fr#identifier> (referer: None)
2016-03-27 10:30:24 [scrapy] DEBUG: Crawled (200) <POST https://accounts.google.com/AccountLoginInfo> (referer: https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=fr)
2016-03-27 10:30:25 [scrapy] DEBUG: Redirecting (302) to <GET https://accounts.google.com/ServiceLogin?service=mail&passive=true&rm=false&continue=https://mail.google.com/mail/&ss=1&scc=1&ltmpl=default&ltmplcache=2&emr=1&osid=1> from <GET https://mail.google.com/mail/u/0/#inbox>
2016-03-27 10:30:30 [scrapy] DEBUG: Crawled (200) <GET https://accounts.google.com/ServiceLogin?service=mail&passive=true&rm=false&continue=https://mail.google.com/mail/&ss=1&scc=1&ltmpl=default&ltmplcache=2&emr=1&osid=1> (referer: https://accounts.google.com/AccountLoginInfo)
2016-03-27 10:30:30 [scrapy] ERROR: Spider error processing <GET https://accounts.google.com/ServiceLogin?service=mail&passive=true&rm=false&continue=https://mail.google.com/mail/&ss=1&scc=1&ltmpl=default&ltmplcache=2&emr=1&osid=1> (referer: https://accounts.google.com/AccountLoginInfo)
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/Scrapy-1.1.0rc3-py2.7.egg/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/Library/Python/2.7/site-packages/Scrapy-1.1.0rc3-py2.7.egg/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/Library/Python/2.7/site-packages/Scrapy-1.1.0rc3-py2.7.egg/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/Library/Python/2.7/site-packages/Scrapy-1.1.0rc3-py2.7.egg/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Library/Python/2.7/site-packages/Scrapy-1.1.0rc3-py2.7.egg/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Users/machine/super/super/spiders/mySuper.py", line 26, in parse_tastypage
    sel = Selector(response)
NameError: global name 'Selector' is not defined
2016-03-27 10:30:30 [scrapy] INFO: Closing spider (finished)
2016-03-27 10:30:30 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1874,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 3,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 197446,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 3,
 'downloader/response_status_count/302': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 3, 27, 14, 30, 30, 741077),
 'log_count/DEBUG': 5,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'log_count/WARNING': 1,
 'request_depth_max': 2,
 'response_received_count': 3,
 'scheduler/dequeued': 4,
 'scheduler/dequeued/memory': 4,
 'scheduler/enqueued': 4,
 'scheduler/enqueued/memory': 4,
 'spider_exceptions/NameError': 1,
 'start_time': datetime.datetime(2016, 3, 27, 14, 30, 19, 82107)}
2016-03-27 10:30:30 [scrapy] INFO: Spider closed (finished)

我该怎么做？

感谢

Answer 1

您缺少导入，因此您的代码出现错误，您既没有导入Selector也没有导入Item:

from scrapy.selector import Selector
from scrapy import Item

如果你看一下输出，你可以清楚地看到：

 NameError: global name 'Selector' is not defined

如果您不同时导入Item，您将看到完全相同的情况。

您也有level=log.ERROR，其中日志没有在任何地方定义或导入，您可能想要日志记录。错误所以您需要导入日志记录：

 import logging

 level=logging.ERROR

这都是非常基本的东西，我建议你在尝试接近类或任何类型的复杂代码之前先看一下教程，如果你不太了解基本知识，那么生活会比需要的更困难。

这是一个有效的实现：

from scrapy.spider import BaseSpider
from scrapy.selector import  Selector
from scrapy.http import FormRequest, Request
import  logging
from scrapy import Field
from scrapy import Item

class Product(Item):
    Test =  Field()

class LoginSpider(BaseSpider):
    name = 'super'
    start_urls = ['https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=fr#identifier']
    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'Email': '[e-mail]', 'Passwd': 'pass'},
                    callback=self.after_login)]
    def after_login(self, response):
      if "authentication failed" in response.body:
        self.log("Login failed", level=logging.ERROR)
        return
    # We've successfully authenticated, let's have some fun!
    print("Login Successful!!")
    return Request(url="https://mail.google.com/mail/u/0/#inbox",
               callback=self.parse_tastypage)

    def parse_tastypage(self, response):
      item = Product()
      item ["Test"] = response.xpath("//h1/text()").extract()
      yield item

使用我的登录名运行的示例：

 DEBUG: Crawled (200) <POST https://accounts.google.com/AccountLoginInfo> (referer: https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=fr)
Login Successful!!
2016-03-28 02:13:27 [scrapy] DEBUG: Redirecting (302) to <GET https://accounts.google.com/ServiceLogin?service=mail&passive=true&rm=false&continue=https://mail.google.com/mail/&ss=1&scc=1&ltmpl=default&ltmplcache=2&emr=1&osid=1> from <GET https://mail.google.com/mail/u/0/#inbox>

我还没有验证它是否真的有效，我只是使用了你自己的逻辑，不管怎样，这是一个如何使用scrapy的工作示例。