Python 抓取器 - 如果目标是 404'd 则套接字错误会中断脚本

Question

在构建webscraper以将数据和输出编译为XLS格式时遇到错误；当再次测试我希望从中抓取的域列表时，程序在收到套接字错误时出错。希望找到一个"if"语句，该语句将对损坏的网站进行null解析，并继续执行我的while循环。有什么想法吗？

workingList = xlrd.open_workbook(listSelection)
workingSheet = workingList.sheet_by_index(0)
destinationList = xlwt.Workbook()
destinationSheet = destinationList.add_sheet('Gathered')
startX = 1
startY = 0
while startX != 21:
    workingCell = workingSheet.cell(startX,startY).value
    print ''
    print ''
    print ''
    print workingCell
    #Setup
    preSite = 'http://www.'+workingCell
    theSite = urlopen(preSite).read()
    currentSite = BeautifulSoup(theSite)
    destinationSheet.write(startX,0,workingCell)

这是一个错误：

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    homeMenu()
  File "C:Python27farming.py", line 31, in homeMenu
    openList()
  File "C:Python27farming.py", line 79, in openList
    openList()
  File "C:Python27farming.py", line 83, in openList
    openList()
  File "C:Python27farming.py", line 86, in openList
    homeMenu()
  File "C:Python27farming.py", line 34, in homeMenu
    startScrape()
  File "C:Python27farming.py", line 112, in startScrape
    theSite = urlopen(preSite).read()
  File "C:Python27liburllib.py", line 84, in urlopen
    return opener.open(url)
  File "C:Python27liburllib.py", line 205, in open
    return getattr(self, name)(url)
  File "C:Python27liburllib.py", line 342, in open_http
    h.endheaders(data)
  File "C:Python27libhttplib.py", line 951, in endheaders
    self._send_output(message_body)
  File "C:Python27libhttplib.py", line 811, in _send_output
    self.send(msg)
  File "C:Python27libhttplib.py", line 773, in send
    self.connect()
  File "C:Python27libhttplib.py", line 754, in connect
    self.timeout, self.source_address)
  File "C:Python27libsocket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 11004] getaddrinfo failed

Answer 1

Ummm，这看起来像是我在互联网连接断开时遇到的错误。HTTP 404错误是指当您确实有连接但找不到指定的URL时会出现的错误。

没有if语句来处理异常；您需要使用try/except构造来"捕获"它们。

更新：下面是一个演示：

import urllib
def getconn(url):
    try:
        conn = urllib.urlopen(url)
        return conn, None
    except IOError as e:
        return None, e
urls = """
    qwerty
    http://www.foo.bar.net
    http://www.google.com
    http://www.google.com/nonesuch
    """
for url in urls.split():
    print
    print url
    conn, exc = getconn(url)
    if conn:
        print "connected; HTTP response is", conn.getcode()
    else:
        print "failed"
        print exc.__class__.__name__
        print str(exc)
        print exc.args

输出：

qwerty
failed
IOError
[Errno 2] The system cannot find the file specified: 'qwerty'
(2, 'The system cannot find the file specified')
http://www.foo.bar.net
failed
IOError
[Errno socket error] [Errno 11004] getaddrinfo failed
('socket error', gaierror(11004, 'getaddrinfo failed'))
http://www.google.com
connected; HTTP response is 200
http://www.google.com/nonesuch
connected; HTTP response is 404

请注意，到目前为止，我们刚刚打开了连接。现在您需要做的是检查HTTP响应代码，并决定是否有任何值得使用conn.read() 检索的内容