尝试重新启动网络爬虫线程时连接被拒绝

Question

我正在使用DryScrape来抓取javascript页面，如果出现错误，偶尔它会杀死进程。我尝试根据文档使用捕获来防止它，但我还没有弄清楚：

        try:
            sess.visit('url'))
        except webkit_server.EndOfStreamError:
            continue
        except webkit_server.NoResponeerror:
            continue
        except webkit_server.InvalidResponseError:
            continue
        except webkit_server.NoX11Error:
            continue

所以我有这样的设置，如果线程崩溃，我会重新启动线程：类检查器（）： def check_if_thread_is_alive（个体经营）： a = 线程类（） a.start（）

    b = ThreadClass()
    b.start()
    c = ThreadClass()
    c.start()
    d = ThreadClass()
    d.start()
    while True:
        if not a.is_alive():
            print "Restarting A"
            a = ThreadClass()
            a.start()
        if not b.is_alive():
            print "Restarting B"
            b = ThreadClass()
            b.start()
        if not c.is_alive():
            print "Restarting C"
            c = ThreadClass()
            c.start()
        if not d.is_alive():
            print "Restarting D"
            d = ThreadClass()
            d.start()

但是，每当我尝试重新启动线程时，我最终都会收到错误：

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "Scrapper.py", line 30, in run
    sess = dryscrape.Session(base_url = 'url')
  File "/usr/local/lib/python2.7/dist-packages/dryscrape/session.py", line 18, in __init__
    self.driver = driver or DefaultDriver()
  File "/usr/local/lib/python2.7/dist-packages/dryscrape/driver/webkit.py", line 30, in __init__
    super(Driver, self).__init__(**kw)
  File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 225, in __init__
    self.conn = connection or ServerConnection()
  File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 444, in __init__
    self._sock = (server or get_default_server()).connect()
  File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 414, in connect
    sock.connect(("127.0.0.1", self._port))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

有没有更好的方法来尝试解决这个问题，或者我缺少什么？

Answer 1

Cos ：你正在尝试连接到自己。

need change target url.

如果要连接到自己，请先创建服务。

File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 414, in connect
    sock.connect(("127.0.0.1", self._port))
  File "/usr/lib/python2.7/socket.py", line 224, in meth <<<--- you're trying to connect to yourself.
    return getattr(self._sock,name)(*args)

Answer 2

如果要跳过异常，始终可以使用像这样的 catch-all 异常处理程序。这通常被认为是非常糟糕的做法，但如果错误只是偶尔发生，它确实可以让您的刮板保持运行：

try:
    sess.visit(url)
except Exception as e:
    # Print the exception for debugging here
    continue

您是否正在启动本地服务器进行测试？从回溯：

File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 414, in connect
sock.connect(("127.0.0.1", self._port))

您实际上是在连接到本地主机。如果启动自己的服务器，请检查服务器日志以了解它停止响应连接请求的原因。

---

刚刚在您的脚本中发现了另一个错误：

sess.visit('url')
# it should be something like:
url = "http://www.google.com/"
sess.visit(url)