Python多处理.完成所有任务后，Pool未能停止

Question

我已经实现了一个这样的解析器，

import multiprocessing
import time
def foo(i):
    try:
        # some codes
    except Exception, e:
        print e
def worker(i):
    foo(i)
    time.sleep(i)
    return i

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=4)
    result = pool.map_async(worker, range(15))
    while not result.ready():
        print("num left: {}".format(result._number_left))
        time.sleep(1)
    real_result = result.get()
    pool.close()
    pool.join()

我的解析器实际上完成了所有的过程，但结果是不可用的，即，它仍然在while循环和打印num left : 2。我该如何阻止这一切?我不想要变量real_result的值。

我正在运行Ubuntu 14.04, python 2.7

我的代码的相应部分看起来像，

    async_args = ((date, kw_dict) for date in dates)
    pool = Pool(processes=4)
    no_rec = []
    def check_for_exit(msg):
        print msg
        if last_date in msg:
            print 'Terminating the pool'
            pool.terminate()
    try:
        result = pool.map_async(parse_date_range, async_args)
        while not result.ready():
            print("num left: {}".format(result._number_left))
            sleep(1)
        real_result = result.get(5)
        passed_dates = []
        for x, y in real_result:
            passed_dates.append(x)
            if y:
                no_rec.append(y[0])
        # if last_date in passed_dates:
        #     print 'Terminating the pool'
        #     pool.terminate()
        pool.close()
    except:
        print 'Pool error'
        pool.terminate()
        print traceback.format_exc()
    finally:
        pool.join()

Answer 1

我打赌你有故障的parse_date_range，导致工作进程终止而不产生任何结果或py异常。可能libc的exit是由C模块/库调用的，因为一个非常严重的错误。
以下代码重现了您所看到的无限循环:

import sys
import multiprocessing
import time
def parse_date_range(i):
    if i == 5:
        sys.exit(1) # or raise SystemExit; 
                    # other exceptions are handled by the pool
    time.sleep(i/19.)
    return i

if __name__ == "__main__":
    pool = multiprocessing.Pool(4)
    result = pool.map_async(parse_date_range, range(15))
    while not result.ready():
        print("num left: {}".format(result._number_left))
        time.sleep(1)
    real_result = result.get()
    pool.close()
    pool.join()