Scons 属性错误：'builtin_function_or_method'对象没有属性'dispatch'

Question

我有一个实例化对象的sconstruct脚本。此对象在内部调用一个方法而不是运行多处理模块。下面的例子

该对象在调用函数之前解pickle文件并将输入传递给多处理模块。

def run_scons(self,inpfile,outfile):
        # Unpickle input parameter
        fid=open(inpfile,'rb')
        input_data=pkls.load(fid)
        my_results=[]
        #run solver in loop
        for my_data in input_data:
            work_ers=len(my_data)
            pool = Pool(processes=work_ers)
            a_result=pool.map_async(my_solver, my_data)
            pool.close()
            pool.join()
            my_results.append(a_result.get())
        fid.close()
        fid_out=open(outfile,'wb+')
        pkls.dump(rot_full_results,fid_out)

当通过scons执行相同的函数时，我得到以下错误:

pool = Pool(processes=work_ers)
  File "C:Python27libmultiprocessing__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "C:Python27libmultiprocessingpool.py", line 138, in __init__
    self._setup_queues()
  File "C:Python27libmultiprocessingpool.py", line 232, in _setup_queues
    from .queues import SimpleQueue
  File "C:Python27libmultiprocessingqueues.py", line 48, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "C:Python27libmultiprocessingsynchronize.py", line 48, in <module>
    from multiprocessing.forking import assert_spawning, Popen
  File "C:Python27libmultiprocessingforking.py", line 60, in <module>
    class ForkingPickler(Pickler):
  File "C:Python27libmultiprocessingforking.py", line 61, in ForkingPickler
    dispatch = Pickler.dispatch.copy()
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'
scons: building terminated because of errors.

在阅读了这个错误之后，我发现SCONS有一个hack，它将pickle模块重命名为cPickle，而多处理模块正在寻找cPickle，一切都下降了。还有别的办法吗?

Answer 1

我正在使用多进程并行处理一些"作业"文件。我在我的SConstruct文件中做了这样的事情:

if GetOption("run_jobs"):
    my_jobs = # some code that produces a list of job objects
    ecode = build_support.mp_run_jobs(my_jobs)
    Exit(ecode)

但是我遇到了上面的异常:

AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'

在Linux上使用' scon_horrible_regression_test_hack '工作:

$ SCONS_HORRIBLE_REGRESSION_TEST_HACK=1 scons --run-jobs

但是我不想在Linux, Windows, Mac等等上导出那个符号

我确实找到了一个解决方案，删除pickle模块并像这样重新导入它们:

if GetOption("run_jobs"):
    # Workaround SCons.compat module renaming
    import imp
    del sys.modules['pickle']
    del sys.modules['cPickle']
    sys.modules['pickle'] = imp.load_module('pickle', *imp.find_module('pickle'))
    sys.modules['cPickle'] = imp.load_module('cPickle', *imp.find_module('cPickle'))
    import pickle
    import cPickle
    print "(pickle == cPickle) = ", (pickle == cPickle)
    my_jobs = # some code that produces a list of job objects
    ecode = build_support.mp_run_jobs(my_jobs)
    Exit(ecode)

现在我的多任务处理工作如预期!