具有多个函数的python全局变量错误

Question

这里有一个使用全局变量的示例代码，它会给我带来错误。全局变量x在调用test2函数之前在test3函数中声明，但test2函数似乎没有得到全局变量x的定义

from multiprocessing import Pool
import numpy as np
global x    
def test1(w, y):
return w+y    
def test2(v):
global x        # x is assigned value in test3 before test2 is called
return test1(x, v)    
def test3():
global x
x = 2
y = np.random.random(10)
with Pool(processes=6) as p:
z = p.map(test2, y)
print(z)
if __name__ == '__main__':
test3()

错误为：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
File "C:WinPython-64bit-3.5.2.1Qt5python-3.5.2.amd64libmultiprocessingpool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:WinPython-64bit-3.5.2.1Qt5python-3.5.2.amd64libmultiprocessingpool.py", line 44, in mapstar
return list(map(*args))
File "...my_global_variable_testcode.py", line 23, in test2
return test1(x, v)
NameError: name 'x' is not defined
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...my_global_variable_testcode.py", line 35, in <module>
test3()
File "...my_global_variable_testcode.py", line 31, in test3
z = p.map(test2, y)
File "C:WinPython-64bit-3.5.2.1Qt5python-3.5.2.amd64libmultiprocessingpool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:WinPython-64bit-3.5.2.1Qt5python-3.5.2.amd64libmultiprocessingpool.py", line 608, in get
raise self._value
NameError: name 'x' is not defined

我已经看了很多关于SO的问题和答案，但仍然无法找出如何修复此代码。如果有人能指出代码的问题，我将不胜感激？

有人能告诉我如何在不改变代码基本结构的情况下重写上面的代码吗？(即，保留test1、test2、test3作为3个单独的函数，因为在我的原始代码中，这些函数相当长且复杂)，这样我就可以实现多处理的目标？

p.s.这个示例代码只是我实际代码的简化版本，我在这里给出这个简化版本是为了弄清楚如何使全局变量工作(而不是试图为2+np.random.random(10)找到一种复杂的方法)。

*编辑*-边界描述

这笔奖金是给那些可以帮助我重写这段代码的人的，保留代码中函数的基本结构：

(i)test1对test2进行多处理调用，test2依次调用test3

(ii)利用全局变量或多处理模块的Manager类或其他任何东西来避免test1将公共变量传递给test2

(iii)在调用多处理代码之前，test1还提供一些值或对全局变量/公共数据进行更改

(iv)代码应该在Windows上运行(就像我使用Windows一样)。目前还没有找到在Linux/OSX上运行的解决方案。

为了帮助获得奖金，让我给出两个不同的测试用例。

*案例1-非多处理版本*

import numpy as np
x = 3
def test1(w, y):
return w+y
def test2(v):
global x
print('x in test2 = ', x)
return test1(x, v)
def test3():
global x
x = 2
print('x in test3 = ', x)
y = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
z = test2(y)
print(z)
if __name__ == '__main__':
test3()

输出(正确)为：

x in test3 =  2
x in test2 =  2
[ 3  4  5  6  7  8  9 10 11 12]

*案例2-多处理版本*

from multiprocessing import Pool
import numpy as np
x = 3
def test1(w, y):
return w+y
def test2(v):
global x
print('x in test2 = ', x)
return test1(x, v)
def test3():
global x
x = 2
print('x in test3 = ', x)
y = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
with Pool(processes=6) as p:
z = p.map(test2, y)
print(z)
if __name__ == '__main__':
test3()

输出(不正确)为

x in test3 =  2
x in test2 =  3
x in test2 =  3
x in test2 =  3
x in test2 =  3
x in test2 =  3
x in test2 =  3
x in test2 =  3
x in test2 =  3
x in test2 =  3
x in test2 =  3
[4, 5, 6, 7, 8, 9, 10, 11, 12, 13]

Answer 1

您的问题是在Process中共享一个变量，而不是在Multiprocess池中。当您使用global x时，它可以在单个流程中工作，但不能跨多个流程工作。在这种情况下，您需要使用multiprocessing中的Value。以下是在多处理中工作的更新代码

from multiprocessing import Pool, Value
import numpy as np
xVal = Value('i', 0)
def test1(w, y):
return w+y
def test2(v):
x = xVal.value
print('x in test2 = ', x)
return test1(x, v)
def test3():
xVal.value = 2
print('x in test3 = ', xVal.value)
y = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
with Pool(processes=6) as p:
z = p.map(test2, y)
print(z)
if __name__ == '__main__':
test3()

程序的输出如下

x in test3 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
x in test2 =  2
[3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

第2版

下面的程序应该在Windows上运行也

from multiprocessing import Pool, Value, Manager, Array
import multiprocessing
import numpy as np
xVal = None
def sharedata(sharedData):
global xVal
xVal = sharedData
def test1(w, y):
return w+y
def test2(v):
global xVal
x = xVal.value
print('x in test2 = ', x)
return test1(x, v)

def test3():
xVal.value = 2
print('x in test3 = ', xVal.value)
y = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
with Pool(processes=6, initializer=sharedata,initargs=(xVal,)) as p:
z = p.map(test2, y)
print('x in test3 = ', xVal.value)
print(z)
if __name__ == '__main__':
xVal = Value('i', 0)
test3()

Answer 2

您必须在函数之外定义变量x，例如，不要定义全局x，比如x=0或任何您喜欢的东西，并在函数中使用全局声明，就像您现在所做的那样。希望能帮助