我测试了map、mp.dummy.Pool.map和mp.Pool.map 的性能
import itertools
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
import numpy as np
# wrapper function
def wrap(args): return args[0](*args[1:])
# make data arrays
x = np.random.rand(30, 100000)
y = np.random.rand(30, 100000)
# map
%timeit -n10 map(wrap, itertools.izip(itertools.repeat(np.correlate), x, y))
# mp.dummy.Pool.map
for i in range(2, 16, 2):
print 'Thread Pool ', i, ' : ',
t = ThreadPool(i)
%timeit -n10 t.map(wrap, itertools.izip(itertools.repeat(np.correlate), x, y))
t.close()
t.join()
# mp.Pool.map
for i in range(2, 16, 2):
print 'Process Pool ', i, ' : ',
p = mp.Pool(i)
%timeit -n10 p.map(wrap, itertools.izip(itertools.repeat(np.correlate), x, y))
p.close()
p.join()
输出
# in this case, one CPU core usage reaches 100%
10 loops, best of 3: 3.16 ms per loop
# in this case, all CPU core usages reach ~80%
Thread Pool 2 : 10 loops, best of 3: 4.03 ms per loop
Thread Pool 4 : 10 loops, best of 3: 3.3 ms per loop
Thread Pool 6 : 10 loops, best of 3: 3.16 ms per loop
Thread Pool 8 : 10 loops, best of 3: 4.48 ms per loop
Thread Pool 10 : 10 loops, best of 3: 4.19 ms per loop
Thread Pool 12 : 10 loops, best of 3: 4.03 ms per loop
Thread Pool 14 : 10 loops, best of 3: 4.61 ms per loop
# in this case, all CPU core usages reach 80-100%
Process Pool 2 : 10 loops, best of 3: 71.7 ms per loop
Process Pool 4 : 10 loops, best of 3: 128 ms per loop
Process Pool 6 : 10 loops, best of 3: 165 ms per loop
Process Pool 8 : 10 loops, best of 3: 145 ms per loop
Process Pool 10 : 10 loops, best of 3: 259 ms per loop
Process Pool 12 : 10 loops, best of 3: 176 ms per loop
Process Pool 14 : 10 loops, best of 3: 176 ms per loop
多线程确实可以提高速度。这是可以接受的，因为锁。
多进程大大降低了速度，这令人惊讶。我有八个3.78MHz的CPU，每个CPU有4个内核。
如果将x和y的形状改为(300, 10000)，即大10倍，则可以看到类似的结果。
但对于(20, 1000)、等小型阵列
10 loops, best of 3: 28.9 µs per loop
Thread Pool 2 : 10 loops, best of 3: 429 µs per loop
Thread Pool 4 : 10 loops, best of 3: 632 µs per loop
...
Process Pool 2 : 10 loops, best of 3: 525 µs per loop
Process Pool 4 : 10 loops, best of 3: 660 µs per loop
...
- 多处理和多线程具有相似的性能
- 单个过程要快得多。（由于多处理和多线程的开销？）
无论如何，即使在执行这样一个简单的函数时，多处理的性能如此糟糕也真的出乎意料。这怎么可能发生
根据@TrevorMerrifield的建议，我修改了代码以避免将大数组传递给wrap。
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
import numpy as np
n = 30
m = 1000
# make data in wrap
def wrap(i):
x = np.random.rand(m)
y = np.random.rand(m)
return np.correlate(x, y)
# map
print 'Single process :',
%timeit -n10 map(wrap, range(n))
# mp.dummy.Pool.map
print '---'
print 'Thread Pool %2d : '%(4),
t = ThreadPool(4)
%timeit -n10 t.map(wrap, range(n))
t.close()
t.join()
print '---'
# mp.Pool.map, function must be defined before making Pool
print 'Process Pool %2d : '%(4),
p = Pool(4)
%timeit -n10 p.map(wrap, range(n))
p.close()
p.join()
输出
Single process :10 loops, best of 3: 688 µs per loop
---
Thread Pool 4 : 10 loops, best of 3: 1.67 ms per loop
---
Process Pool 4 : 10 loops, best of 3: 854 µs per loop
- 没有任何改进
我尝试了另一种方法，将一个标记传递给wrap，以从全局数组x和y获取数据。
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
import numpy as np
# make data arrays
n = 30
m = 10000
x = np.random.rand(n, m)
y = np.random.rand(n, m)
def wrap(i): return np.correlate(x[i], y[i])
# map
print 'Single process :',
%timeit -n10 map(wrap, range(n))
# mp.dummy.Pool.map
print '---'
print 'Thread Pool %2d : '%(4),
t = ThreadPool(4)
%timeit -n10 t.map(wrap, range(n))
t.close()
t.join()
print '---'
# mp.Pool.map, function must be defined before making Pool
print 'Process Pool %2d : '%(4),
p = Pool(4)
%timeit -n10 p.map(wrap, range(n))
p.close()
p.join()
输出
Single process :10 loops, best of 3: 133 µs per loop
---
Thread Pool 4 : 10 loops, best of 3: 2.23 ms per loop
---
Process Pool 4 : 10 loops, best of 3: 10.4 ms per loop
- 太糟糕了
我尝试了另一个简单的例子（不同的wrap）。
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
# make data arrays
n = 30
m = 10000
# No big arrays passed to wrap
def wrap(i): return sum(range(i, i+m))
# map
print 'Single process :',
%timeit -n10 map(wrap, range(n))
# mp.dummy.Pool.map
print '---'
i = 4
print 'Thread Pool %2d : '%(i),
t = ThreadPool(i)
%timeit -n10 t.map(wrap, range(n))
t.close()
t.join()
print '---'
# mp.Pool.map, function must be defined before making Pool
print 'Process Pool %2d : '%(i),
p = Pool(i)
%timeit -n10 p.map(wrap, range(n))
p.close()
p.join()
激励：
10 loops, best of 3: 4.28 ms per loop
---
Thread Pool 4 : 10 loops, best of 3: 5.8 ms per loop
---
Process Pool 4 : 10 loops, best of 3: 2.06 ms per loop
- 现在
multiprocessing更快了
但如果m变为10倍大（即100000），
Single process :10 loops, best of 3: 48.2 ms per loop
---
Thread Pool 4 : 10 loops, best of 3: 61.4 ms per loop
---
Process Pool 4 : 10 loops, best of 3: 43.3 ms per loop
- 同样，没有任何改进
您正在将wrap映射到(a, b, c)，其中a是一个函数，b和c是100K元素向量。所有这些数据在发送到池中选定的进程时都会被pickle，然后在到达时被取消pickle。这是为了确保进程对数据具有互斥访问权限。
您的问题是，酸洗过程比关联过程更昂贵。作为指导原则，您希望最大限度地减少在进程之间发送的信息量，并最大限度地增加每个进程所做的工作量，同时仍然分布在系统上的#个核心中。
如何做到这一点取决于你试图解决的实际问题。通过调整你的玩具示例，使你的向量更大一点（100万个元素），并在wrap函数中随机生成，通过使用一个包含4个元素的进程池，我可以在单核上获得2倍的加速。代码如下：
def wrap(a):
x = np.random.rand(1000000)
y = np.random.rand(1000000)
return np.correlate(x, y)
p = Pool(4)
p.map(wrap, range(30))