Python 中小型进程的快速多处理

Question

我面临以下问题： 在一个更大的python类中，我想在多个进程中执行其中的一部分，以加快我的程序。正如您在我的最小示例中所看到的，我有这个函数 f(x(，我调用了 10 次。通过单个进程执行此操作大约需要 1 秒。对于多处理，我希望尽可能接近 100 毫秒。因此，我已经尝试了Pool.map和Pool.imap_unordered方法。如您所见，它们要快得多，但没有给我想要的结果。

你可以帮我吗？

import time
from multiprocessing import Pool

class TestClass(object):
def __init__(self):
# Single Process
self.result_list = []
self.single_process()
print(self.result_list)
# Multiprocess ordered
self.result_list = []
self.multiprocessing_ordered()
print(self.result_list)
# Multiprocess unordered
self.result_list = []
self.multiprocessing_unordered()
print(self.result_list)
def f(self, x):
time.sleep(0.1)
self.result_list.append(x**2)
def single_process(self):
# Single process
start_time = time.time()
for x in range(10):
self.f(x)
print("Time with a single process: {0:.1f}".format((time.time() - start_time)*1e3))
def multiprocessing_ordered(self):
start_time = time.time()
pool = Pool(10)
pool.map(self.f, list(range(10)))
pool.close()
print("Time with multiprocessing (ordered): {0:.1f}".format((time.time() - start_time)*1e3))
def multiprocessing_unordered(self):
start_time = time.time()
pool = Pool(10)
pool.imap_unordered(self.f, list(range(10)))
pool.close()
print("Time with multiprocessing (unordered): {0:.1f}".format((time.time() - start_time)*1e3))

if __name__ == '__main__':
test_object = TestClass()

结果：

Time with a single process: 1013.7 ms
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Time with multiprocessing (ordered): 280.3 ms
[]
Time with multiprocessing (unordered): 100.7 ms
[]

Answer 1

进程位于自己的内存空间中，因此self.result_list不会在父进程和子进程之间共享。

您知道有多种选择：

1. 使用管道或队列或任何其他技术，以便在子进程和主进程之间进行通信。

2. 由函数返回结果，并在主进程中检索它，如下所示：

import time
from multiprocessing import Pool

class TestClass(object):
def __init__(self):
# Single Process
self.result_list = []
self.single_process()
print(self.result_list)
# Multiprocess
self.result_list = []
self.multiprocessing()
print(self.result_list)
def f(self, x):
time.sleep(0.1)
return x**2
def single_process(self):
# Single process
start_time = time.time()
result = []
for x in range(10):
self.result_list.append(self.f(x))
print("Time with a single process: {0:.1f}".format((time.time() - start_time)*1e3))
def multiprocessing(self):
pool = Pool(10)
# Start calculation
start_time = time.time()
multiple_results = [pool.apply_async(self.f, (i,)) for i in range(10)]
# Receive answer
self.result_list = [res.get(timeout=1) for res in multiple_results]
# Evaluate result
print("Time with multiprocessing: {0:.1f}".format((time.time() - start_time)*1e3))
pool.close()

if __name__ == '__main__':
test_object = TestClass()

结果：

Time with a single process: 1002.0
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Time with multiprocessing: 102.8
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

3. 尝试线程模块或线程池。线程共享相同的内存空间，因此只需将其附加到列表中即可共享结果。使用 ThreadPool：

import time
from multiprocessing.pool import ThreadPool
class TestClass(object):
def __init__(self):
# Single Process
self.result_list = []
self.single_process()
print(self.result_list)
# Multiprocess ordered
self.result_list = []
self.multiprocessing_ordered()
print(self.result_list)
# Multiprocess unordered
self.result_list = []
self.multiprocessing_unordered()
print(self.result_list)
def f(self, x):
time.sleep(0.1)
self.result_list.append(x**2)
def single_process(self):
# Single process
start_time = time.time()
for x in range(10):
self.f(x)
print("Time with a single process: {0:.1f}".format((time.time() - start_time)*1e3))
def multiprocessing_ordered(self):
start_time = time.time()
pool = ThreadPool(10)
pool.map(self.f, list(range(10)))
pool.close()
print("Time with multiprocessing (ordered): {0:.1f}".format((time.time() - start_time)*1e3))
def multiprocessing_unordered(self):
start_time = time.time()
pool = ThreadPool(10)
[_ for _ in pool.imap_unordered(self.f, list(range(10)))]
pool.close()
print("Time with multiprocessing (unordered): {0:.1f}".format((time.time() - start_time)*1e3))

if __name__ == '__main__':
test_object = TestClass()

线程池的结果：

Time with a single process: 1002.0
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Time with multiprocessing (ordered): 116.1
[0, 4, 1, 25, 36, 9, 16, 49, 81, 64]
Time with multiprocessing (unordered): 109.4
[0, 1, 4, 16, 25, 36, 9, 49, 81, 64]