我有一些(只读)数据(在实际用例中约为1或2 GB)，我想将其发送到进程池(每个可用处理器一个减去一个，在我2011年末的macbook pro上总共有7)，以便使用Python 3.9进行一些计算。
最简单的方法是使用标准库的multiprocessing模块提供的工具。我在下面的test_standard_ipc函数中实现了这个解决方案。据我所知，这是调用该函数时发生的情况:使用pickle序列化data字典，然后一个单个os管道负责将序列化的数据流式传输到池的第一个进程;只有当这流完成相同的操作系统管用于流序列化数据池中的第二个进程等等。这意味着pool中的每个进程都需要等待轮到它接收序列化的数据(然后反序列化它并开始工作)。当数据的维度在1或2 GB左右时，这意味着池中的许多进程在开始执行操作之前必须等待很长时间。
为了克服这个问题，我提出了以下想法:由于我的data仅由内置数据类型组成，我将使用marshal模块(比pickle快得多)来序列化data，我将把得到的字节放在共享数组中，并将此数组的地址传递给池中的每个进程。通过这种方式，我应该能够立即启动池中的所有进程，这些进程将并发地反序列化数据并开始工作。然而，似乎从池中的每个进程读取共享字节数组非常慢(即使我使用只有一个进程的池)。我的猜测是，具有序列化数据的字节数组仍然是通过单个操作系统管道从子进程访问的，而不是直接访问它。
有没有办法加速这个方法?
下面是测试代码:# test_ipc.py
import marshal
import os
from multiprocessing import Manager, Pool
from time import time
from timeit import timeit
from pympler.asizeof import asizeof
def marshal_worker(data_array):
pid = os.getpid()
print(f" -> [pid {pid}] Marshal worker ready at {time()}")
# Building a bytearray is a waste of time but I did not found
# found a way to feed `data_array` directly to marshal.loads()
t = time()
ba = bytearray(data_array)
print(f" -> [pid {pid}] Building bytearray took {time() - t} s")
t = time()
data = marshal.loads(ba)
print(f" -> [pid {pid}] Marshal loads() took {time() - t} s")
return len(data)
def test_marshal_ipc(data):
print("Running test_marshal_ipc():")
n_processes = os.cpu_count() - 1 or 1
with Manager() as manager:
with Pool(processes=n_processes) as pool:
data_bytes = marshal.dumps(data)
data_array = manager.Array('B', data_bytes, lock=False)
async_results = [pool.apply_async(marshal_worker, (data_array,)) for _ in range(n_processes)]
subprocess_results = [res.get() for res in async_results]
return subprocess_results
def standard_worker(data):
print(f" -> [pid {os.getpid()}] Standard worker ready at {time()}")
return len(data)
def test_standard_ipc(data):
print("Running test_standard_ipc():")
n_processes = os.cpu_count() - 1 or 1
with Pool(processes=n_processes) as pool:
async_results = [pool.apply_async(standard_worker, (data,)) for _ in range(n_processes)]
subprocess_results = [res.get() for res in async_results]
return subprocess_results
if __name__ == '__main__':
REPETITIONS = 1
DATA_SIZE = 10_000
data = {
'foo': list(range(DATA_SIZE)),
'bar': dict(zip(range(DATA_SIZE), range(DATA_SIZE)))
}
print(f"Data size: {asizeof(data)} bytes")
marsall_time = timeit(
stmt="test_marshal_ipc(data)",
setup="from __main__ import test_marshal_ipc, data",
number=REPETITIONS
)
print(f"marshal ipc took: {marsall_time} s")
standard_time = timeit(
stmt="test_standard_ipc(data)",
setup="from __main__ import test_standard_ipc, data",
number=REPETITIONS
)
print(f"standard ipc took: {standard_time} s")
和输出:
$ python test_ipc.py
Data size: 1318944 bytes
Running test_marshal_ipc():
-> [pid 17950] Marshal worker ready at 1633625344.844704
-> [pid 17953] Marshal worker ready at 1633625344.8449469
-> [pid 17951] Marshal worker ready at 1633625344.8453
-> [pid 17955] Marshal worker ready at 1633625344.860242
-> [pid 17954] Marshal worker ready at 1633625344.864512
-> [pid 17952] Marshal worker ready at 1633625344.871718
-> [pid 17956] Marshal worker ready at 1633625344.876148
-> [pid 17950] Building bytearray took 58.384530782699585 s
-> [pid 17950] Marshal loads() took 0.0020139217376708984 s
-> [pid 17952] Building bytearray took 58.448140144348145 s
-> [pid 17952] Marshal loads() took 0.0024509429931640625 s
-> [pid 17956] Building bytearray took 58.71299409866333 s
-> [pid 17956] Marshal loads() took 0.002827167510986328 s
-> [pid 17954] Building bytearray took 58.93824005126953 s
-> [pid 17954] Marshal loads() took 0.0023200511932373047 s
-> [pid 17955] Building bytearray took 59.62452507019043 s
-> [pid 17955] Marshal loads() took 0.001924276351928711 s
-> [pid 17951] Building bytearray took 59.66379499435425 s
-> [pid 17951] Marshal loads() took 0.002319812774658203 s
-> [pid 17953] Building bytearray took 59.7155179977417 s
-> [pid 17953] Marshal loads() took 0.0018548965454101562 s
marshal ipc took: 60.396030886999995 s
Running test_standard_ipc():
-> [pid 17974] Standard worker ready at 1633625405.037303
-> [pid 17975] Standard worker ready at 1633625405.0419872
-> [pid 17974] Standard worker ready at 1633625405.043684
-> [pid 17975] Standard worker ready at 1633625405.045311
-> [pid 17974] Standard worker ready at 1633625405.047421
-> [pid 17974] Standard worker ready at 1633625405.05076
-> [pid 17975] Standard worker ready at 1633625405.05163
standard ipc took: 0.4552726120000017 s
更新:从manager.Array切换到multiprocessing.Array会抛出以下错误:
$ python test_ipc.py
Data size: 1318944 bytes
Running test_marshal_ipc():
Traceback (most recent call last):
File "test_ipc.py", line 67, in <module>
marsall_time = timeit(
File "***OMISSIS***/python3.9/timeit.py", line 233, in timeit
return Timer(stmt, setup, timer, globals).timeit(number)
File "***OMISSIS***/python3.9/timeit.py", line 177, in timeit
timing = self.inner(it, self.timer)
File "<timeit-src>", line 6, in inner
File "test_ipc.py", line 36, in test_marshal_ipc
subprocess_results = [res.get() for res in async_results]
File "test_ipc.py", line 36, in <listcomp>
subprocess_results = [res.get() for res in async_results]
File "***OMISSIS***/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "***OMISSIS***/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "***OMISSIS***/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "***OMISSIS***/python3.9/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "***OMISSIS***/python3.9/multiprocessing/sharedctypes.py", line 129, in reduce_ctype
assert_spawning(obj)
File "***OMISSIS***/python3.9/multiprocessing/context.py", line 359, in assert_spawning
raise RuntimeError(
RuntimeError: c_ubyte_Array_150019 objects should only be shared between processes through inheritance
使用SharedMemory从multiprocessing.shared_memory的建议，由@MarkSetchell我获得了以下解决方案，似乎比默认的略快。但是，这种方法需要在创建共享内存实例时事先知道序列化数据的大小。
import marshal
import os
import random
from multiprocessing import Pool
from multiprocessing.shared_memory import SharedMemory
from time import time
from timeit import timeit
from pympler.asizeof import asizeof
def marshal_worker(shm_name):
pid = os.getpid()
print(f" -> [pid {pid}] Marshal worker ready at {time()}")
t = time()
shm_data = SharedMemory(name=shm_name)
data = marshal.loads(shm_data.buf)
print(f" -> [pid {pid}] Marshal loads() took {time() - t} s")
shm_data.close()
return len(data)
def test_marshal_ipc(data):
print("Running test_marshal_ipc():")
t = time()
shm = SharedMemory(create=True, size=150_000_000) # unfortunately size must be known
marshal.dump(data, shm.buf.obj)
print(f"Marshal dump took {time() - t} s")
t = time()
n_processes = os.cpu_count() - 1 or 1
with Pool(processes=n_processes) as pool:
async_results = [pool.apply_async(marshal_worker, (shm.name,)) for _ in range(n_processes)]
subprocess_results = [res.get() for res in async_results]
print(f"Marshall process pool took {time() - t} s")
shm.close()
shm.unlink()
return subprocess_results
def standard_worker(data):
print(f" -> [pid {os.getpid()}] Standard worker ready at {time()}")
return len(data)
def test_standard_ipc(data):
print("Running test_standard_ipc():")
n_processes = os.cpu_count() - 1 or 1
with Pool(processes=n_processes) as pool:
async_results = [pool.apply_async(standard_worker, (data,)) for _ in range(n_processes)]
subprocess_results = [res.get() for res in async_results]
return subprocess_results
if __name__ == '__main__':
REPETITIONS = 10
DATA_SIZE = 1_000_000
data = {
'foo': list([random.randint(0, DATA_SIZE) for _ in range(DATA_SIZE)]),
'bar': dict(zip(range(DATA_SIZE), range(DATA_SIZE)))
}
print(f"Data size: {asizeof(data)} bytes")
standard_time = timeit(
stmt="test_standard_ipc(data)",
setup="from __main__ import test_standard_ipc, data",
number=REPETITIONS
)
print(f"standard ipc took: {standard_time} s")
marsall_time = timeit(
stmt="test_marshal_ipc(data)",
setup="from __main__ import test_marshal_ipc, data",
number=REPETITIONS
)
print(f"marshal ipc took: {marsall_time} s")
输出:
python speed_tests/test_ipc.py
Data size: 145927240 bytes
Running test_standard_ipc():
-> [pid 10505] Standard worker ready at 1633972725.070535
-> [pid 10506] Standard worker ready at 1633972725.2342331
-> [pid 10502] Standard worker ready at 1633972725.3869429
-> [pid 10504] Standard worker ready at 1633972725.666108
-> [pid 10501] Standard worker ready at 1633972725.869937
-> [pid 10500] Standard worker ready at 1633972726.000248
-> [pid 10503] Standard worker ready at 1633972726.223212
Running test_standard_ipc():
-> [pid 10510] Standard worker ready at 1633972727.024792
-> [pid 10512] Standard worker ready at 1633972727.155295
-> [pid 10509] Standard worker ready at 1633972727.3437078
-> [pid 10515] Standard worker ready at 1633972727.5215812
-> [pid 10511] Standard worker ready at 1633972727.651475
-> [pid 10513] Standard worker ready at 1633972727.818348
-> [pid 10514] Standard worker ready at 1633972727.934499
Running test_standard_ipc():
-> [pid 10517] Standard worker ready at 1633972728.7086048
-> [pid 10518] Standard worker ready at 1633972728.832938
-> [pid 10516] Standard worker ready at 1633972728.951583
-> [pid 10519] Standard worker ready at 1633972729.1504939
-> [pid 10520] Standard worker ready at 1633972729.333531
-> [pid 10521] Standard worker ready at 1633972729.488627
-> [pid 10522] Standard worker ready at 1633972729.597609
Running test_standard_ipc():
-> [pid 10529] Standard worker ready at 1633972730.453954
-> [pid 10526] Standard worker ready at 1633972730.581558
-> [pid 10523] Standard worker ready at 1633972730.693886
-> [pid 10528] Standard worker ready at 1633972730.9765298
-> [pid 10527] Standard worker ready at 1633972731.084288
-> [pid 10525] Standard worker ready at 1633972731.305248
-> [pid 10524] Standard worker ready at 1633972731.4186292
Running test_standard_ipc():
-> [pid 10532] Standard worker ready at 1633972732.1911612
-> [pid 10535] Standard worker ready at 1633972732.313635
-> [pid 10533] Standard worker ready at 1633972732.4908679
-> [pid 10536] Standard worker ready at 1633972732.671965
-> [pid 10534] Standard worker ready at 1633972732.81501
-> [pid 10538] Standard worker ready at 1633972732.948011
-> [pid 10537] Standard worker ready at 1633972733.072738
Running test_standard_ipc():
-> [pid 10542] Standard worker ready at 1633972733.840136
-> [pid 10540] Standard worker ready at 1633972733.9747682
-> [pid 10543] Standard worker ready at 1633972734.0945249
-> [pid 10541] Standard worker ready at 1633972734.2657769
-> [pid 10545] Standard worker ready at 1633972734.439857
-> [pid 10546] Standard worker ready at 1633972734.5595148
-> [pid 10544] Standard worker ready at 1633972734.69104
Running test_standard_ipc():
-> [pid 10548] Standard worker ready at 1633972735.461774
-> [pid 10549] Standard worker ready at 1633972735.670877
-> [pid 10547] Standard worker ready at 1633972735.793212
-> [pid 10551] Standard worker ready at 1633972735.991044
-> [pid 10550] Standard worker ready at 1633972736.182882
-> [pid 10553] Standard worker ready at 1633972736.3228412
-> [pid 10552] Standard worker ready at 1633972736.457024
Running test_standard_ipc():
-> [pid 10561] Standard worker ready at 1633972737.239666
-> [pid 10556] Standard worker ready at 1633972737.381736
-> [pid 10558] Standard worker ready at 1633972737.5161831
-> [pid 10557] Standard worker ready at 1633972737.769748
-> [pid 10560] Standard worker ready at 1633972737.9337258
-> [pid 10559] Standard worker ready at 1633972738.072955
-> [pid 10562] Standard worker ready at 1633972738.215271
Running test_standard_ipc():
-> [pid 10568] Standard worker ready at 1633972739.167847
-> [pid 10569] Standard worker ready at 1633972739.3302982
-> [pid 10565] Standard worker ready at 1633972739.501533
-> [pid 10564] Standard worker ready at 1633972739.687853
-> [pid 10567] Standard worker ready at 1633972739.8774452
-> [pid 10563] Standard worker ready at 1633972739.9893918
-> [pid 10566] Standard worker ready at 1633972740.126915
Running test_standard_ipc():
-> [pid 10573] Standard worker ready at 1633972741.1613598
-> [pid 10574] Standard worker ready at 1633972741.324167
-> [pid 10575] Standard worker ready at 1633972741.50316
-> [pid 10577] Standard worker ready at 1633972741.688416
-> [pid 10576] Standard worker ready at 1633972741.862753
-> [pid 10578] Standard worker ready at 1633972742.004191
-> [pid 10579] Standard worker ready at 1633972742.124938
standard ipc took: 18.082103576 s
Running test_marshal_ipc():
Marshal dump took 0.0761253833770752 s
-> [pid 10586] Marshal worker ready at 1633972742.639751
-> [pid 10583] Marshal worker ready at 1633972742.643724
-> [pid 10582] Marshal worker ready at 1633972742.644029
-> [pid 10585] Marshal worker ready at 1633972742.649504
-> [pid 10584] Marshal worker ready at 1633972742.651476
-> [pid 10587] Marshal worker ready at 1633972742.6582599
-> [pid 10588] Marshal worker ready at 1633972742.6726432
-> [pid 10584] Marshal loads() took 0.79646897315979 s
-> [pid 10586] Marshal loads() took 0.8943278789520264 s
-> [pid 10588] Marshal loads() took 0.8631129264831543 s
-> [pid 10582] Marshal loads() took 0.9072751998901367 s
-> [pid 10583] Marshal loads() took 0.948674201965332 s
-> [pid 10585] Marshal loads() took 1.0486416816711426 s
-> [pid 10587] Marshal loads() took 1.0870471000671387 s
Marshall process pool took 2.148102045059204 s
Running test_marshal_ipc():
Marshal dump took 0.15317106246948242 s
-> [pid 10589] Marshal worker ready at 1633972744.9749272
-> [pid 10590] Marshal worker ready at 1633972745.0029051
-> [pid 10592] Marshal worker ready at 1633972745.0056598
-> [pid 10594] Marshal worker ready at 1633972745.00722
-> [pid 10591] Marshal worker ready at 1633972745.008074
-> [pid 10595] Marshal worker ready at 1633972745.0141618
-> [pid 10593] Marshal worker ready at 1633972745.026942
-> [pid 10590] Marshal loads() took 0.5827910900115967 s
-> [pid 10594] Marshal loads() took 0.7236881256103516 s
-> [pid 10589] Marshal loads() took 0.7649509906768799 s
-> [pid 10593] Marshal loads() took 0.7148597240447998 s
-> [pid 10592] Marshal loads() took 0.8380041122436523 s
-> [pid 10591] Marshal loads() took 0.8357858657836914 s
-> [pid 10595] Marshal loads() took 1.0720829963684082 s
Marshall process pool took 1.6821980476379395 s
Running test_marshal_ipc():
Marshal dump took 0.08144736289978027 s
-> [pid 10599] Marshal worker ready at 1633972746.720268
-> [pid 10598] Marshal worker ready at 1633972746.720905
-> [pid 10601] Marshal worker ready at 1633972746.721835
-> [pid 10600] Marshal worker ready at 1633972746.7261682
-> [pid 10604] Marshal worker ready at 1633972746.729784
-> [pid 10602] Marshal worker ready at 1633972746.733629
-> [pid 10603] Marshal worker ready at 1633972746.7364821
-> [pid 10600] Marshal loads() took 0.707287073135376 s
-> [pid 10602] Marshal loads() took 0.7031078338623047 s
-> [pid 10603] Marshal loads() took 0.7563698291778564 s
-> [pid 10601] Marshal loads() took 0.844376802444458 s
-> [pid 10598] Marshal loads() took 0.8574948310852051 s
-> [pid 10604] Marshal loads() took 0.8612852096557617 s
-> [pid 10599] Marshal loads() took 0.8725550174713135 s
Marshall process pool took 1.5558290481567383 s
Running test_marshal_ipc():
Marshal dump took 0.08075690269470215 s
-> [pid 10605] Marshal worker ready at 1633972748.349489
-> [pid 10608] Marshal worker ready at 1633972748.351562
-> [pid 10606] Marshal worker ready at 1633972748.363499
-> [pid 10610] Marshal worker ready at 1633972748.37245
-> [pid 10607] Marshal worker ready at 1633972748.375197
-> [pid 10609] Marshal worker ready at 1633972748.379363
-> [pid 10611] Marshal worker ready at 1633972748.386376
-> [pid 10611] Marshal loads() took 0.5947482585906982 s
-> [pid 10608] Marshal loads() took 0.7976229190826416 s
-> [pid 10610] Marshal loads() took 0.7809698581695557 s
-> [pid 10609] Marshal loads() took 0.7778501510620117 s
-> [pid 10607] Marshal loads() took 0.7842419147491455 s
-> [pid 10605] Marshal loads() took 0.8898868560791016 s
-> [pid 10606] Marshal loads() took 0.8804740905761719 s
Marshall process pool took 1.554405927658081 s
Running test_marshal_ipc():
Marshal dump took 0.07740521430969238 s
-> [pid 10612] Marshal worker ready at 1633972750.222378
-> [pid 10615] Marshal worker ready at 1633972750.22273
-> [pid 10613] Marshal worker ready at 1633972750.225989
-> [pid 10614] Marshal worker ready at 1633972750.226241
-> [pid 10616] Marshal worker ready at 1633972750.232606
-> [pid 10618] Marshal worker ready at 1633972750.2384949
-> [pid 10617] Marshal worker ready at 1633972750.240912
-> [pid 10614] Marshal loads() took 0.7261221408843994 s
-> [pid 10613] Marshal loads() took 0.7348630428314209 s
-> [pid 10615] Marshal loads() took 0.7396330833435059 s
-> [pid 10612] Marshal loads() took 0.7663540840148926 s
-> [pid 10617] Marshal loads() took 0.7640130519866943 s
-> [pid 10616] Marshal loads() took 0.8179440498352051 s
-> [pid 10618] Marshal loads() took 0.8299651145935059 s
Marshall process pool took 1.716017723083496 s
Running test_marshal_ipc():
Marshal dump took 0.07379508018493652 s
-> [pid 10623] Marshal worker ready at 1633972751.784754
-> [pid 10624] Marshal worker ready at 1633972751.785677
-> [pid 10621] Marshal worker ready at 1633972751.786172
-> [pid 10625] Marshal worker ready at 1633972751.787801
-> [pid 10622] Marshal worker ready at 1633972751.793521
-> [pid 10626] Marshal worker ready at 1633972751.809876
-> [pid 10627] Marshal worker ready at 1633972751.810213
-> [pid 10625] Marshal loads() took 0.7419121265411377 s
-> [pid 10624] Marshal loads() took 0.7632110118865967 s
-> [pid 10623] Marshal loads() took 0.7657632827758789 s
-> [pid 10622] Marshal loads() took 0.76255202293396 s
-> [pid 10621] Marshal loads() took 0.775353193283081 s
-> [pid 10626] Marshal loads() took 0.8764150142669678 s
-> [pid 10627] Marshal loads() took 0.8768341541290283 s
Marshall process pool took 1.545395851135254 s
Running test_marshal_ipc():
Marshal dump took 0.07918524742126465 s
-> [pid 10630] Marshal worker ready at 1633972753.529343
-> [pid 10634] Marshal worker ready at 1633972753.533827
-> [pid 10629] Marshal worker ready at 1633972753.534485
-> [pid 10631] Marshal worker ready at 1633972753.535668
-> [pid 10635] Marshal worker ready at 1633972753.536008
-> [pid 10633] Marshal worker ready at 1633972753.538961
-> [pid 10632] Marshal worker ready at 1633972753.5452669
-> [pid 10634] Marshal loads() took 0.7382628917694092 s
-> [pid 10630] Marshal loads() took 0.743880033493042 s
-> [pid 10632] Marshal loads() took 0.729856014251709 s
-> [pid 10633] Marshal loads() took 0.7377548217773438 s
-> [pid 10635] Marshal loads() took 0.745603084564209 s
-> [pid 10629] Marshal loads() took 0.7638940811157227 s
-> [pid 10631] Marshal loads() took 0.8159899711608887 s
Marshall process pool took 1.6014928817749023 s
Running test_marshal_ipc():
Marshal dump took 0.08459973335266113 s
-> [pid 10637] Marshal worker ready at 1633972755.063104
-> [pid 10636] Marshal worker ready at 1633972755.0935879
-> [pid 10638] Marshal worker ready at 1633972755.113735
-> [pid 10640] Marshal worker ready at 1633972755.115721
-> [pid 10639] Marshal worker ready at 1633972755.117336
-> [pid 10641] Marshal worker ready at 1633972755.132608
-> [pid 10642] Marshal worker ready at 1633972755.1404412
-> [pid 10640] Marshal loads() took 0.7361090183258057 s
-> [pid 10638] Marshal loads() took 0.7426490783691406 s
-> [pid 10636] Marshal loads() took 0.7684330940246582 s
-> [pid 10641] Marshal loads() took 0.7523651123046875 s
-> [pid 10642] Marshal loads() took 0.7477109432220459 s
-> [pid 10637] Marshal loads() took 0.8323469161987305 s
-> [pid 10639] Marshal loads() took 0.8116521835327148 s
Marshall process pool took 1.4935619831085205 s
Running test_marshal_ipc():
Marshal dump took 0.07755017280578613 s
-> [pid 10647] Marshal worker ready at 1633972756.652229
-> [pid 10645] Marshal worker ready at 1633972756.672211
-> [pid 10646] Marshal worker ready at 1633972756.685754
-> [pid 10651] Marshal worker ready at 1633972756.695128
-> [pid 10649] Marshal worker ready at 1633972756.6956391
-> [pid 10648] Marshal worker ready at 1633972756.696948
-> [pid 10650] Marshal worker ready at 1633972756.710447
-> [pid 10645] Marshal loads() took 0.6723520755767822 s
-> [pid 10647] Marshal loads() took 0.711341142654419 s
-> [pid 10648] Marshal loads() took 0.695042610168457 s
-> [pid 10650] Marshal loads() took 0.6845788955688477 s
-> [pid 10649] Marshal loads() took 0.7024290561676025 s
-> [pid 10651] Marshal loads() took 0.7083790302276611 s
-> [pid 10646] Marshal loads() took 0.7469940185546875 s
Marshall process pool took 1.3334569931030273 s
Running test_marshal_ipc():
Marshal dump took 0.0805509090423584 s
-> [pid 10653] Marshal worker ready at 1633972758.09553
-> [pid 10652] Marshal worker ready at 1633972758.095892
-> [pid 10654] Marshal worker ready at 1633972758.102331
-> [pid 10655] Marshal worker ready at 1633972758.1137128
-> [pid 10656] Marshal worker ready at 1633972758.122677
-> [pid 10657] Marshal worker ready at 1633972758.12444
-> [pid 10658] Marshal worker ready at 1633972758.133462
-> [pid 10653] Marshal loads() took 0.7108800411224365 s
-> [pid 10652] Marshal loads() took 0.7952299118041992 s
-> [pid 10656] Marshal loads() took 0.768604040145874 s
-> [pid 10655] Marshal loads() took 0.7811801433563232 s
-> [pid 10654] Marshal loads() took 0.8009500503540039 s
-> [pid 10658] Marshal loads() took 0.770319938659668 s
-> [pid 10657] Marshal loads() took 0.8010349273681641 s
Marshall process pool took 1.4294342994689941 s
marshal ipc took: 16.948040361000004 s