并行创建复杂的数据框架

Question

下面的代码似乎有一些问题。目的是将new_df()的每个结果附加到某个列表中，例如out.

import pandas as pd
import random
import time
from multiprocessing import Pool
def new_df(rows=10000):  # proxy for complex dataframe
temp = pd.DataFrame({'a': [''.join(chr(random.randint(65,122)) for _ in range(200))
for _ in range(rows)]})
temp['b'] = temp['a'].str.lower()
temp['c'] = temp['a'].str.upper()
return temp
pool = Pool(4)
start = time.time()
out = pool.map(new_df, [9999,10000,10001,10002])
print(f"{time.time() - now} sec")

问题- VisualStudioCode

raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.

Answer 1

使用主模块习惯用法重建的代码:

import pandas as pd
import random
import time
from multiprocessing import Pool
def new_df(rows=10000):
temp = pd.DataFrame({'a': [''.join(chr(random.randint(65,122)) for _ in range(200))
for _ in range(rows)]})
temp['b'] = temp['a'].str.lower()
temp['c'] = temp['a'].str.upper()
return temp
def main():
start = time.perf_counter()
with Pool(4) as pool:
pool.map(new_df, [9999, 10000, 10001, 10002])
print(f"{time.perf_counter() - start:.2f}s")
if __name__ == '__main__':
main()

输出:

1.24s