python多处理在循环工作时不起作用

Question

我正在运行多处理以缩短多个时期的优化时间。

在我使用循环之前，当我的数据不大时，这很快。

for date in DatesOpt:
  x = X.loc[X['Date'] == np.int(date)].drop('Date',1) 
  f = F.loc[F['Date'] == np.int(date)].drop('Date',1)
  d = D.loc[D['Date'] == np.int(date)].drop('Date',1)
  r = R.loc[R['Date'] == np.int(date)].drop('Date',1)
  optimize(date,x,f,d,r)

优化函数将优化结果输出到CSV文件。

但是，当我将其更改为

时

if __name__=='__main__':
  pool = mp.Pool(mp.cpu_count()-1)
  for date in DatesOpt:
    x = X.loc[X['Date'] == np.int(date)].drop('Date',1) 
    f = F.loc[F['Date'] == np.int(date)].drop('Date',1)
    d = D.loc[D['Date'] == np.int(date)].drop('Date',1)
    r = R.loc[R['Date'] == np.int(date)].drop('Date',1)
    pool.apply_async(optimize,args=(date,x,f,d,r,))
  print('Waiting for all subprocesses done')
  pool.close()
  print('Pool Closed')
  pool.join()
  print('All subprocess done.')

一切都停在"池封闭"输出处，优化器永无止境。

此代码有任何问题吗？

Answer 1

您要在等待之前关闭池，最好反转：

if __name__=='__main__':
  pool = mp.Pool(mp.cpu_count()-1)
  for date in DatesOpt:
    x = X.loc[X['Date'] == np.int(date)].drop('Date',1) 
    f = F.loc[F['Date'] == np.int(date)].drop('Date',1)
    d = D.loc[D['Date'] == np.int(date)].drop('Date',1)
    r = R.loc[R['Date'] == np.int(date)].drop('Date',1)
    pool.apply_async(optimize,args=(date,x,f,d,r,))
  print('Waiting for all subprocesses done')
  pool.join()
  pool.close()
  print('Pool Closed')
  print('All subprocess done.')