Python 线程中的多处理不会使每个子进程都正常工作

Question

当使用 python 中的模块在每个线程中执行子进程时threading某些进程无法像示例代码的输出那样正确启动和挂起。

这似乎启动一个进程需要独占控制，因为 python 中的 IPC。对吗？

使用螺纹锁，它可以完美地工作。我只是想做一个线程来控制一个子进程的生死管理。

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# vim:fenc=utf-8
import multiprocessing
import threading
import fire
from logzero import logger

def process_method():
logger.info("process log")

def start_process():
logger.info("starting thread")
process = multiprocessing.Process(target=process_method)
process.daemon = True
process.start()
process.join()

def main(num_of_threads=3):
threads = []
for _ in range(num_of_threads):
t = threading.Thread(target=start_process)
t.daemon = True
t.start()
threads.append(t)
for t in threads:
t.join()
logger.info("program done")

if __name__ == "__main__":
fire.Fire(main)

输出

$ python '/tmp/tmp.lY3YDIltYg/test.py' 30
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log

环境

• 蟒蛇 3.6
• 乌博托 64位

Answer 1

我假设你在类UNIX系统上？因为如果是这样，你在这里做坏事;将fork与线程混合是一个坏主意，Python 在类 UNIX 系统上的默认Process实现使用fork，因此通过在线程中启动Processes，您正在从多线程进程中fork。

这里有两种解决方案：

1. 停止使用线程。您的线程在这里不是必需的，因为它们实际上并不管理生命周期(线程和进程都是daemon的，因此当主进程的主线程完成时，它们将被毫不客气地杀死(。
2. 如果你必须使用线程，并且你使用的是Python 3.4+，你可以切换到fork-server start方法(或'spawn'方法，使你的代码可移植到Windows，代价是使Process创建在类UNIX系统上慢一点(，所以主进程的唯一fork是在你启动任何线程之前完成的， 所有未来的fork都是从(无线程(fork服务器完成的。您要做的就是将multiprocessing.set_start_method('forkserver')添加为if __name__ == "__main__":守卫之后的第一行。