如何对值列表进行多处理?

Question

• 我在python中有1000个值
• 每个值必须单独处理
• 如何独立完成流程
• 这里的问题是关于多进程或独立进程

我的当前代码

import mysql.connector
for each in value_list:

# Connecting to the Database
mydb = mysql.connector.connect(
host ='localhost',
database ='College',
user ='root',
)
cs = mydb.cursor()

# drop clause
statement ="UPDATE STUDENT SET AGE = 23 WHERE Name=each"

cs.execute(statement)
mydb.commit()

# Disconnecting from the database
mydb.close()

我上面的代码将逐个处理。由于每次更新都是独立的，所以如何使用multiprocessing

实现有没有办法使用像from joblib import Parallel, delayed

Answer 1

下面是如何使用多进程来实现这一点的示例。

from multiprocessing import Process, Queue
import mysql.connector as MYSQL
NPROCS = 5
value_list = [] # List of names to be updated
CONFIG = {
'host': 'localhost',
'user': 'root',
#'passwd': 'secret',
'database': 'College'
}
def process(queue):
conn = None
try:
conn = MYSQL.connect(**CONFIG)
while param := queue.get():
cursor = conn.cursor()
sql = f'UPDATE STUDENT SET AGE=23 WHERE Name="{param}"'
cursor.execute(sql)
cursor.close()
except Exception as e:
print(e)
finally:
if conn:
conn.commit()
conn.close()
def main():
queue = Queue()
procs = [Process(target=process, args=(queue,)) for _ in range(NPROCS)]
for p in procs:
p.start()
for each in value_list:
queue.put(each)
for _ in range(NPROCS):
queue.put(None)
for p in procs:
p.join()
if __name__ == '__main__':
main()