为什么增加工作线程数量(超过内核数量)仍然会减少执行时间?

Question

我总是确信线程/进程比CPU内核多是没有意义的(从性能的角度来看(。但是，我的 python 示例向我显示了不同的结果。

import concurrent.futures
import random
import time

def doSomething(task_num):
print("executing...", task_num)
time.sleep(1)  # simulate heavy operation that takes ~ 1 second    
return random.randint(1, 10) * random.randint(1, 500)  # real operation, used random to avoid caches and so on...

def main():
# This part is not taken in consideration because I don't want to
# measure the worker creation time
executor = concurrent.futures.ProcessPoolExecutor(max_workers=60)
start_time = time.time()
for i in range(1, 100): # execute 100 tasks
executor.map(doSomething, [i, ])
executor.shutdown(wait=True)
print("--- %s seconds ---" % (time.time() - start_time))

if __name__ == '__main__':
main()

项目成果：

1 个工作线程--- 100.28233647346497 秒---
2 个工人--- 50.50 秒。 26122164726257 秒 ---3 个工作--- 33.32741022109985 秒 ---4 个工人 --- 25.399883031845093 秒 ---5 个工人 --- 20.434186220169067 秒 ---10 个工作人员--- 10.903695344924924927 秒 ---50 个工作人员--- 6.363946914672852 秒 ---
60 个工人--- 4.819359302520752 秒---

只有 4 个逻辑处理器如何更快地工作？

这是我的计算机规格(在Windows 8和Ubuntu 14上测试(：

处理器 英特尔® 酷睿(TM( i5-3210M 处理器 @ 2.50GHz 插座： 1 核心：2逻辑处理器：4

Answer 1

原因是sleep()只使用微不足道的CPU。在这种情况下，它是对线程执行的实际工作的不良模拟。

sleep()真正做的只是挂起线程，直到计时器过期。当线程挂起时，它不使用任何 CPU 周期。

Answer 2

我用更密集的计算(例如矩阵反演(扩展了您的示例。您将看到您所期望的：计算时间将减少到内核数量，然后增加(因为上下文切换的成本(。

import concurrent.futures
import random
import time
import numpy as np
import matplotlib.pyplot as plt

def doSomething(task_num):
print("executing...", task_num)
for i in range(100000):
A = np.random.normal(0,1,(1000,1000))
B = np.inv(A)
return random.randint(1, 10) * random.randint(1, 500)  # real operation, used random to avoid caches and so on...
def measureTime(nWorkers: int):
executor = concurrent.futures.ProcessPoolExecutor(max_workers=nWorkers)
start_time = time.time()
for i in range(1, 40):  # execute 100 tasks
executor.map(doSomething, [i, ])
executor.shutdown(wait=True)
return (time.time() - start_time)
def main():
# This part is not taken in consideration because I don't want to
# measure the worker creation time
maxWorkers = 20
dT = np.zeros(maxWorkers)
for i in range(maxWorkers):
dT[i] = measureTime(i+1)
print("--- %s seconds ---" % dT[i])
plt.plot(np.linspace(1,maxWorkers, maxWorkers), dT)
plt.show()
if __name__ == '__main__':
main()