使用多线程 python3 加速

Question

>实际上我正在创建一个代理检查器，但问题是检查需要很多时间，因为有很多代理

def proxy():
lives = []
allproxy = []
def fetch_proxy():
raw_proxy = []
res = requests.get(proxy_api)
raw_proxy = res.text.splitlines()
return raw_proxy
allproxy = fetch_proxy()
for proxy in allproxy:
try:
proxyDictChk = { 
"https"  : "https://"+proxy, 
"http" : "http://"+proxy,
}
res = requests.get("http://httpbin.org/ip",proxies=proxyDictChk,timeout=3)
print("Proxy is Working")
lives.append(proxy)
except Exception as e:
print("Proxy Dead")
return lives
print(proxy())

我很好奇我如何在这里使用多线程来快速

提前致谢

Answer 1

python文档提供了一个很好的例子，https://docs.python.org/3/library/concurrent.futures.html

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
# Start the load operations and mark each future with its URL
future_to_url = {executor.submit(check_proxy, url, 60): url for url in allproxy}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
is_valid = future.result()
except Exception as exc:
print('%r generated an exception: %s' % (url, exc))
else:
print('%s page is %s' % (url, is_valid))

因此，您只需要定义函数check_proxy。

def check_proxy( proxy ):
try:
proxyDictChk = { 
"https"  : "https://"+proxy, 
"http" : "http://"+proxy,
}
res = requests.get("http://httpbin.org/ip",proxies=proxyDictChk,timeout=3)
print("Proxy is Working")
return True
except Exception as e:
print("Proxies Dead!")
return False

本质上，使用执行器并提交一个执行您想要的函数。然后使用将来在函数完成时获取函数的结果。

此外，由于这允许异常冒泡，因此您不必在函数中处理它。

def check_proxy( proxy ):
proxyDictChk = { "https"  : "https://"+proxy, 
"http" : "http://"+proxy,
}
res = requests.get("http://httpbin.org/ip",proxies=proxyDictChk,timeout=3)
return True

现在，可以在将来的状态处理异常。您可以将返回类型更改为更有意义的类型。