同时进行期货网络交易

Question

我目前正在尝试开发一个快速的网络抓取功能，这样我就可以抓取大量文件。

这是我目前拥有的代码：

import time
import requests
from bs4 import BeautifulSoup
from concurrent.futures import ProcessPoolExecutor, as_completed
def parse(url):
r = requests.get(url)
soup = BeautifulSoup(r.content, 'lxml')
return soup.find_all('a')
with ProcessPoolExecutor(max_workers=4) as executor:
start = time.time()
futures = [ executor.submit(parse, url) for url in URLs ]
results = []
for result in as_completed(futures):
results.append(result)
end = time.time()
print("Time Taken: {:.6f}s".format(end-start))

这会为网站带来结果，例如www.google.com，然而，我的问题是，我不知道如何查看它返回的数据我只得到未来的物品。

请有人给我解释/教我怎么做。

我很感激你随时帮助我。

Answer 1

您也可以通过dict理解来实现它，如下所示。

with ProcessPoolExecutor(max_workers=4) as executor:
start = time.time()
futures = { executor.submit(parse, url): url for url in URLs }
for result in as_completed(futures):
link = futures.get(result)
try:
data = result.result()
except Exception as e:
print(e)
else:
print("Link: {}, data: {}".format(link, data))
end = time.time()
print("Time Taken: {:.6f}s".format(end-start))