需要一些关于在python中加速HTTP请求的建议

Question

我需要发送超过100万个HTTP请求，到目前为止，我尝试过的每个选项都太慢了。 我以为我可以用 aiohttp 加快速度，但这似乎并不比请求快。

我试图用python来做这件事，但我也对其他选择持开放态度。

这是使用请求和 aiohttp 的代码，有什么技巧可以加快这个过程吗？

请求代码：

import requests
url = 'https://mysite.mysite:443/login'
users = [line.strip() for line in open("ids.txt", "r")]
try:
for user in users:
r = requests.post(url,data ={'username':user})
if 'login.error.invalid.username' not in r.text:
print(user, " is valid")
else:
print(user, " not found")
except Exception as e:
print(e)

AIOHTTP 代码：

import aiohttp
import asyncio
url = 'https://mysite.mysite:443/login'
users = [line.strip() for line in open("ids.txt", "r")]   
async def main():
async with aiohttp.ClientSession() as session:
try:
for user in users:
payload = {"timeZoneOffSet": "240", "useragent": '', "username": user}
async with session.post(url, data=payload) as resp:           
if 'login.error.invalid.username' not in await resp.text():
print(user, " is valid")
else:
print(user, " not found")
except Exception as e:
print(e)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Answer 1

您可以使用asyncio.gather从一堆并行工作的请求中收集结果。

警告：代码只是一个示例，未经测试。

import asyncio
from aiohttp import ClientSession
async def fetch(url, session, payload):
async with session.post(url, data=payload) as resp:           
if 'login.error.invalid.username' not in await resp.text():
print(user, " is valid")
else:
print(user, " not found")
async def run(r):
url = "http://your_url:8000/{}"
tasks = []
async with ClientSession() as session:
for i in range(r):
task = asyncio.ensure_future(fetch(url.format(i), session))
tasks.append(task)
responses = await asyncio.gather(*tasks)
# you now have all response bodies in this variable
def print_responses(result):
print(result)
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(4))
loop.run_until_complete(future)