任何可能加快处理速度的方法

Question

我现在有一个80个用户名的列表，我有我的脚本检查每个用户名是否存在。但是，它需要比我喜欢的时间长一点，所以我想知道是否有任何我可以做的事情来加快检查每个用户名是否存在所需的时间。

# ------------------------------
# Mass Kik Username Checker
# Script Made by: Ski
# ------------------------------
import requests, threading
def check(username):
    try:
        req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code
        if req == 302:
            return False
        if req == 200:
            return True
    except Exception as e:
        print e
        exit()

def _loadList(filename):
    item_list = []
    for item in str(open(filename, "r").read()).split("n"):
        item_list.append(item)
    return item_list
def _thread(items):
    global _usernames
    for username in _usernames[items[0]:items[1]]:
        exists = check(username)
        if exists:
            print username+" existsn"
        if not exists:
            print username+" doesn't existn"
if __name__ == '__main__':
    _usernames = _loadList("usernames.txt")
    thread1 = threading.Thread(target=_thread, args=([0, 20], )).start()
    thread2 = threading.Thread(target=_thread, args=([20, 40], )).start()
    thread3 = threading.Thread(target=_thread, args=([40, 60], )).start()
    thread4 = threading.Thread(target=_thread, args=([60, 80], )).start()

Answer 1

试试Python 3。x线程池。您可以定义有多少工作人员将执行请求。使用比4更多的值(例如32)，将显著提高代码的速度。

import requests
from concurrent.futures import ThreadPoolExecutor

NUM_OF_WORKERS=32

def check(username):
    try:
        req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code
        if req == 302:
            print(username, " does not exist.")
        if req == 200:
            print(username, "exists.")
    except Exception as error:
        print(error)

usernames = _loadList(filename)
with ThreadPoolExecutor(max_workers=NUM_OF_WORKERS) as pool:
    pool.map(check, usernames)

这使得你的代码方式更容易读。

编辑:现在注意到Python 2.7标签。

Python 2有一个线程池，可以在multiprocessing模块下使用。不幸的是，由于没有可用的测试，因此没有记录。

import requests
from multiprocessing.pool import ThreadPool

NUM_OF_WORKERS=32

def check(username):
    try:
       req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code
       if req == 302:
           print(username, " does not exist.")
       if req == 200:
           print(username, "exists.")
    except Exception as error:
        print(error)

usernames = _loadList(filename)

pool = ThreadPool(processes=NUM_OF_WORKERS)
pool.map_async(check, usernames)
pool.close()
pool.join()

如果你想要一个更好的Python 2线程池，你可以试试Pebble模块