如何在遍历http请求后使用python多处理来获得全局总和

Question

我正在尝试编写一种算法，该算法遍历整个节点集合并返回它们的奖励总和。每个奖励只能计算一次。算法的输入将是节点开始的URL，例如http://fake.url/a.

URL的每个get请求都会返回一个JSON，如下所示：

{
"children":[
"http://fake.url/b",
"http://fake.url/c"
],
"reward":1
}

以下是我尝试过的：

import multiprocessing
import requests
import json
my_q = multiprocessing.Queue()
my_list  =['http://fake.url/']
reward_sum = 0
def enqueue(q):
for data in my_list:
q.put(data)
def get_it(q):
while not q.empty():
item = q.get()
print(item)
response = requests.get(item)
kids = json.loads(response.content)
print(f'URL: {item} --> {kids["reward"]}')
for kid in kids['children']:
print(kid)
q.put(kid)
p1 = multiprocessing.Process(target=enqueue, args=(my_q,))
p2 = multiprocessing.Process(target=get_it, args=(my_q,))
p1.start()
p2.start()
p1.join()
p2.join()

上面的工作原理：

1. 我正在使用多处理
2. 我正在正确地访问孩子和奖励
3. 我得到的输出是这样的：

http://fake.url/a
URL: http://fake.url/a --> 1
{'children': ['http://fake.url/b', 'http://fake.url/c'], 'reward': 1}
http://fake.url/b
http://fake.url/c
http://fake.url/b
URL: http://fake.url/b --> 2
{'children': ['http://fake.url/d', 'http://fake.url/e'], 'reward': 2}
http://fake.url/d
http://fake.url/e
http://fake.url/c
URL: http://fake.url/c --> 3
{'children': ['http://fake.url/f', 'http://fake.url/g'], 'reward': 3}
http://fake.url/f
http://fake.url/g
http://fake.url/d
URL: http://fake.url/d --> 4
{'reward': 4}
http://fake.url/e
URL: http://fake.url/e --> 5
{'reward': 5}
http://fake.url/f
URL: http://fake.url/f --> 6
{'children': ['http://fake.url/h'], 'reward': 6}
http://fake.url/h
http://fake.url/g

我需要帮助的问题有哪些：

1. 如何跟踪全局变量中的总奖励总额
2. 如何跟踪全球"；"看见"；设置，这样我就不会在总奖励金额中添加重复项

Answer 1

def get_it(q):
rewards_total = 0
seen = set()
while not q.empty():
item = q.get()
print(item)
if item in seen:
continue
seen.add(item)
response = requests.get(item)
kids = json.loads(response.content)
rewards_total += kids["reward"]
print(f'URL: {item} --> {kids["reward"]}')
for kid in kids['children']:
print(kid)
q.put(kid)
return rewards_total