Google 任务队列中的递归不遵守速率限制

Question

我在Google App Engine中使用递归方法来循环浏览分页页面。队列中只能同时有一个任务，因为您只能获取当前页面的下一页，即无法知道要同时添加到队列中的所有页面。

递归函数工作正常，但是，它不遵守 queue.yaml 文件中的说明。

我创建了一个测试示例来说明问题。我希望这个过程需要五分钟。原因是 queue.yaml 文件每分钟只应该执行一个任务，而我正在向队列发送五个任务。但是，所有任务都会同时执行。

我可以在 test() 方法中与睡眠函数一起破解一些东西来减慢它的速度，但这违背了在 queue.yaml 文件中使用速率限制的目的。

任何帮助将不胜感激！

main.py

@app.route('/test',methods=['POST'])
def test():     
data = json.loads(request.data)
current_num,max_num = data['current_num'],data['max_num']
if current_num <= max_num:      
current_num += 1
task = taskqueue.add(
queue_name='test-queue',
method='POST',
url='/test',
payload=json.dumps({'current_num':current_num,
'max_num':max_num}))
message = "current_num: {}, max_num: {}".format(current_num,max_num)                
print message
return message
else:
print 'finished'
return 'finished'
@app.route('/task')
def test_task():
max_num = 5
task = taskqueue.add(
queue_name='test-queue',
method='POST',
url='/test',
payload=json.dumps({'current_num':0,'max_num':max_num})
)
print "current_num: {}, max_num: {}".format(0,max_num)
return 'submitted task'

队列.yaml

queue:
- name: test-queue
target: app.main
rate: 1/m
bucket_size: 1
max_concurrent_requests: 1
retry_parameters:
min_backoff_seconds: 5
task_retry_limit: 1

日志文件：

WARNING  2018-04-18 12:24:19,824 urlfetch_stub.py:555] Stripped prohibited headers from URLFetch request: ['content-length']
current_num: 0, max_num: 5
INFO     2018-04-18 12:24:20,299 module.py:835] default: "GET /task HTTP/1.1" 200 14
WARNING  2018-04-18 12:24:20,338 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 1, max_num: 5
INFO     2018-04-18 12:24:20,350 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,350 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 2, max_num: 5
INFO     2018-04-18 12:24:20,362 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,363 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 3, max_num: 5
INFO     2018-04-18 12:24:20,380 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,381 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 4, max_num: 5
INFO     2018-04-18 12:24:20,399 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,400 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 5, max_num: 5
INFO     2018-04-18 12:24:20,415 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,416 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 6, max_num: 5
INFO     2018-04-18 12:24:20,441 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,441 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
finished
INFO     2018-04-18 12:24:20,490 module.py:835] default: "POST /test HTTP/1.1" 200 8

请注意执行时间：

2018-04-18 12:24:20,299
2018-04-18 12:24:20,350
2018-04-18 12:24:20,362
2018-04-18 12:24:20,380
2018-04-18 12:24:20,399
2018-04-18 12:24:20,415
2018-04-18 12:24:20,441

Answer 1

这是开发服务器的预期行为，如文档中所述：

开发服务器不考虑队列的速率和存储桶大小属性。因此，任务的执行时间尽可能接近其预计到达时间。将速率设置为 0 不会阻止任务自动执行。