批量并行HTTP请求，每个请求都有异步for循环

Question

我正在尝试使用数组中的一组关键字批量运行对API的并行请求。Denis Fatkhudinov的文章。

我遇到的问题是，对于每个关键字，我需要使用不同的page参数再次运行请求，次数与pages变量中的数字一样多。

为了返回chainNext函数，我一直在获取Cannot read property 'then' of undefined。

批量并行请求本身，没有for循环，效果很好，我正在努力将for循环融入到进程中。

// Parallel requests in batches
async function runBatches() {
// The keywords to request with 
const keywords = ['many keyword strings here...'];
// Set max concurrent requests
const concurrent = 5;
// Clone keywords array
const keywordsClone = keywords.slice()
// Array for future resolved promises for each batch
const promises = new Array(concurrent).fill(Promise.resolve());
// Async for loop
const asyncForEach = async (pages, callback) => {
for (let page = 1; page <= pages; page++) {
await callback(page);
}
};
// Number of pages to loop for
const pages = 2;
// Recursively run batches
const chainNext = (pro) => {
// Runs itself as long as there are entries left on the array
if (keywordsClone.length) {
// Store the first entry and conviently also remove it from the array
const keyword = keywordsClone.shift();
// Run 'the promise to be' request
return pro.then(async () => {
// ---> Here was my problem, I am declaring the constant before running the for loop
const promiseOperation = await asyncForEach(pages, async (page) => {
await request(keyword, page)
});
// ---> The recursive invocation should also be inside the for loop
return chainNext(promiseOperation);
});
}
return pro;
}
return await Promise.all(promises.map(chainNext));
}
// HTTP request
async function request(keyword, page) { 
try {
// request API 
const res = await apiservice(keyword, page);
// Send data to an outer async function to process the data
await append(res.data);
} catch (error) {
throw new Error(error) 
}
}

runBatches()

Answer 1

问题只是pro没有定义，因为您还没有初始化它。

你基本上执行这个代码：

Promise.all(new Array(concurrent).fill(Promise.resolve().map(pro => {
// pro is undefined here because the Promise.resolve had no parameter
return pro.then(async () => {})
}));

我不完全确定你背后的想法，但这是一个更精简的版本中的问题。

Answer 2

我通过在for循环中移动实际请求promiseOperation并在那里返回递归函数使其工作

// Recursively run batches
const chainNext = async (pro) => {
if (keywordsClone.length) {
const keyword = keywordsClone.shift()
return pro.then(async () => {
await asyncForEach(pages, (page) => {
const promiseOperation = request(keyword, page)
return chainNext(promiseOperation)
})
})
}
return pro
}

批量并行请求的贷项转到https://itnext.io/node-js-handling-asynchronous-operations-in-parallel-69679dfae3fc