如何阻止 NodeJS 脚本在超过超时时崩溃

Question

您可以在下面找到发生的错误。

我正在尝试使用 NodeJS 和木偶师抓取网站的内容。有时，代码会停止并显示错误"超时"。有没有办法，如果超过页面加载的超时，我可以运行一个函数，该函数将重新加载页面或让脚本等待几秒钟，然后重新加载页面，直到它正确获取数据，而不会崩溃？如果是这样，我将如何继续实施它？

谢谢。

(node:8300) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (C:Usersdanienode_modulespuppeteerlibLifecycleWatcher.js:143:21)
  -- ASYNC --
    at Frame.<anonymous> (C:Usersdanienode_modulespuppeteerlibhelper.js:108:27)
    at Page.goto (C:Usersdanienode_modulespuppeteerlibPage.js:656:49)
    at Page.<anonymous> (C:Usersdanienode_modulespuppeteerlibhelper.js:109:23)
    at scrape (C:UsersdanieDocumentsNode Projectsp-download.js:23:14)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:8300) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:8300) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

我的代码：

const puppeteer = require('puppeteer');
let scrape = async () => {
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();
    await page.setRequestInterception(true);    
    page.on('request', (req) => {
        if(req.resourceType() == 'stylesheet' || req.resourceType() == 'script' || req.resourceType() == 'font' || req.resourceType() == 'media' || req.resourceType() == 'image'){
            req.abort();
        }
        else {
            req.continue();
        }
    }); //Disables loading CSS, images and scripts
    for(i=0; i<5000; i++){
        await page.goto('https://website.com/' + i);
        let result = await page.evaluate(() => {
            var result = '';
            for (i=1; i<=10; i++){
                result += document.getElementsByTagName('td')[i].innerText;
                result += ',';
            }
            result += 'n';
            return result;
        });
    }
}
scrape();

Answer 1

将您的代码放在try/catch块中以避免崩溃...我会将循环代码移动到另一个函数中

for(i=0; i<5000; i++){
    result = await open_page(page , i );
}

async function open_page(page , i ){
    try {
            await page.goto('https://website.com/' + i);
            let result = await page.evaluate(() => {
                var result = '';
                for (i=1; i<=10; i++){
                    result += document.getElementsByTagName('td')[i].innerText;
                    result += ',';
                }
                result += 'n';
                return result;
            });

            return {stat:1 , result : result } ;
    }
    catch(e){
        return {stat:0 , error : e } ;
    }
}