Crawler4j抓取jquery实时内容

Question

我有一个网站，但在其类别页面上，通过javascript加载页面后生成的产品列表。然后我的爬虫去找，找不到任何产品。我怎样才能解决这个问题?

        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder(rootFolder);
        config.setMaxPagesToFetch(100000000);
        config.setMaxDepthOfCrawling(-1);
        config.setPolitenessDelay(1);
        config.setUserAgentString("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36");
        //config.setResumableCrawling(true);
        config.setIncludeHttpsPages(true);

        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        robotstxtConfig.setEnabled(false);
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        controller.addSeed(siteDomain);
        for(int i = 4; i<=14; i++)
        {
            if(i < args.length)
        {
            controller.addSeed(args[i]);
        }
        }

        controller.start(Crawling.class, numberOfCrawlers);

        List<Object> crawlersLocalData = controller.getCrawlersLocalData();

Answer 1

不幸的是，crawler4j只支持静态内容。对于javascript和ajax的支持，使用爬虫，如crawljax或nutch with selenium.