使用 crawler4j 和 Jsoup 获取 Http 状态

Question

我正在创建一个Groovy &在后端使用MongoDB的Grails应用程序。我使用crawler4j进行爬行，使用JSoup进行解析功能。我需要得到一个URL的http状态，并将其保存到数据库。我正在尝试以下操作:

@Override
void visit(Page page) {
try{
    Document doc = Jsoup.connect(url).get();
    Connection.Response response = Jsoup.connect(url)
            .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chroe/19.0.1042.0 Safari/535.21")
            .timeout(10000)
            .execute();
    int statusCode = response.statusCode();
    println "statuscode is " + statusCode
    if (statusCode == 200)
        urlExists = true    //urlExists is a boolean variable
    else 
        urlExists = false
    //save to database
    resource = new Resource(mimeType : "text/html", URLExists: urlExists)
    if (!resource.save(flush: true, failOnError: true)) {
        resource.errors.each { println it }
    }
    //other code
    }catch(Exception e) {
        log.error "Exception is ${e.message}"
    }
}
@Override
protected void handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) {
    if (statusCode != HttpStatus.SC_OK) {
        if (statusCode == HttpStatus.SC_NOT_FOUND) {
        println "Broken link: " + webUrl.getURL() + ", this link was found in page: " + webUrl.getParentUrl()
        }
        else {
            println  "Non success status for link: " + webUrl.getURL() + ", status code: " + statusCode + ", description: " + statusDescription
        }
    }
}

问题是，只要我得到一个url与http状态以外的200(ok)，它直接去到handlePageStatusCode()方法(由于固有的crawler4j功能)和打印非成功消息，但它没有被保存到数据库。当页面状态不是200时，是否有任何方法可以保存到数据库?如果我做错了什么，请告诉我。由于

Answer 1

为什么不保存到数据库当它下降到handlePageStatusCode?

    protected void handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) {
if (statusCode != HttpStatus.SC_OK) {
    if (statusCode == HttpStatus.SC_NOT_FOUND) {
    println "Broken link: " + webUrl.getURL() + ", this link was found in page: " + webUrl.getParentUrl()
    //save to database
    }
    else {
        println  "Non success status for link: " + webUrl.getURL() + ", status code: " +        tatusCode + ", description: " + statusDescription
    }
  }

}

它会尝试下一个链接，你可以做同样的事情。

或者保存在

之前

  if (statusCode == 200)
    urlExists = true    //urlExists is a boolean variable
     else {
           //save to database
           urlExists = false
        }

编辑* * * *

将webUrl.getURL()添加到ArrayList中，然后最后将其保存到数据库中