日志无法添加字段？

Question

我一直在使用logstash来读取一些数据库还原日志。下面是一些示例记录行。

07/08/2016  6:33:22.50: START restore database                      
SQL2540W  Restore is successful, however a warning "2539" was encountered 
during Database Restore while processing in No Interrupt mode.
07/08/2016  6:33:28.93: END restore database                        
SQL4406W  The DB2 Administration Server was started successfully.
07/08/2016  6:35:35.29: END restart server                          
connect reset
DB20000I  The SQL command completed successfully.
07/08/2016  6:35:38.48: END p:s6sourcesystemCMDres_uw.cmd      

这是我的 conf 文件的过滤器部分。

if ([message] =~ /Backup successful/){
    grok{
        match => {"message" => ['%{GREEDYDATA:Message}'] }
    }
    mutate {
        add_tag => "send_to_es"
        add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
    }
}
if ([message] =~ /warning "2539"/){
    grok{
        match => {"message" => ['%{GREEDYDATA:Message}'] }
    }
    mutate {
        add_tag => "send_to_es"
        add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
    }
}
if ([message] =~ /(END p:|END P:)/){
    grok{
        match => {"message" => ['%{GREEDYDATA:DATETIME}:%{SPACE}END%{SPACE}%{GREEDYDATA:Mis}'] }
        remove_field => "%{GREEDYDATA:Mis}"
    }
    mutate {
        add_tag => "send_to_es"
    }
}   

我想将从记录的最后一行中提取的数据"DATETIME"添加到消息中，同时添加到其他消息以进行索引。但是，它无法成功添加该字段。输出将变为

      "message": "SQL2540W  Restore is successful, however a warning "2539" was encountered rr",
      "@version": "1",
      "@timestamp": "2016-07-12T02:28:52.337Z",
      "path": "C:/CIGNA/hkiapp67_db_restore/res_uw.log",
      "host": "SIMSPad",
      "type": "txt",
      "Message": "SQL2540W  Restore is successful, however a warning "2539" was encountered rr",
      "Timestamp": "%{GREEDYDATA:DATETIME}",
      "tags": [
        "send_to_es"
      ]

我该如何解决这个问题？

Answer 1

Logstash 在收到一行时，不知道任何其他行。您必须使用多行编解码器/过滤器来使用带有日期的行重新组合所需的所有行。然后，使用 grok 筛选器提取日期并将其添加到文档中。

多行编解码器/过滤器的配置如下所示：

multiline {
  pattern => "%{DATE}"
  negate => "true"
  what => "next"
}

这样，所有不以模式 DATE 开头的行将与下一行连接。