如何使用grok在logstash中转换日志

Question

我的日志格式如下:

TIME|HOST-IP|REQUEST-ID|UID|USERNAME|USER-AGENT|METHOD|URI|CONTROLLER-METHOD|PARAMS-MAP|RESPONSE-CODE

和，日志看起来像:

2016-11-12 21:02:45.878|192.168.31.205|368284310235387-20|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200
2016-11-12 21:02:46.100|192.168.31.205|368284310235387-21|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200
2016-11-12 21:02:46.389|192.168.31.205|368284310235387-22|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200
2016-11-12 21:02:46.507|192.168.31.205|368284310235387-23|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200

日志库版本是5.0.0，日志是由服务器上的Filebeat发送的

我的logstash配置是:

input {
    beats {
        port => "5043"
    }
}
filter {
    grok {
        match => { "message" => "%{WORD:time}|%{IP:hostIP}|%{WORD:requestId}|%{NUMBER:uid:int}|%{WORD:username}|%{WORD:method}|%{URIPATHPARAM:uri}|%{WORD:cont
rollerMethod}|%{WORD:params}" }
    }
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => [ "10.94.66.193:9200" ]
    }
}

但是，我只得到消息字段:

{
    "@timestamp" => 2016-11-12T13:02:48.607Z,
        "offset" => 31831,
      "@version" => "1",
    "input_type" => "log",
          "beat" => {
        "hostname" => "localhost",
            "name" => "localhost",
         "version" => "5.0.0"
    },
          "host" => "localhost",
        "source" => "logs/km-access.2016-11-12.log",
          "time" => "2016",
       "message" => "2016-11-12 21:02:46.507|192.168.31.205|368284310235387-23|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200",
          "type" => "log",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}

Answer 1

使用以下模式匹配此日志:

%{TIMESTAMP_ISO8601}|%{IP}|(?<requestID>d+-d+)|%{INT:uid}|%{WORD:username}|(?<ua>(w|/|.|s|(|;|)|,)+)|%{WORD:method}|(?<uri>(w|/)+)|(?<controllerMethod>(w|d|s|.|#|[|])+)|(?<param>(w|{|}|"|:|[|])+)|%{NUMBER:statusCode}

谢谢https://grokdebug.herokuapp.com/