Grok 过滤器与 bro httplog 数据不匹配

Question

我正在尝试使用ELK来可视化BRO日志数据。我在网上发现了多个grok过滤器，它总是无法将模式与数据匹配。我尝试使用的过滤器之一是：

grok {
match => [ "message", "(?<ts>(.*?))t(?<uid>(.*?))t(?<id.orig_h>(.*?))t(?<id.orig_p>(.*?))t(?<id.resp_h>(.*?))t(?<id.resp_p>(.*?))t(?<trans_depth>(.*?))t(?<method>(.*?))t(?<bro_host>(.*?))t(?<uri>(.*?))t(?<referrer>(.*?))t(?<user_agent>(.*?))t(?<request_body_len>(.*?))t(?<response_body_len>(.*?))t(?<status_code>(.*?))t(?<status_msg>(.*?))t(?<info_code>(.*?))t(?<info_msg>(.*?))t(?<filename>(.*?))t(?<http_tags>(.*?))t(?<username>(.*?))t(?<password>(.*?))t(?<proxied>(.*?))t(?<orig_fuids>(.*?))t(?<orig_mime_types>(.*?))t(?<resp_fuids>(.*?))t(?<resp_mime_types>(.*))" ]
}

我试图获取的bro数据如下：

#separator x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   http
#open   2018-11-27-18-31-02
#fields ts  uid id.orig_h   id.orig_p   id.resp_h   id.resp_p   trans_depth method  host    uri referrer    version user_agent  request_body_len    response_body_len   status_code status_msg  info_code   info_msg    tags    username    password    proxied orig_fuids  orig_filenames  orig_mime_types resp_fuids  resp_filenames  resp_mime_types
#types  time    string  addr    port    addr    port    count   string  string  string  string  string  string  count   count   count   string  count   string  set[enum]   string  string  set[string] vector[string]  vector[string]  vector[string]  vector[string]  vector[string]  vector[string]
1543343462.308603   CrJmZi31EU3tUXba3c  10.100.130.72   38396   216.58.217.110  80  1   -   -   -   -   1.1 -   0   219 301 Moved Permanently   -   -   (empty) -   -   -   -   -   -   FXhQ5K1ydVhFnz9Agi  -   text/html
1543344229.051726   CLj9eD4BcFR42BRHV1  10.100.130.72   37452   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FO0Zko4uvyxeC8LDx4  -   text/plain
1543345395.827176   C6Kdv49oODjjkgeFk   10.100.130.72   37464   169.254.169.254 80  1   -   -   -   -   1.0 -   0   345 404 Not Found   -   -   (empty) -   -   -   -   -   -   FW4NGDCyMNR43J4Hf   -   text/html
1543345691.165771   CNaObqkLN9imdehl4   10.100.130.72   37466   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FmUSygO8ocHKTN8L3   -   text/plain
1543347316.900516   Ck5CsV2hr56axo3rzl  10.100.130.72   37486   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FXKDmj3kllpKuJnSkg  -   text/plain
1543348718.870063   CFBClg1jRpmBp4ElYb  10.100.130.72   37506   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   F02j4T12ssIF2tYFF5  -   text/plain
1543348995.827387   CPMwHt2g13sPqdiXE1  10.100.130.72   37508   169.254.169.254 80  1   -   -   -   -   1.0 -   0   345 404 Not Found   -   -   (empty) -   -   -   -   -   -   FsbLPY8A3gpuBkM7l   -   text/html
1543350095.640070   CObHQk2ARejHIWBcgc  10.100.130.72   37518   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FxCY9C2fOP4dHO2Dkj  -   text/plain

谢谢-JP-

Answer 1

1. 忽略以#开头的行
2. 我使用了给出的Grok解析器，它很有效。您可以使用grokdebugger来检查解析器。https://grokdebug.herokuapp.com/
3. 数据源使用的可能是空格而不是制表符。验证数据

4. 我使用了=>而不是逗号(，(。将代码更改为

   grok {
   match => [ "message" => "(?<ts>(.*?))t(?<uid>(.*?))t(?<id.orig_h>(.*?))t(?<id.orig_p>(.*?))t(?<id.resp_h>(.*?))t(?<id.resp_p>(.*?))t(?<trans_depth>(.*?))t(?<method>(.*?))t(?<bro_host>(.*?))t(?<uri>(.*?))t(?<referrer>(.*?))t(?<user_agent>(.*?))t(?<request_body_len>(.*?))t(?<response_body_len>(.*?))t(?<status_code>(.*?))t(?<status_msg>(.*?))t(?<info_code>(.*?))t(?<info_msg>(.*?))t(?<filename>(.*?))t(?<http_tags>(.*?))t(?<username>(.*?))t(?<password>(.*?))t(?<proxied>(.*?))t(?<orig_fuids>(.*?))t(?<orig_mime_types>(.*?))t(?<resp_fuids>(.*?))t(?<resp_mime_types>(.*))" ]
   }
   

5. 如果错误仍然存在，您也可以使用csv解析器而不是Grok。