小贝子编程

更正输入日志的正则表达式



输入日志如下所示，其中包含"|"的数据。数据包含 id |类型 |请求 |响应

110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
210000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
340000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
450000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
590000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

期望的输出：

第一个日志：

id- 110000
type-read
request-<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>
response-<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

对于第二个日志：

id - 210000
type - read
request -
response -

同样，对于日志的"n"号也是如此

使用的配置文件：

input {
file {
path => "/opt/test5/practice_new/final_xml.dat"
start_position => "beginning"
codec => multiline {
pattern => "^%{NUMBER:method_id}|%{DATA:method_type}|<soapenv:Envelope>"
negate => true
what => previous
}
}
}
filter {
grok {
match => [ "message", "(?m)^(?<method_id>d+)|(?<method_type>w+)|(?<request><soapenv:Envelope>.*?</soapenv:Envelope>)|(?<response><soapenv:Envelope>.*?</soapenv:Envelope>)" ]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "final"
}
stdout {}
}

我尝试在 Grok 中使用正则表达式，但当前表达式不适用于输入日志。

请帮助我使用正则表达式。

您当前使用的正则表达式是(?m)^(?<method_id>d+)|(?<method_type>w+)|(?<request><soapenv:Envelope>.*?</soapenv:Envelope>)|(?<response><soapenv:Envelope>.*?</soapenv:Envelope>)，并且它只能解析出第 3 列和第 4 列，如果它们以<soapenv:Envelope>开头并以中间有|</soapenv:Envelope>结束。

似乎您需要一个正则表达式，它将第 3 列标识为|以外的任何字符序列，并且第 4 列应将除|以外的任意数量的字符整理到换行符，后跟 1 位或多位数字，然后|。

用

(?m)^(?<method_id>d+)|(?<method_type>w+)|(?<request>[^|]*)|(?<response>[^|n]*(?:n(?!d+|)[^|n]*)*)

请参阅正则表达式演示。

详

  • (?m)- 使.匹配换行符字符的 Ruby 修饰符
  • ^- 行的开头
  • (?<method_id>d+)- 组"方法"：一个或多个数字
  • |- 烟斗炭
  • (?<method_type>w+)- 组"method_type"：一个或多个字母、数字或_
  • |- 一根烟斗
  • (?<request>[^|]*)- 组"请求"：除|以外的任何 0+ 字符
  • |- 管道
  • (?<response>[^|n]*(?:n(?!d+|)[^|n]*)*)- 组"响应"：
    • [^|n]*- 除|和 LF(换行符(以外的任何 0+ 字符
    • (?:n(?!d+|)[^|n]*)*- 0+ 出现：
      • n- 换行符
      • (?!d+|)- 不后跟 1+ 数字 +|
      • [^|n]*- 除|和 LF 以外的任何 0+ 字符(换行符(

相关内容

  • 没有找到相关文章

最新更新


热门标签：

javascript python java c# php android html jquery c++ css ios sql mysql arrays asp.net json python-3.x ruby-on-rails .net sql-server django objective-c excel regex ruby linux ajax iphone xml vba spring asp.net-mvc database wordpress string postgresql wpf windows xcode bash git oracle list vb.net multithreading eclipse algorithm macos powershell visual-studio image forms numpy scala function api selenium