扫描日志文件以查找最近 30 分钟的数据

Question

我必须编写一个 shell/perl 脚本来扫描日志文件以获取最近 30 分钟的数据。要求是在 Cron 中安排此脚本每 30 分钟运行一次，并查找错误字符串。

OS: Solaris
Shell:Bash

我已经尝试了下面的脚本，但它变得太长和笨拙，我们还有其他方法可以让它更短一点吗？

blogs=/opt/docs/datapower/prod/business.log
slogs=/opt/docs/datapower/prod/system.log

starttime=$(date +'%H')
currmin=$(date +'%M')
curdate=`date|cut -d' ' -f5`
echo $(date)
if [ $currmin -le 29 ] && [ $starttime -ne 00 ] ; then
starttime1=`echo "$(date +'%H') - 1" | bc`
logtime="$starttime1"
logtime="$logtime:[3-5][0-9]"
echo $logtime

elif [ $currmin -le 29 ] && [ $starttime -eq 00 ] ; then
logtime="23:[3-5][0-9]"
echo $logtime

else
logtime="$starttime"
logtime="$logtime:[0-2][0-9]"
echo $logtime
fi
if ( grep "$logtime" $slogs | egrep "AAA Authentication Failure|AAA Authorization Failure") > dptest 2>&1;then
       Do something
fi 

更新：添加示例日志语句。下面是日志语句的示例： Nov 20 06:06:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>

Answer 1

我认为你做得有点倒退 - 构建一个 RE 以从日志文件中 grep 日期。

在perl中解决这个问题，我希望读取整个日志文件，对其进行标记 - 提取时间戳 - 然后根据消息内容发出警报。

Perl 在第一部分有一个很好的模块 - Time::Piece 。它有点像这样：

use strict;
use warnings;
use Time::Piece;
my $HALF_HOUR = 30 * 60;
while (<DATA>) {
    #extract timestamp via regular expression
    my ( $timestamp, $message ) = (m/A(w+s+d+s+d+:d+:d+) (.*)/);
    #convert text timestamp to 'unix time'. 
    #need the year in here because your log doesn't include it. 
    my $t = localtime();
    $t = $t->strptime( $timestamp . " " . $t->year, "%b %d %H:%M:%S %Y" );

    #skip if parsed time is more than half an hour ago. 
    next if ( $t < time() - $HALF_HOUR );
    if (   $message =~ m/AAA Authentication failure/i
        or $message =~ m/AAA Authorization failure/i )
    {
        print "Alert: ( $t )  $messagen";
    }
}
__DATA__
Nov 20 13:46:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
Nov 20 13:00:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
Nov 20 10:06:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>

后续问：

"你能解释一下这个说法的作用吗，my ( $timestamp, $message ) = (m/A(w+s+d+s+d+:d+:d+) (.*)/);

这样做有两件事：

• Perl 中的一个技巧是，你可以通过将正则表达式放在括号中来捕获正则表达式的某些部分。所以A(w+s+d+s+d+:d+:d+) - 将从行首匹配：
  • 一个或多个"单词"字符。
  • 一个或多个"数字"
  • d+:d+:d+将捕捉到一个时间。（任意 3 个冒号分隔的数字）。

当然，另一部分捕捉了"其余的"。

• 然后，我们将模式匹配返回的数组分配给命名变量数组（$timestamp 和 $message ）。

最终结果是 - 给定以下行：

 Nov 20  13:46:58       business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
(w+ d+ d+:d+:d+)   (.*)

我们的正则表达式分别返回两个"块"，然后将它们放入两个变量中。

Answer 2

你对使用 sqlite3 做过滤工作有什么看法 - 它为你解析时间的好处可能非常方便。唯一的背面是您必须规范化数据。

function sqlite-filter-time() {
    if [ '0' = "$#" ]; then
        echo "Usage: $FUNCNAME <file> <timespan> <where>"
        return
    fi
    local year="$(date '+%Y')"
    local ofs='___FS___'
    sed  "s,^([^ ]* [^ ]*) ([^ ]*),1 2$ofs," "$1" | sed "s,Jan ,$year-01-,;s,Feb ,$year-02-,;s,Mar ,$year-03-,;s,Apr ,$year-04-,;s,May ,$year-05-,;s,June ,$year-06-,;s,July ,$year-07-,;s,Aug ,$year-08-,;s,Sep ,$year-09-,;s,Oct ,$year-10-,;s,Nov ,$year-11-,;s,Dec ,$year-12-," > "$1.tmp" # normalize data for sqlite - command to extract the date and the rest of the text
    {
        echo '.mode csv'
        echo 'DROP TABLE IF EXISTS sft;'
        echo 'CREATE TEMPORARY TABLE sft ('
        echo '  sft_date TEXT,'
        echo '  sft_text TEXT'
        echo ');'
        echo ".headers off"
        echo ".nullvalue ''"
        echo ".separator '$ofs'"
        echo ".import $1.tmp sft"
        echo ".separator ' '"
        echo "SELECT *"
        echo "FROM sft"
        echo "WHERE sft_date > datetime('now', '$2')"
        echo " AND (sft_text like '%AAA Authentication Failure%'"
        echo "   OR sft_text like '%AAA Authorization Failure%'"
        echo " )"
        echo ";"
    } | sqlite3
    rm "$1.tmp"
}
$ sqlite-filter-time "$slogs" '-30 minutes'
"2014-11-20 16:01:58" " business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>"
$