Shell-Bash 根据日期时间统计消息行数

清疚 2022-05-28 00:54 233阅读 0赞

需求：统计某个文件或者特定目录下Kafka消息一段时间范围内每个小时消息条数或者每天消息条数。

一行数据行实例：2018-03-28T03：00：02 ...

分析：

参数：1.指定文件的文件绝对路径

2.指定日期或者开始日期和结束日期

3.指定小时或者开始小时和结束小时

4.指定统计维度（小时或者天）

逻辑：1.输出逻辑 -- 文件 + 屏幕打印

2.统计逻辑 -- grep + wc -l

3.按天范围统计逻辑

4.按小时范围统计逻辑

5.按统计维度选择3/4的函数

6.文件分类处理（单个文件或者特定目录下文件）

7.对用户输入的日期进行判断和初始化处理

8.对用户输入的小时进行判断和初始化处理

9.辅助函数，修剪、加前缀0、自定义LOGGER（检查各函数参数）、文件名格式化

扩展：1.增加目录参数，不再限定单一目录。

2.绝对路径和相对路径均适用。

3.对特大文件进行按行拆分处理后再统计。

4.兼容时间戳格式的时间范围。

Shell阶段性总结：

1. Shell脚本调试全局开启，首行声明 \#!/bin/bash -xv .

2.获取脚本目录, $( cd "$(dirname $0)" && pwd) .

3.参数选项设置，-o为短参数列表，-l为长参数列表，双引号内不带冒号的为不能带参数值的参数，带冒号的必须带参数值.

OPTS=\`getopt -o "s:,e:,dd:,df:,sh:,eh:,dh:,hd,h" -l "start-date:,end-date:,designed-date:,designed-file:,start-hour:,end-hour:,designed-hour:,hour-dimension,help" -- "$@"\`

eval set -- "$OPTS"

4.Trim函数在shell中实现，%%为右边最长匹配去除，\#\#为左边最长匹配去除.

trimmed=""  
function trim()\{  
trimmed=$1  
trimmed=$\{trimmed%%\\ \}  
trimmed=$\{trimmed\#\#\\ \}  
trimmed=$\{trimmed%%\\t\}  
trimmed=$\{trimmed\#\#\\t\}

5.数值比较用符号应使用 （（>,>=,<,<=,=）），用字母应该使用\[ -eq,-le,-ge,-lt,-gt,-ne \] .

6.字符串比较使用\[ -z,-n,=,==,>,<\]，其中>,<比较字母ASCII顺序.

7.获取距离指定日期后i天的日期，转换成特定格式字符串，$(date +%Y-%m-%d --date="$\{from\_date\} + $\{i\} day") .

8.获取指定日期的时间戳，$(date -d "$to\_date" +%s)，用于日期比较.

9.函数名变量，$FUNCNAME；函数名调用栈数组 $\{FUNCNAME\[@\]\}，如 another\_func test\_func main ；主要用于日志定位.

10.函数中局部变量声明，local designed\_date .

11.测试使用全部写在USAGE的Example里面的例子，因此Example要写的较为全面，同时写在Example中便于随时补充。

12.help中USAGE变量填充，<< EOF .. EOF，告诉主shell，后续的输入，是其他命令或者子shell的输入，直到遇到EOF为止，再回到主shell.

read -d '' USAGE << EOF  
\#\#填充具体的使用说明

EOF

附Shell脚本：

\#!/bin/bash  
read -d '' USAGE << EOF  
USAGE  
$0 \[--start-date=yyyy-mm-dd --end-date=yyyy-mm-dd\] \[--designed-date=yyyy-mm-dd\] \[--start-hour=HH --end-hour=HH\] \[--designed-hour=HH\] \[--designed-file=absulte path\] \[--hour-dimension\] \[--help\]  
  
  
DESCRIPTION  
This script is for count lines between start-date&start-hour and end-date&end-hour.  
Or count lines between designed-date&start-hour and designed-date&end-hour.  
Or count lines at designed-date&designed-hour.  
Notice, this script can not count greater than 30 days.  
The following arguments are optional.  
  
  
\--start-date  
The begin of date window which lines should be counted, cooperate with --start-hour,--end-date,--end\_hour.  
If not specified, it is set to today by default.  
  
  
\--end-date  
The end of date window which lines should be counted, cooperate with --start-hour,--start-date,--end\_hour.  
If not specified, it is set to today by default.  
  
\--start-hour  
The begin of hour window which lines should be counted, cooperate with (--start-date,--end-date) or --designed-date,--end\_hour.  
If not specified, it is set to 0 by default.  
\--end-hour  
The end of hour window which lines should be counted, cooperate with --start-hour,(--end-date,--end\_date) or --designed-date.  
If not specified, it is set to now hour by default.  
\--designed-date  
The designed date to be counted.  
If not specified, use --start-date and --end-date as default.  
\--designed-hour  
The designed hour to be counted.  
If not specified, use --start-hour and --end-hour as default.  
\--designed-file  
The absulte path of file to be counted.  
If not specified, all path under absulte path /data/rawdata/kafka/ will be checked.  
\--help  
Display this HELP information  
  
  
EXAMPLE:  
$0 --designed-date=2017-10-25  
$0 --designed-date=2017-10-25 --designed-hour=06 --hour-dimension  
$0 --designed-date=2017-10-25 --designed-file=/data/123.txt  
$0 --designed-date=2017-10-25 --designed-hour=06 --designed-file=/data/123.txt --hour-dimension  
$0 --start-date=2017-10-25 --end-date=2017-10-26  
$0 --start-date=2017-10-25 --end-date=2017-10-26 --start-hour=06 --end-hour=07 --hour-dimension  
$0 --start-date=2017-10-25 --end-date=2017-10-26 --designed-file=/data/123.txt  
$0 --start-date=2017-10-25 --end-date=2017-10-26 --start-hour=06 --end-hour=07 --designed-file=/data/123.txt --hour-dimension  
EOF  
  
  
start\_date=""  
end\_date=""  
designed\_date=""  
designed\_file=""  
start\_hour=0  
end\_hour=0  
designed\_hour=0  
hour\_dimension="False"  
  
  
OPTS=\`getopt -o "s:,e:,dd:,df:,sh:,eh:,dh:,hd,h" -l "start-date:,end-date:,designed-date:,designed-file:,start-hour:,end-hour:,designed-hour:,hour-dimension,help" -- "$@"\`  
eval set -- "$OPTS"  
  
  
while true  
do  
case "$1" in  
\--start-date ) start\_date="$2"; shift 2 ;;  
\--end-date ) end\_date="$2"; shift 2 ;;  
\--designed-date ) designed\_date="$2"; shift 2 ;;  
\--designed-file ) designed\_file="$2"; shift 2 ;;  
\--start-hour ) start\_hour="$2"; shift 2 ;;  
\--end-hour ) end\_hour="$2"; shift 2 ;;  
\--designed-hour ) designed\_hour="$2"; shift 2 ;;  
\--hour-dimension ) hour\_dimension="True"; shift ;;  
\--help )  
echo "$USAGE"  
shift ;  
exit 0 ;;  
\-- ) shift; break ;;  
\* ) break ;;  
esac  
done  
  
  
  
  
now\_date=$(date "+%Y-%m-%d");  
now\_hour=$(date "+%H");  
now\_time=\`date "+%Y-%m-%d %H:%M:%S"\`  
IS\_FILE\_OUT=1  
ALLOWED\_MAX\_PERIOD=30  
DIR="$( cd "$( dirname "$0" )" && pwd )"  
echo "Today is $now\_date,now hour is $now\_hour"  
echo "Script directory:$DIR"  
  
  
num=""  
function add\_prefix\_zero()\{  
case "$1" in  
\[0-9\]\*) ;;  
\*) echo "Error:not a number."; exit 1;;  
esac  
num=$1  
if \[ $num -ge 0 \] && \[ $num -le 9 \] ;then  
num="0$num"  
fi  
\}  
trimmed=""  
function trim()\{  
trimmed=$1  
trimmed=$\{trimmed%%\\ \}  
trimmed=$\{trimmed\#\#\\ \}  
trimmed=$\{trimmed%%\\t\}  
trimmed=$\{trimmed\#\#\\t\}  
\}  
  
  
IS\_INFO\_OUT=0  
function info()\{  
if \[ $IS\_INFO\_OUT -eq 1 \] ; then  
echo "\[\[ Function=$1 --> Info=$2 \]\]"  
fi  
\}  
  
  
saved\_filename=""  
function format\_filename()\{  
info "$FUNCNAME" "from\_date:$1 to\_date:$2 from\_hour:$3 to\_hour:$4"  
local from\_date=$1  
local to\_date=$2  
local from\_hour=$3  
local to\_hour=$4  
if \[ -z $designed\_file \] ; then  
saved\_filename="$from\_date-$from\_hour--$to\_date-$to\_hour-\`echo $now\_time | md5sum | cut -d- -f 1\`"  
trim $saved\_filename  
saved\_filename="$trimmed.txt"  
else  
 file\_prefilx=$\{designed\_file\#\#\*/\}  
saved\_filename="$\{file\_prefilx%%\\.txt\}-$from\_date-$from\_hour--$to\_date-$to\_hour-\`echo $now\_time | md5sum | cut -d- -f 1\`"  
trim $saved\_filename  
saved\_filename="$trimmed.txt"  
fi  
info "$FUNCNAME" "saved\_filename:$saved\_filename"  
\}  
  
  
  
  
function out\_sum\_date\_hour()\{  
info "$FUNCNAME" "designed\_file:$1 designed\_date:$2 designed\_hour:$3"  
local designed\_file=$1  
local designed\_date=$2  
local designed\_hour=$3  
local output\_message=""  
if \[ 0 -eq $IS\_FILE\_OUT \] ; then  
 output\_message="Filename:$\{designed\_file\} Date:$\{designed\_date\} $\{designed\_hour+Hour:$designed\_hour\} Count:\`grep "$\{designed\_date\}$\{designed\_hour+T$designed\_hour\}" $\{designed\_file\} | wc -l\`"  
echo $output\_message  
else  
 output\_message="Filename:$\{designed\_file\} Date:$\{designed\_date\} $\{designed\_hour+Hour:$designed\_hour\} Count:\`grep "$\{designed\_date\}$\{designed\_hour+T$designed\_hour\}" $\{designed\_file\} | wc -l\`"  
 echo $output\_message  
echo $output\_message >> $DIR/"$saved\_filename"  
fi  
\}  
  
  
function file\_from\_to\_date()\{  
info "$FUNCNAME" "from\_date:$1 to\_date:$2 designed\_file:$3"  
local from\_date=$1  
local to\_date=$2  
local designed\_file=$3  
local to\_timestamp=\`date -d "$to\_date" +%s\`  
local someday=""  
local timestamp=0  
for ((i=0; i<=$ALLOWED\_MAX\_PERIOD; i++))  
do  
someday=$(date +%Y-%m-%d --date="$\{from\_date\} + $\{i\} day")  
  
out\_sum\_date\_hour $designed\_file $someday  
  
timestamp=\`date -d "$someday" +%s\`  
if \[ $timestamp -eq $to\_timestamp \] ; then  
break  
fi  
done  
\}  
  
  
function file\_from\_to\_date\_hour()\{  
info "$FUNCNAME" "from\_date:$1 to\_date:$2 from\_hour:$3 to\_hour:$4 designed\_file:$5"  
local from\_date=$1  
local to\_date=$2  
local from\_hour=$3  
local to\_hour=$4  
local designed\_file=$5  
local to\_timestamp=\`date -d "$to\_date" +%s\`  
for ((i=0; i<=$ALLOWED\_MAX\_PERIOD; i++))  
do  
someday=$(date +%Y-%m-%d --date="$\{from\_date\} + $\{i\} day")  
timestamp=\`date -d "$someday" +%s\`  
for ((j=$from\_hour; j<=$to\_hour; j++))  
do  
add\_prefix\_zero $j  
out\_sum\_date\_hour $designed\_file $someday $num  
done  
if \[ $timestamp -eq $to\_timestamp \] ; then  
break  
fi  
done  
\}  
  
  
function judge\_date\_hour()\{  
info "$FUNCNAME" "from\_date:$1 to\_date:$2 from\_hour:$3 to\_hour:$4 designed\_file:$5"  
local from\_date=$1  
local to\_date=$2  
local from\_hour=$3  
local to\_hour=$4  
local designed\_file=$5  
if \[ "$hour\_dimension" == "True" \] ; then  
file\_from\_to\_date\_hour $from\_date $to\_date $from\_hour $to\_hour $designed\_file  
  
else  
file\_from\_to\_date $from\_date $to\_date $designed\_file  
fi  
\}  
  
  
function from\_to\_date\_hour()\{  
info "$FUNCNAME" "from\_date:$1 to\_date:$2 from\_hour:$3 to\_hour:$4"  
local from\_date=$1  
local to\_date=$2  
local from\_hour=$3  
local to\_hour=$4  
local to\_timestamp=\`date -d "$to\_date" +%s\`  
if \[ ! -z $designed\_file \] ; then  
judge\_date\_hour $from\_date $to\_date $from\_hour $to\_hour $designed\_file  
else  
cd /data/rawdata/kafka/  
for file in $(ls \*.txt\*)  
do  
if \[ ! -d $file \]; then  
judge\_date\_hour $from\_date $to\_date $from\_hour $to\_hour $file  
fi  
done  
fi  
\}  
  
  
  
  
function judge\_default\_date()\{  
info "$FUNCNAME" "start\_date:$start\_date end\_date:$end\_date"  
if \[ ! -z $start\_date \] || \[ ! -z $end\_date \] ; then  
if \[ -z $start\_date \] ; then  
start\_date=$now\_date  
fi  
if \[ -z $end\_date \] ; then  
end\_date=$now\_date  
fi  
return 1  
else  
return 0  
fi  
\}  
  
  
function judge\_default\_hour()\{  
info "$FUNCNAME" "start\_hour:$start\_hour end\_hour:$end\_hour"  
if \[ ! 0 -eq $start\_hour \] || \[ ! 0 -eq $end\_hour \] ; then  
if \[ 0 -eq $start\_hour \] ; then  
start\_hour=0  
fi  
if \[ 0 -eq $end\_hour \] ; then  
end\_hour=$now\_hour  
fi  
return 1  
else  
return 0  
fi  
\}  
  
  
\#\#\# \\$start\_date and \\$end\_date can not coexist with \\$designed\_date  
if \[ ! -z $start\_date \] || \[ ! -z $end\_date \] && \[ ! -z $designed\_date \] ; then  
exit 1  
fi  
\#\#\# \\$start\_hour and \\$end\_hour can not coexist with \\$designed\_hour  
if \[ ! 0 -eq $start\_hour \] || \[ ! 0 -eq $end\_hour \] && \[ ! 0 -eq $designed\_hour \] ; then  
exit 1  
fi  
  
  
if \[ -z $start\_date \] && \[ -z $end\_date \] && \[ -z $designed\_date \] ; then  
start\_date=$now\_date  
end\_date=$now\_date  
fi  
  
  
if \[ 0 -eq $start\_hour \] && \[ 0 -eq $end\_hour \] && \[ 0 -eq $designed\_hour \] ; then  
start\_hour=0  
end\_hour=$now\_hour  
fi  
  
  
  
  
judge\_default\_date  
if \[ $? -eq 1 \] ; then  
judge\_default\_hour  
if \[ $? -eq 1 \] ; then  
format\_filename $start\_date $end\_date $start\_hour $end\_hour  
from\_to\_date\_hour $start\_date $end\_date $start\_hour $end\_hour  
else  
format\_filename $start\_date $end\_date $designed\_hour $designed\_hour  
from\_to\_date\_hour $start\_date $end\_date $designed\_hour $designed\_hour  
fi  
else  
judge\_default\_hour  
if \[ $? -eq 1 \] ; then  
format\_filename $designed\_date $designed\_date $start\_hour $end\_hour  
from\_to\_date\_hour $designed\_date $designed\_date $start\_hour $end\_hour  
else  
format\_filename $designed\_date $designed\_date $designed\_hour $designed\_hour  
from\_to\_date\_hour $designed\_date $designed\_date $designed\_hour $designed\_hour  
fi  
fi