Scribe分布式日志系统

落日映苍穹つ 2022-01-16 23:45 290阅读 0赞

[为什么80%的码农都做不了架构师？>>> ][80_]  ![hot3.png][]

Scribe分布式日志系统 一．介绍 Scribe是facebook开发的分布式日志系统，使用thrift传输log，因此无论是什么语言开发的项目都可以实现log收集，传送到远程或主从同步到远程。Scribe由两部分组成：central scribe server和local scribe server。在分布式系统中，每一个节点都会运行一个local scribe server，收集此节点的日志信息，并将其发送给central scribe server。 各个数据源需通过thrift向scribe传输数据，每条数据记录包含一个category和一个message。可以在scribe配置中指定thrift线程数，默认是3。在后端，scribe可以将不同category的数据存放到不同目录中，以便于进行分别处理。后端的日志存储方式可以是各种各样的store，包括file，buffer（双层存储，一个主存储，一个副存储），network（另一个scribe服务器），bucket（包含多个store，通过hash将数据存到不同store中），null（忽略数据），thriftfile（写到一个thrift tfile transport文件中）和multi（把数据同时存放到不同store中）。 [![scribe][]][scribe 1] 二．安装 \# yum install gcc-c++ libevent libevent-devel automake autoconf m4 bison zlib zlib-devel bzip2 bzip2-devel flex pkgconfig python python-devel ruby ruby-devel mono-devel libxml2 libxml2-devel ant openssl-devel \# wget http://nchc.dl.sourceforge.net/project/boost/boost/1.45.0/boost\_1\_45\_0.tar.gz \# tar zxvf boost\_1\_45\_0.tar.gz -C ../software/ \# ./bjam \# ./bjam -s HAVE\_ICU=1 --prefix=/usr/local/boost-1.45 --includedir=/usr/local/boost-1.45/include --libdir=/usr/local/boost-1.45/lib \# ./bjam install --prefix=/usr/local/boost-1.45 \# wget http://www.apache.org/dist//thrift/0.7.0/thrift-0.7.0.tar.gz \# tar zxvf thrift-0.7.0.tar.gz -C ../software/ \# ./configure --prefix=/usr/local/thrift-0.7.0 --with-boost=/usr/local/boost-1.45 \# make \# make install \# cd ./contrib/fb303/ \# ./bootstrap.sh --with-boost=/usr/local/boost-1.45 \# ./configure --prefix=/usr/local/thrift-0.7.0/fb303 --with-boost=/usr/local/boost-1.45 --with-thriftpath=/usr/local/thrift-0.7.0 \# make \# make install https://github.com/pcting/scribe/downloads 下载pcting-scribe-2ee14d3.tar.gz \# tar zxvf pcting-scribe-2ee14d3.tar.gz -C ../software/ \# export BOOST\_ROOT=/usr/local/boost-1.45 \# export LD\_LIBRARY\_PATH=/usr/local/thrift-0.7.0/lib:/usr/lib:/usr/local/lib:/usr/local/boost-1.45/lib:/usr/local/thrift-0.7.0/fb303/lib \# ./bootstrap.sh --with-boost=/usr/local/boost-1.45 \# ./configure --prefix=/usr/local/scribe --with-boost=/usr/local/boost-1.45 --with-thriftpath=/usr/local/thrift-0.7.0 --with-fb303path=/usr/local/thrift-0.7.0/fb303 \# make \# make install 三．配置 \# mkdir etc \# cp /usr/local/src/software/pcting-scribe-2ee14d3/examples/example1.conf ./scribe.conf \# mkdir /tmp/scribetest 配置参数解释如下： 具体参数含义参见：http://sourceforge.net/apps/mediawiki/scribeserver/index.php?title=Configuration //全局配置参数 port=1463 max\_msg\_per\_second=2000000 //每秒最大日志并发数 check\_interval=3 //检测每个store频率 max\_queue\_size=10000000 //最大队列大小，单位是byte new\_thread\_per\_category //是否为每个category创建一个新的线程。默认是ture //存储配置 scribe是根据存储配置来记录信息的。每个store必须指定以下消息类(category)： default store：只能有一个默认store。用来处理任何store没有处理的category prefix store：如果指定category以\*结尾，store将处理任何以指定的前缀的categories category=default //用来决定哪些信息由这个store来处理 type=buffer //类型 Store::createStore target\_write\_size=20480 //在处理消息前允许多大的消息队列增长 单位是byte max\_write\_interval=1 //在处理消息前多久允许消息队列增长间隔 buffer\_send\_rate=2 //从secondary store多少次读取一个组消息发送到primary store retry\_interval=30 //写入到primary store失败后，多久时间再次发送到primary store retry\_interval\_range=10 //重试间隔 fs\_type=std //目前只支持std file\_path=/tmp/scribetest //存储目录 base\_filename=thisisoverwritten //文件名称，默认是category名称 max\_size=1000000 //文件轮转大小 add\_newlines=1 //每条日志是否一行 rotate\_period=daily //轮转周期，hourly，daily，never默认是never 创建新文件的频率 rotate\_hour=0 //0-23，默认是1 如果rotate\_period=daily，几点轮转 rotate\_minute=10 //0-59，默认是15 如果rotate\_period=daily/hourly，几点几分轮转 remote\_host //将日志发送到远程scribe服务器上 remote\_port timeout //socket超时时间，单位ms use\_conn\_pool //是否使用连接池代替开放的多连接，默认是false \# vi scribe.conf port=1463 max\_msg\_per\_second=2000000 check\_interval=3 category=default type=buffer target\_write\_size=20480 max\_write\_interval=1 buffer\_send\_rate=2 retry\_interval=30 retry\_interval\_range=10 type=file fs\_type=std file\_path=/tmp/scribetest base\_filename=xuhh max\_size=1000000 add\_newlines=1 \# ../bin/scribed ./scribe.conf & //开启scribed服务 \# echo "test scribe" | ../bin/scribe\_cat scribe\_test //测试 \# cat /tmp/scribetest/scribe\_test/scribe\_test\_current //查看日志信息 \# ../bin/scribe\_ctrl counters //查看统计信息 scribe\_test:received good: 2 集中存储到scribe-central配置 [![scribe-2][]][scribe-2 1] Central端配置： \# vi scribe-central.conf port=1463 max\_msg\_per\_second=2000000 check\_interval=3 category=ignore\* type=null category=bucket\_me type=buffer target\_write\_size=20480 max\_write\_interval=1 buffer\_send\_rate=2 retry\_interval=30 retry\_interval\_range=10 type=bucket num\_buckets=5 bucket\_subdir=bucket bucket\_type=key\_hash delimiter=58 type=file fs\_type=std file\_path=/tmp/scribetest base\_filename=bucket\_me max\_size=10000 category=default type=buffer target\_write\_size=20480 max\_write\_interval=1 buffer\_send\_rate=2 retry\_interval=30 retry\_interval\_range=10 type=file fs\_type=std file\_path=/tmp/scribetest base\_filename=thisisoverwritten max\_size=1000000 Client端配置： \# vi scribe-client.conf port=1464 max\_msg\_per\_second=2000000 check\_interval=3 category=default type=buffer target\_write\_size=20480 max\_write\_interval=1 buffer\_send\_rate=1 retry\_interval=30 retry\_interval\_range=10 type=network remote\_host=192.168.1.173 remote\_port=1463 \[client\]\# echo "from scribe\_client to scribe\_central" | ../bin/scribe\_cat -h localhost:1464 scribe\_central \[Thu Dec 1 14:30:37 2011\] "Successfully sent messages to remote scribe server " \[client\]\# echo "this message will be ignored" | ../bin/scribe\_cat -h localhost:1464 ignore\_me //将忽略掉 \[client\]\# echo "this message will be bucketed" | ../bin/scribe\_cat -h localhost:1464 bucket\_me \[Thu Dec 1 15:04:21 2011\] "\[bucket\_me\] Creating new category store from model default" \[Thu Dec 1 15:04:21 2011\] "Opened connection to remote scribe server " \[Thu Dec 1 15:04:21 2011\] "\[bucket\_me\] Opened file for writing" \[Thu Dec 1 15:04:21 2011\] "\[bucket\_me\] Changing state from to <SENDING\_BUFFER>" \[Thu Dec 1 15:04:24 2011\] "\[bucket\_me\] read entries of bytes from file " \[Thu Dec 1 15:04:28 2011\] "Successfully sent messages to remote scribe server " \[Thu Dec 1 15:04:28 2011\] "\[bucket\_me\] No more buffer files to send, switching to streaming mode" \[Thu Dec 1 15:04:28 2011\] "\[bucket\_me\] Changing state from <SENDING\_BUFFER> to " \[central\]\# cat /tmp/scribetest/scribe\_central/scribe\_central\_current from scribe\_client to scribe\_central \[central\]\# cat bucket000/bucket\_me\_current this message will be bucketed \[central\]\# /home/xuhh/scribe-2.2/bin/scribe\_ctrl counters bucket\_me:received good: 1 scribe\_overall:received good: 5 scribe\_central:received good: 3 scribe\_overall:ignored: 1 ignore\_me:ignored: 1 ignore\_me:received good: 1 \[client\]\# /usr/local/scribe/bin/scribe\_ctrl counters 1464 bucket\_me:received good: 1 scribe\_overall:received good: 4 scribe\_central:received good: 2 ignore\_me:received good: 1 scribe\_overall:sent: 4 配置PHP接口： 生成PHP库： \# cd /usr/local/scribe/ \# /usr/local/thrift-0.7.0/bin/thrift -o . -I /usr/local/thrift-0.7.0/fb303/share --gen php /usr/local/thrift-0.7.0/fb303/share/fb303/if/fb303.thrift \# /usr/local/thrift-0.7.0/bin/thrift -o . -I /usr/local/thrift-0.7.0/fb303/share/ --gen php /data/src/software/pcting-scribe-2ee14d3/if/scribe.thrift \# cp -r /data/src/software/thrift-0.7.0/lib/php/src includes \# mkdir -p includes/packages/fb303 \# mkdir -p includes/packages/scribe \# mv gen-php/fb303/FacebookService.php ./includes/packages/fb303/ \# mv gen-php/fb303/fb303\_types.php ./includes/packages/fb303/ \# mv gen-php/scribe/scribe\_types.php ./includes/packages/scribe/ \# mv gen-php/scribe/scribe.php ./includes/ 测试程序具体参见：http://www.ruturaj.net/scribe-php-logging

转载于:https://my.oschina.net/766/blog/211501

[80_]: https://my.oschina.net/u/2663968/blog/3051541
[hot3.png]: /images/20220117/be0e8c88ce6046b6ad88e9fcd5c816a7.png
[scribe]: http://static.oschina.net/uploads/img/201403/23012125_w2zY.jpg
[scribe 1]: http://www.ttlsa.com/wp-content/uploads/2011/12/scribe1.jpg
[scribe-2]: /images/20220117/3d884393d4cd4d9eb37aa2cd7a6ad9a6.png
[scribe-2 1]: http://www.ttlsa.com/wp-content/uploads/2011/12/scribe-21.png