flume-ng 使用系列

对变名日志的处理,对于java工程可以简单修改log4j的配置文件,增加flume配置即可

对于非java程序,采取规避的方案,当然如果你足够强大可以重写编译flume代码,自定义一个source type支持,将spoolDir 类型的source加强

监控变化的文件。


采用tail -F收集日志,在空闲时刻将进程重启,收集指定日志即可


样例:

1:client-agent配置

a1.sinks = remotesink
a1.channels = c1
a1.sources = r1

a1.sources.r1.type = exec
a1.sources.r1.shell = /bin/bash -c
a1.sources.r1.command = /usr/local/flume/conf/flume.sh
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = ip
a1.sources.r1.interceptors.i1.value = 192.168.254.162
a1.sources.r1.interceptors.i2.type = static
a1.sources.r1.interceptors.i2.key = NodeName
a1.sources.r1.interceptors.i2.value = 1mynode
a1.sources.r1.interceptors.i3.type = static
a1.sources.r1.interceptors.i3.key = ModuleType
a1.sources.r1.interceptors.i3.value = miner

a1.channels.c1.type = memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 100

a1.sinks.remotesink.type = avro
a1.sinks.remotesink.hostname = 192.168.252.167
a1.sinks.remotesink.port = 44446
a1.sinks.remotesink.channel = c1


2:flume.sh

#!/bin/bash
tail -F /usr/local/flume/$(date '+%Y%m%d').log


3:定时任务

#!/bin/bash
ps aux|grep flume|grep -v grep|awk '{print $2}'|xargs kill -9

/usr/local/flume/bin/flume-ng agent --conf-file /usr/local/flume/conf/client.conf --name a1 -Dflume.root.logger=INFO,LOGFILE -c /usr/local/flume/conf/ -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

4:计划任务

0 0 * * * sh /usr/local/flume/conf/kill.sh >/dev/null
1 0 * * * sh /usr/local/flume/conf/restart.sh >/dev/null



發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章