flume-ng 使用系列

對變名日誌的處理,對於java工程可以簡單修改log4j的配置文件,增加flume配置即可

對於非java程序,採取規避的方案,當然如果你足夠強大可以重寫編譯flume代碼,自定義一個source type支持,將spoolDir 類型的source加強

監控變化的文件。


採用tail -F收集日誌,在空閒時刻將進程重啓,收集指定日誌即可


樣例:

1:client-agent配置

a1.sinks = remotesink
a1.channels = c1
a1.sources = r1

a1.sources.r1.type = exec
a1.sources.r1.shell = /bin/bash -c
a1.sources.r1.command = /usr/local/flume/conf/flume.sh
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = ip
a1.sources.r1.interceptors.i1.value = 192.168.254.162
a1.sources.r1.interceptors.i2.type = static
a1.sources.r1.interceptors.i2.key = NodeName
a1.sources.r1.interceptors.i2.value = 1mynode
a1.sources.r1.interceptors.i3.type = static
a1.sources.r1.interceptors.i3.key = ModuleType
a1.sources.r1.interceptors.i3.value = miner

a1.channels.c1.type = memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 100

a1.sinks.remotesink.type = avro
a1.sinks.remotesink.hostname = 192.168.252.167
a1.sinks.remotesink.port = 44446
a1.sinks.remotesink.channel = c1


2:flume.sh

#!/bin/bash
tail -F /usr/local/flume/$(date '+%Y%m%d').log


3:定時任務

#!/bin/bash
ps aux|grep flume|grep -v grep|awk '{print $2}'|xargs kill -9

/usr/local/flume/bin/flume-ng agent --conf-file /usr/local/flume/conf/client.conf --name a1 -Dflume.root.logger=INFO,LOGFILE -c /usr/local/flume/conf/ -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

4:計劃任務

0 0 * * * sh /usr/local/flume/conf/kill.sh >/dev/null
1 0 * * * sh /usr/local/flume/conf/restart.sh >/dev/null



發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章