對變名日誌的處理,對於java工程可以簡單修改log4j的配置文件,增加flume配置即可
對於非java程序,採取規避的方案,當然如果你足夠強大可以重寫編譯flume代碼,自定義一個source type支持,將spoolDir 類型的source加強
監控變化的文件。
採用tail -F收集日誌,在空閒時刻將進程重啓,收集指定日誌即可
樣例:
1:client-agent配置
a1.sinks = remotesink
a1.channels = c1
a1.sources = r1
a1.sources.r1.type = exec
a1.sources.r1.shell = /bin/bash -c
a1.sources.r1.command = /usr/local/flume/conf/flume.sh
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = ip
a1.sources.r1.interceptors.i1.value = 192.168.254.162
a1.sources.r1.interceptors.i2.type = static
a1.sources.r1.interceptors.i2.key = NodeName
a1.sources.r1.interceptors.i2.value = 1mynode
a1.sources.r1.interceptors.i3.type = static
a1.sources.r1.interceptors.i3.key = ModuleType
a1.sources.r1.interceptors.i3.value = miner
a1.channels.c1.type = memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 100
a1.sinks.remotesink.type = avro
a1.sinks.remotesink.hostname = 192.168.252.167
a1.sinks.remotesink.port = 44446
a1.sinks.remotesink.channel = c1
2:flume.sh
#!/bin/bash
tail -F /usr/local/flume/$(date '+%Y%m%d').log
3:定時任務
#!/bin/bash
ps aux|grep flume|grep -v grep|awk '{print $2}'|xargs kill -9
/usr/local/flume/bin/flume-ng agent --conf-file /usr/local/flume/conf/client.conf --name a1 -Dflume.root.logger=INFO,LOGFILE -c /usr/local/flume/conf/ -Dflume.monitoring.type=http -Dflume.monitoring.port=34545
4:計劃任務
0 0 * * * sh /usr/local/flume/conf/kill.sh >/dev/null
1 0 * * * sh /usr/local/flume/conf/restart.sh >/dev/null