尝试flume配置文件从启动命令接收参数

接着上一篇flume接收数据传入hbase。
这次的目的是:
flume配置文件sink指定hbase的表名可以当成参数进行接收,以便于能随外部切换hbase不同的表。
例如在test.conf中

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = http
a1.sources.r1.port = 44444
a1.sources.r1.bind = 10.0.0.183

# Describe the sink
a1.sinks.k1.type = hbase
a1.sinks.k1.channel = c1
#a1.sinks.k1.table = httpdata
a1.sinks.k1.table = ${htable}   #传入表名
a1.sinks.k1.columnFamily = a
a1.sinks.k1.serializer = com.hbase.Rowkey
a1.sinks.k1.channel = memoryChannel

# Use a channel which buffers events in memory
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /home/mathartsys/oyzm_test/flu-hbase/checkpoint/
a1.channels.c1.useDualCheckpoints = false
a1.channels.c1.dataDirs = /home/mathartsys/oyzm_test/flu-hbase/flumedir/
a1.channels.c1.maxFileSize = 2146435071
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 10000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 
a1.sinks.k1.channel = c1

尝试了两种传参方法:
1.从flume命令中携带变量参数进入test.conf中
官网外部传参示例:
配置:

a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = ${NC_PORT}
a1.sources.r1.channels = c1

启动命令:

NC_PORT=44444 bin/flume-ng agent –conf conf –conf-file example.conf –name a1 -Dflume.root.logger=INFO,console -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties

实际上如果在配置文件中配置了sink和channels后,外部参数是无法传入的,
报错提示:
java.lang.NumberFormatException: For input string: “${NC_PORT}”

2.通过flume中的正则拦截器interceptors( regex_extractor)接收hbase表名
source和sink的配置

a1.sources.r1.interceptors = i1  
a1.sources.r1.interceptors.i1.type = regex_extractor
a1.sources.r1.interceptors.i1.regex = (\\d)
a1.sources.r1.interceptors.i1.serializers = s1
a1.sources.r1.interceptors.i1.serializers.s1.name = table

a1.sinks.k1.type = hbase
a1.sinks.k1.channel = c1
a1.sinks.k1.table = ${s1}
a1.sinks.k1.columnFamily = cf
a1.sinks.k1.serializer = com.hbase.Rowkey
a1.sinks.k1.channel = memoryChannel 

写入命令

curl -X POST -d'[{"body":"1"}]'  http://10.0.0.183:44444

报错:Illegal character code:36, <$> at 0. User-space table qualifiers can only contain ‘alphanumeric characters’: i.e. [a-zA-Z_0-9-.]: ${s1}
原因:在.conf配置文件启动后,sinks会直接去连接指定的hbase表。如果想要更换表,只能是重新启动新的conf配置文件

结论:实现flume配置灵活指定hbase表名失败

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章