flume的ChannelExceptio以及memeryChannel中transactionCapacity和sink的batchsize需要注意事項

最近在做flume的日誌收集,在用flume的時候發現一個報錯.

14 Mar 2020 14:23:58,194 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.AbstractRpcSink.process:384)  - Rpc Sink k1: Unable to get event from channel c1. Exception follows.
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
	at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:95)
	at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
	at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:351)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:745)

看了一下自己的配置,發現配置裏這麼寫的

# Name the components on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1

# Describe/configure the source
a2.sources.r1.type = exec
a2.sources.r1.command = tail -F /home/logs/spring.log
a2.sources.r1.shell = /bin/bash -c

# Describe the sink
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = Linux1
a2.sinks.k1.port = 4141
a1.sinks.k1.batch-size = 1000

# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

查閱資料發現,是batchsize和transactionCapacity 這兩個參數起了衝突.
這個sink的batchsize是什麼意思呢，就是sink會一次從channel中取多少個event去發送，而這個發送是要最終以事務的形式去發送的，因此這個batchsize的event會傳送到一個事務的緩存隊列中（takeList），這是一個雙向隊列，這個隊列可以在事務失敗時進行回滾（也就是把取出來的數據吐memeryChannel的queue中），它的初始大小就是transactionCapacity定義的大小，源碼中有： takeList = new LinkedBlockingDeque(transCapacity); 源碼來自https://segmentfault.com/a/1190000003586635的分享。

再看這個錯誤拋出的地方：

if(takeList.remainingCapacity() == 0) {
throw new ChannelException("Take list for MemoryTransaction, capacity " +
takeList.size() + " full, consider committing more frequently, " +
“increasing capacity, or increasing thread count”);
}

在上面的情況中，sink一次取1000個events，塞到takelist中，在塞了100個後，就會引發上述異常，因此，這個錯誤的解決辦法就是：在sink中，channel的transactionCapacity參數不能小於sink的batchsize

參考:flume的memeryChannel中transactionCapacity和sink的batchsize需要注意事項
Flume MemoryChannel源碼分析