一、问题描述
flume监控日志，同时，发送到flume监控的avro端口，当大量数据1000万条数据适合，flume监控日志的报错。然后，查看flume监控界面，发现flume监控界面消费突然消失。查看监控avro端口的agent的日志报错
Avro source avro_source: Unable to process event batch. Exception follows.
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:402)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
at org.apache.avro.ipc.Responder.respond(Responder.java:151)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
二、问题原因
日志信息报错，很清楚，是由于Channel的大小报错的，容量不足：
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
而导致容量不足的原因如下：memory chanell的结构如下
因为“source往putList放数据，然后提交到queue中”与“sink从channel中取数据到sink和takeList，然后再从putList取数据到queue中”这两部分是分开来，任他们自由抢锁，所以，当前者多次抢到锁，后者没有抢到锁，同时queue的大小又太小，撑不住多次往里放数据，就会导致触发这个异常。
解决这个问题最直接的办法就是增大queue的大小，增大capacity和transacCapacity之间的差距，queue能撑住多次往里面放数据即可。
三、解决办法
增加agent的容量
avro_memory_kafka.channels.memory_channel.type = memory
修改为
avro_memory_kafka.channels.memory_channel.type = memory
avro_memory_kafka.channels.memory_channel.keep-alive = 60
avro_memory_kafka.channels.memory_channel.transactionCapacity = 1000
avro_memory_kafka.channels.memory_channel.capacity = 1000000
其中
|
type |
- |
组件类型名称必须是memory |
|
capacity |
100 |
存储在 Channel 当中的最大 events 数 |
|
transactionCapacity |
100 |
同时刻从Source 获取，或发送到 Sink 的最大 events 数 |
|
keep-alive |
3 |
添加或删除一个 event 超时的秒数 |
同时，失败后，flume会暂停source向channel放数据，等待几秒钟，这期间sink应该会消费channel中的数据，当source再次开始想channel放数据时channel就有足够的空间了。
四、参考
1.https://blog.csdn.net/gaopu12345/article/details/77922924
2.https://www.cnblogs.com/justinyang/p/8675414.html