問題描述：

實時處理程序：streaming消費kafka一段時間後，報錯：ERROR kafka.server.ReplicaManager: [Replica Manager on Broker 105]: Error processing fetch operation on partition [TOPICXXXX,1] offset 57479118
java.lang.IllegalStateException: Failed to read complete buffer for targetOffset 61099082 startPosition 2147483494 in /opt/data/kafka/data/TOPICXXXX-1/00000000000043109340.log

2019-01-10 10:05:01,732 ERROR kafka.server.ReplicaManager: [Replica Manager on Broker 105]: Error processing fetch operation on partition [CLLSEN-BRIDGE-HF-STRAIN,1] offset 57479118
java.lang.IllegalStateException: Failed to read complete buffer for targetOffset 61099082 startPosition 2147483494 in /opt/data/kafka/data/CLLSEN-BRIDGE-HF-STRAIN-1/00000000000043109340.log
at kafka.log.FileMessageSet.searchFor(FileMessageSet.scala:137)
at kafka.log.LogSegment.translateOffset(LogSegment.scala:106)
at kafka.log.LogSegment.read(LogSegment.scala:151)
at kafka.log.Log.read(Log.scala:532)
at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:543)
at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:513)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Map$Map4.foreach(Map.scala:188)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:513)
at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:468)
at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:496)
at kafka.server.KafkaApis.handle(KafkaApis.scala:77)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
2019-01-10 10:05:01,733 ERROR kafka.server.ReplicaManager: [Replica Manager on Broker 105]: Error processing fetch operation on partition [CLLSEN-BRIDGE-HF-STRAIN,1] offset 57479118
java.lang.IllegalStateException: Failed to read complete buffer for targetOffset 61099082 startPosition 2147483494 in /opt/data/kafka/data/CLLSEN-BRIDGE-HF-STRAIN-1/00000000000043109340.log
at kafka.log.FileMessageSet.searchFor(FileMessageSet.scala:137)
at kafka.log.LogSegment.translateOffset(LogSegment.scala:106)
at kafka.log.LogSegment.read(LogSegment.scala:151)
at kafka.log.Log.read(Log.scala:532)
at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:543)
at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:513)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Map$Map4.foreach(Map.scala:188)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:513)
at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:468)
at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:496)
at kafka.server.KafkaApis.handle(KafkaApis.scala:77)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)

實時處理程序：storm消費kafka一段時間後，也報錯以上錯誤！

程序消費不過來嗎還是別的原因呢?

原因分析：

由於程序消費不過來，kafka中的消息數據已經過時了。但是我配置的是保留7天啊？而且也沒有配置消息保存的大小。重新修改groupid消費也是重最新的消息（auto.offset.reste=latest）；怎麼瘋狂的報錯，由於數據量很大，導致日誌非常快：1小時將近20G的日誌，沒過幾天服務器就磁盤滿了，集羣都崩潰了。。。。