kafka high-level consumer 多線程訪問異常

在使用kafka high-level的consumer，使用多線程消費數據時報錯，簡單分析一下原因下載 ，ConsumerIterator取不到消息時會阻塞，並且將內部狀態置爲FAILED，當其他線程訪問時就會拋出異常。

 

 

Java代碼  

 def hasNext(): Boolean = {
    if(state == FAILED)         //處於FAILED狀態時，另外線程訪問會直接異常
      throw new IllegalStateException("Iterator is in failed state")
    state match {
      case DONE => false
      case READY => true
      case _ => maybeComputeNext()
    }
  }


  def maybeComputeNext(): Boolean = {
    state = FAILED              //重置了狀態
    nextItem = Some(makeNext())
    if(state == DONE) {
      false
    } else {
      state = READY
      true
    }
  }
  下載

protected def makeNext(): MessageAndMetadata[K, V] = {
    var currentDataChunk: FetchedDataChunk = null
    // if we don't have an iterator, get one
    var localCurrent = current.get()
    if(localCurrent == null || !localCurrent.hasNext) {
      if (consumerTimeoutMs < 0)
        currentDataChunk = channel.take             //channel是BlockingQueue這裏會阻塞

      else {
        currentDataChunk = channel.poll(consumerTimeoutMs, TimeUnit.MILLISECONDS)
        if (currentDataChunk == null) {
          // reset state to make the iterator re-iterable
          resetState()
          throw new ConsumerTimeoutException
        }
      }
//省略部分代碼
}