SparkStreaming+Kafka0.9.0.0多主題多分區偏移量維護

偏移量保存到數據庫

一、版本介紹

具體使用方法，請看這兩篇文章，下面只介紹需要修改的地方
Kafka1.0.x偏移量管理以及維護思路使用等的問題看這一篇文章
https://blog.csdn.net/qq_41922058/article/details/86478250
Kafka2.0.0偏移量管理以及維護思路使用等的問題看這一篇文章
https://blog.csdn.net/qq_41922058/article/details/86545211

二、代碼改動

比較kafka-1.0.x版本，需要改動獲取偏移量範圍的方法，改動處非常小

/**
  * 獲取當前主題的偏移量範圍
  *
  * @param topicName The Topic Name
  * @param MinOrMax
  * @return
  */
def getTopicOffset(topicName: String, MinOrMax: Int): Map[TopicPartition, Long] = {
  val parser = new OptionParser

  val clientId = "GetOffsetShell"
  val brokerList = brokerListOpt
  ToolsUtils.validatePortOrDie(parser, brokerList)
  val metadataTargetBrokers = ClientUtils.parseBrokerList(brokerList)
  val topic = topicName
  val time = MinOrMax

  val topicsMetadata = ClientUtils.fetchTopicMetadata(Set(topic), metadataTargetBrokers, clientId, 1000).topicsMetadata
  if (topicsMetadata.size != 1 || !topicsMetadata.head.topic.equals(topic)) {
    System.err.println(("Error: no valid topic metadata for topic: %s, " + " probably the topic does not exist, run ").format(topic) +
      "kafka-list-topic.sh to verify")
    System.exit(1)
  }
  val partitions = topicsMetadata.head.partitionsMetadata.map(_.partitionId)
  val fromOffsets = collection.mutable.HashMap.empty[TopicPartition, Long]
  partitions.foreach { partitionId =>
    val partitionMetadataOpt = topicsMetadata.head.partitionsMetadata.find(_.partitionId == partitionId)
    partitionMetadataOpt match {
      case Some(metadata) =>
        metadata.leader match {
          case Some(leader) =>
            val consumer = new SimpleConsumer(leader.host, leader.port, 10000, 100000, clientId)
            val topicAndPartition = TopicAndPartition(topic, partitionId)
            val request = OffsetRequest(Map(topicAndPartition -> PartitionOffsetRequestInfo(time, 1)))
            val offsets = consumer.getOffsetsBefore(request).partitionErrorAndOffsets(topicAndPartition).offsets
            fromOffsets += (new TopicPartition(topic, partitionId.toInt) -> offsets.mkString(",").toLong)
          case None => System.err.println("Error: partition %d does not have a leader. Skip getting offsets".format(partitionId))
        }
      case None => System.err.println("Error: partition %d does not exist".format(partitionId))
    }
  }
  fromOffsets.toMap
}

三、使用問題

看上面兩篇博客