kafka系列第5篇：一文读懂消费者背后的那点"猫腻"

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"经过前几篇文章的介绍，大致了解了生产者背后的运行原理。消息有生产就得有人去消费，今天我们就来介绍下消费端消费消息背后发生的那点事儿。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"文章概览"}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消费者与消费组的“父子关系”。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Repartition 触发时机。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消费者与 ZK 的关系。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消费端工作流程。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消费者的三种消费情况。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"消费者与消费组的“父子关系”"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8d/8da1a0069d0e377f69d4556d8a0cb345.png","alt":null,"title":"消费者消费组关系图","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka 消费端确保一个 Partition 在一个消费者组内只能被一个消费者消费。这句话改怎么理解呢？"}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在同一个消费者组内，一个 Partition 只能被一个消费者消费。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在同一个消费者组内，所有消费者组合起来必定可以消费一个 Topic 下的所有 Partition。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在同一个消费组内，一个消费者可以消费多个 Partition 的信息。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在不同消费者组内，同一个分区可以被多个消费者消费。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"每个消费者组一定会完整消费一个 Topic 下的所有 Partition。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"消费组存在的意义"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"了解了消费者与消费组的关系后，有朋友会比较疑惑消费者组有啥实际存在的意义呢？或者说消费组的作用是什么？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者对消费组的作用归结了如下两点。"}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在实际生产中，对于同一个 Topic，可能有 A、B、C 等 N 个消费方想要消费。比如一份用户点击日志，A 消费方想用来做一个用户近 N 天点击过哪些商品；B 消费方想用来做一个用户近 N 天点击过前 TopN 个相似的商品；C 消费方想用来做一个根据用户点击过的商品推荐相关周边的商品需求。对于多应用场景，就可以使用消费组来隔离不同的业务使用场景，从而达到一个 Topic 可以被多个消费组重复消费的目的。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消费组与 Partition 的消费进度绑定。当有新的消费者加入或者有消费者从消费组退出时，会触发消费组的 Repartition 操作（后面会详细介绍 Repartition）；在 Repartition 前，Partition1 被消费组的消费者 A 进行消费，Repartition 后，Partition1 消费组的消费者 B 进行消费，为了避免消息被重复消费，需要从消费组记录的 Partition 消费进度读取当前消费到的位置（即 OffSet 位置），然后在继续消费，从而达到消费者的平滑迁移，同时也提高了系统的可用性。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Repartition 触发时机"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用过 Kafka 消费者客户端的同学肯定知道，消费者组内偶尔会触发 Repartition 操作，所谓 Repartition 即 Partition 在某些情况下重新被分配给参与消费的消费者。基本可以分为如下几种情况。"}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消费组内某消费者宕机，触发 Repartition 操作，如下图所示。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c6/c67d21142809f374c9c3763c14d7fc5e.png","alt":null,"title":"消费者宕机情况","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消费组内新增消费者，触发 Repartition 操作，如下图所示。一般这种情况是为了提高消费端的消费能力，从而加快消费进度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f9/f99aa0b64317fadfb3e8466523289a34.png","alt":null,"title":"新增消费者情况","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Topic 下的 Partition 增多，触发 Repartition 操作，如下图所示。一般这种调整 Partition 个数的情况也是为了提高消费端消费速度的，因为当消费者个数大于等于 Partition 个数时，在增加消费者个数是没有用的（原因是：在一个消费组内，消费者:Partition = 1:N，当 N 小于 1 时，相当于消费者过剩了），所以一方面增加 Partition 个数同时增加消费者个数可以提高消费端的消费速度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c3/c3e265105db418c0a2a8e489da38f23c.png","alt":null,"title":"新增Partition个数情况","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"消费者与 ZK 的关系"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"众所周知，ZK 不仅保存了消费者消费 partition 的进度，同时也保存了消费组的成员列表、partition 的所有者。消费者想要消费 Partition，需要从 ZK 中获取该消费者对应的分区信息及当前分区对应的消费进度，即 OffSert 信息。那么 Partition 应该由那个消费者进行消费，决定因素有哪些呢？从之前的图中不难得出，两个重要因素分别是：消费组中存活的消费者列表和 Topic 对应的 Partition 列表。通过这两个因素结合 Partition 分配算法，即可得出消费者与 Partition 的对应关系，然后将信息存储到 ZK 中。Kafka 有高级 API 和低级 API，如果不需要操作 OffSet 偏移量的提交，可通过高级 API 直接使用，从而降低使用者的难度。对于一些比较特殊的使用场景，比如想要消费特定 Partition 的信息，Kafka 也提供了低级 API 可进行手动操作。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"消费端工作流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在介绍消费端工作流程前，先来熟悉一下用到的一些组件。"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"KakfaConsumer"}],"marks":[{"type":"strong"}]},{"type":"text","marks":[{"type":"strong"}],"text":"：消费端，用于启动消费者进程来消费消息。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"ConsumerConfig"}],"marks":[{"type":"strong"}]},{"type":"text","marks":[{"type":"strong"}],"text":"：消费端配置管理，用于给消费端配置相关参数，比如指定 Kafka 集群，设置自动提交和自动提交时间间隔等等参数，都由其来管理。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"ConsumerConnector"}],"marks":[{"type":"strong"}]},{"type":"text","marks":[{"type":"strong"}],"text":"：消费者连接器，通过消费者连接器可以获得 Kafka 消息流，然后通过消息流就能获得消息从而使得客户端开始消费消息。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上三者之间的关系可以概括为：消费端使用消费者配置管理创建出了消费者连接器，通过消费者连接器创建队列（这个队列的作用也是为了缓存数据），其中队列中的消息由专门的拉取线程从服务端拉取然后写入，最后由消费者客户端轮询队列中的消息进行消费。具体操作流程如下图所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/20/2095e6f53affde2c1257120e979240c1.png","alt":null,"title":"消费端工作流程","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们在从消费者与 ZK 的角度来看看其工作流程是什么样的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/20/2090028905b43e81c84e426c313969b0.png","alt":null,"title":"消费端与ZK之间的工作流程","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从上图可以看出，首先拉取线程每拉取一次消息，同步更新一次拉取状态，其作用是为了下一次拉取消息时能够拉取到最新产生的消息；拉取线程将拉取到的消息写入到队列中等待消费消费线程去真正读取处理。消费线程以轮询的方式持续读取队列中的消息，只要发现队列中有消息就开始消费，消费完消息后更新消费进度，此处需要注意的是，消费线程不是每次都和 ZK 同步消费进度，而是将消费进度暂时写入本地。这样做的目的是为了减少消费者与 ZK 的频繁同步消息，从而降低 ZK 的压力。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"消费者的三种消费情况"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消费者从服务端的 Partition 上拉取到消息，消费消息有三种情况，分别如下："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"至少一次。即一条消息至少被消费一次，消息不可能丢失，但是可能会被重复消费。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"至多一次。即一条消息最多可以被消费一次，消息不可能被重复消费，但是消息有可能丢失。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"正好一次。即一条消息正好被消费一次，消息不可能丢失也不可能被重复消费。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.至少一次"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消费者读取消息，先处理消息，在保存消费进度。消费者拉取到消息，先消费消息，然后在保存偏移量，当消费者消费消息后还没来得及保存偏移量，则会造成消息被重复消费。如下图所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/00/0008d90a22a4be1d5a9f274a0832f763.png","alt":null,"title":"先消费后保存消费进度","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.至多一次"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消费者读取消息，先保存消费进度，在处理消息。消费者拉取到消息，先保存了偏移量，当保存了偏移量后还没消费完消息，消费者挂了，则会造成未消费的消息丢失。如下图所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b9/b99c8f1f65fbbe7805cd28d6c4b18dbf.png","alt":null,"title":"先保存消费进度后消费消息","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.正好一次"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正好消费一次的办法可以通过将消费者的消费进度和消息处理结果保存在一起。只要能保证两个操作是一个原子操作，就能达到正好消费一次的目的。通常可以将两个操作保存在一起，比如 HDFS 中。正好消费一次流程如下图所示。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a0/a0bbb114e2ad551227f81c1f26d4bd5d.png","alt":null,"title":"正好消费一次","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文讲解了消费组与消费者之间的关系，及 Repartition 的触发时机，然后讲述了消费端的基本工作流程，最后提出了一条消息被重复消费的几种情况。下篇文章我们来讲讲"},{"type":"text","marks":[{"type":"strong"}],"text":"消息在服务端是怎么存储的"},{"type":"text","text":"，敬请期待。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"微信公众号搜索"},{"type":"text","marks":[{"type":"size","attrs":{"size":12}},{"type":"strong"}],"text":"【z小赵】"},{"type":"text","text":"，更多系列精彩文章等你解锁。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/86/862b51ac70a151f66fa21406f2affc39.gif","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}