| 持久性级别 | 分区 | Pulsar | Kafka 发送和实时数据读取最大吞吐(MB\/s) | 1 级持久性 | 1 | 300 MB\/s | 160 MB\/s100 | 300 MB\/s | 420 MB\/s2000 | 300 MB\/s | 300 MB\/s2 级持久性 | 1 | 300 MB\/s | 180 MB\/s100 | 605 MB\/s | 605 MB\/s2000 | 605 MB\/s | 300 MB\/s发送和历史数据读取最大吞吐 (MB\/s) | 1 级持久性 | 100 | 1.7 GB\/s | 1 GB\/s2 级持久性 | 100 | 3.5 GB\/s | 1 GB\/s"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"表 1：在不同工作负载及不同持久性保证下，Pulsar 与 Kafka 的吞吐量差异"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":2,"normalizeStart":2},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在不同的测试实例（包括不同订阅数量、不同主题数量和不同持久性保证）中，Pulsar 的延迟显著低于 Kafka。"},{"type":"text","text":"Pulsar P99 延迟在 5 到 15 毫秒之间。Kafka P99 延迟可能长达数秒，并且会因主题数量、订阅数量和不同持久性保证而受到巨大影响。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" | 分区和订阅 | 本地持久性 | 复制持久性 | Pulsar | Kafka端到端 P99 延迟（毫秒） （发布+ Tailing Reads） | 100 个分区，1 个订阅 | 同步 | Ack-1 | 5.86 | 18.75Ack-2 | 11.64 | 64.62异步 | Ack-1 | 5.33 | 6.94Ack-2 | 5.55 | 10.43100 个分区，10 个订阅 | 同步 | Ack-1 | 7.12 | 145.10Ack-2 | 14.65 | 1599.79异步 | Ack-1 | 6.84 | 89.80Ack-2 | 6.94 | 1295.78"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"表 2：在不同订阅数量及不同持久性保证下，Pulsar 与 Kafka 端到端 P99 延迟差异"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" | 本地持久性 | 复制持久性 | 分区 | Pulsar | Kafka端到端 P99 延迟（毫秒） （发布+ Tailing Reads） | 同步 | Ack-1 | 100 | 5.86 | 18.755000 | 6.26 | 7923610000 | 6.67 | 187840Ack-2 | 100 | 11.64 | 64.625000 | 14.38 | 15796010000 | 15.78 | 197140异步 | Ack-1 | 100 | 5.33 | 6.945000 | 5.75 | 8664110000 | 6.64 | 184513Ack-2 | 100 | 5.55 | 10.435000 | 6.20 | 11602810000 | 7.50 | 200793"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"表 3：在不同主题数量及不同持久性保证下，Pulsar 与 Kafka 端到端 P99延迟差异"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":3,"normalizeStart":3},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Pulsar 的 I\/O 隔离显著优于 Kafka。在有消费者 catch up 读取历史数据时，Pulsar P99 发布延迟仍然在 5 毫秒左右。相比之下，Kafka 的延迟会因 catch up read 而受到严重影响。Kafka P99 发布延迟可能会从几毫秒增加到几秒。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" | 本地持久性 | 复制持久性 | Pulsar | Kafka发布 P99 延迟（毫秒） （混合负载） | 同步 | Ack-1 | 5.89 | 13.48Ack-2 | 15.39 | 2091.31异步 | Ack-1 | 10.44 | 9.51Ack-2 | 35.51 | 1014.95"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"表 4：在 catch up read 下，Pulsar 和 Kafka P99发布延迟差异"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们所有的基准测试均"},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/openmessaging-benchmark","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"开源"}]},{"type":"text","marks":[{"type":"underline"}],"text":"（Github网址："},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/openmessaging-benchmark","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/streamnative\/openmessaging-benchmark"}]},{"type":"text","text":"），感兴趣的读者可以自行生成结果，也可更深入研究该测试结果及仓库中提供的指标。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"尽管我们的基准测试比 Confluent 的基准测试更准确全面，但并未涵盖全部场景。归根结底，通过自己的硬件\/实际工作负载进行的测试，是任何一个基准测试都替代不了的。我们也鼓励各位读者评估其他变量和场景，并利用自己的设置和环境进行测试。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"深入探究 Confluent 基准测试"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Confluent 将 "},{"type":"link","attrs":{"href":"http:\/\/openmessaging.cloud\/docs\/benchmarks\/","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"OpenMessaging 基准测试（OMB）框架"}]},{"type":"text","text":"作为其基准测试的依据，并进行了一些修改。在本节中，我们将说明在 Confluent 基准测试中发现的问题，并阐述这些问题如何影响 Confluent 测试结果的准确性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Confluent 的设置问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Confluent 基准测试结论不正确是因为 Pulsar 参数设置不合理。我们会在 StreamNative 基准测试部分详细讲解这些问题。除了 Pulsar 调优问题，Confluent 针对 Pulsar 和 Kafka 设置了不同的持久性保证。持久性级别会影响性能，两个系统的持久性设置相同，对比才具有参考价值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Confluent 工程师对 Pulsar 采用默认持久性保证，该保证比 Kafka 的持久性级别高。增加持久性级别会严重影响延迟和吞吐量，所以 Confluent 测试对 Pulsar 提出了比 Kafka 更高的要求。Confluent 使用的 Pulsar 版本尚不支持将持久性降低到与 Kafka 相同的级别，但 Pulsar 即将发布的版本支持该级别，在本次测试中也使用了该级别。如果 Confluent 工程师在两个系统上使用的持久性设置相同，那么测试结果显示的对比应该是准确的。我们当然不会因 Confluent 工程师未使用尚未发布的功能而指责他们。然而，测试记录并不能提供必要的情景，而且将其视为同等持久性设置的结果。本文会提供额外的情境说明。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"OMB 框架问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Confluent 基准测试遵循 OMB 框架指南，该指南建议在多个事件流系统中使用同一实例类型。但在测试中，我们发现同一类型的不同实例存在大量偏差，尤其是发生磁盘 I\/O 故障的情况下。为了最大程度地减少这种差异，我们在每次运行 Pulsar 和 Kafka 时都使用了相同实例，我们发现这些实例在很大程度上改进了结果的准确性，磁盘 I\/O 性能的微小差异可能会对系统整体性能造成较大差异。我们提议更新 OMB 框架指南，并在未来考虑采用这个建议。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Confluent 研究方法的问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Confluent 基准测试仅测试了几种有限的场景。例如，实际工作负载包括写入、 tailing read 和 catch-up read。当某一消费者正在读取日志“尾部”附近的最新消息时，即发生 tailing-read，Confluent 只测试了这一种场景。相比之下，catch-up read 在消费者有大量历史消息时发生，必须消耗至 “catch-up” 位置到日志的尾部消息，这是实际系统中常见的关键任务。如果不考虑 catch-up read，则会严重影响写入和 tailing read 的延迟。由于 Confluent 基准测试只关注吞吐量和端到端延迟，所以未能就各种工作负载下的预期行为提供全面结果。为了进一步让结果更接近实际应用场景，我们认为对不同数量的订阅和分区进行基准测试至关重要。很少有企业只关心具有几个分区和消费者的少量主题，他们需要有能力容纳具有不同主题\/分区的大量不同消费者，以映射到业务用例中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们在下表中概述了 Confluent 研究方法的具体问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"测试参数排除项局限性 Write 和 tailing read | Catch-up read | 最大吞吐量和端到端延迟有助于阐明事件流系统的基本性能特征，但把研究范围限定在两个参数中，结果会有片面性。1 个订阅 | 订阅\/消费者分组数量不等 | 未显示订阅数量如何影响吞吐量和延迟。100 个分区 | 分区数量不等 | 未显示分区数量如何影响吞吐量和延迟。"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"表 5：Confluent 基准测试研究方法的问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Confluent 基准测试的诸多问题源于对 Pulsar 的了解有限。为帮助大家后续开展基准测试时避免这些问题，我们和大家分享一些 Pulsar 技术见解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了开展准确的基准测试，需了解 Pulsar 的持久性保证。我们将以此问题作为切入点进行探讨，先总体概述分布式系统的持久性，然后说明 Pulsar 和 Kafka 在持久性保证上的差异。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分布式系统持久性概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"持久性是指在面对诸如硬件或操作系统故障等外部问题时，对系统一致性和可用性的维持能力。诸如 RDBMS 单节点存储系统依靠 fsync 写入磁盘来确保最大的持久性。操作系统通常会缓存写入，在发生故障时，写入可能会丢失，但 fsync 将确保将这些数据写入物理存储中。在分布式系统中，持久性通常来自数据复制，即将数据的多个副本分布到可独立失效的不同节点。但不应将本地持久性（fsync 数据）与复制持久性混为一谈，两者目的不同。接下来我们会解释这些特性的重要性及主要区别。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"复制持久性和本地持久性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分布式系统通常同时具备复制持久性和本地持久性。各种类型的持久性由单独的机制控制。可以灵活组合使用这些机制，根据需要设置不同的持久性级别。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"复制持久性通过一种算法创建数据的多个副本实现，所以同一数据可存储在多个位置，可用性和可访问性均得以提高。副本的数量 N 决定了系统的容错能力，很多系统需要“仲裁”或 N\/2 + 1 个节点来确认写入。在任何单个副本仍然可用的情况下，一些系统可以继续服务现有数据。这种复制机制对于处理彻底丢失实例数据至关重要，新实例可从现有副本中重新复制数据，这对可用性和共识性也至关重要（本节不展开探讨该问题）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比之下，本地持久性决定了各个节点级别对确认的不同理解。本地持久性要求把数据 fsync 到持久存储，确保即使发生断电或硬件故障，也不会丢失任何数据。数据的 fsync 可确保机器在短时间内出现故障恢复后，节点拥有先前确认的全部数据。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"持久性模式：同步和异步"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不同类型的系统提供不同级别的持久性保证。通常，系统的"},{"type":"text","marks":[{"type":"italic"}],"text":"整体持久性受以下因素影响"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系统是否将数据 fsync 到本地磁盘"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系统是否将数据复制到多个位置"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系统何时确认复制到对等系统"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系统何时确认写入客户端"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在不同的系统中，这些选择差异很大，并非所有系统都支持用户控制这些值。缺少其中某些机制的系统（例如非分布式系统中的复制），持久性更低。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们可以定义两种持久性模式，两者均可控制系统何时确认写入供内部复制，以及何时 写入到客户端，即“同步”和“异步”。这两种模式操作如下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"同步持久性："},{"type":"text","text":"仅在数据成功 fsync 到本地磁盘（本地持久性）或复制到多个位置（复制持久性）后，系统才向对等系统\/客户端返回写入响应。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"异步持久性："},{"type":"text","text":"在数据成功 fsync 到本地磁盘（本地持久性）或复制到多个位置（复制持久性）前，系统会向对等系统\/客户端返回写入响应。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"持久性级别：测量持久性保证"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"持久性保证以多种形式存在，这取决于以下变量："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据是否存储在本地，是否在多个位置复制或符合这两种情况"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"何时确认写入（同步\/异步）"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"与持久性模式一样，为区分不同的分布式系统，我们为持久性定义了四个级别。表 6 列出了从最高持久性到最低持久性的各个级别。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"级别复制本地操作1 | 同步 | 同步 | 仅在数据已复制到多个（至少大部分）位置并且每个副本已成功 fsync 到本地磁盘之后，系统才会向客户端返回写入响应。2 | 同步 | 异步 | 仅在数据已复制到多个（至少大部分）位置后，系统才会向客户端返回写入响应，但不保证每个副本会成功 fsync 到本地磁盘。3 | 异步 | 同步 | 在某个副本已成功 fsync 到本地磁盘后，系统向客户端返回写入响应，但不保证将数据复制到其他位置。4 | 异步 | 异步 | 数据复制到多个位置后，系统会立即向客户端返回写入响应，但不对复制或本地持久性作出保证。"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"表 6：分布式系统的持久性级别"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多数分布式关系数据库管理系统（例如 NewSQL 数据库）均可保证最高级别的持久性，所以将这类系统归为 1 级。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"与数据库一样，Pulsar 属于 1 级系统，默认提供最高级别的持久性。此外，Pulsar 可针对每种应用分别自定义所需的持久性级别。相比之下，Kafka 大部分的生产环境部署都配置在 2 级或 4 级。据悉，通过设置 flush.messages=1 和 flush.ms=0，Kafka 也能达到 1 级标准。但这两项配置会严重影响吞吐量和延迟，我们会在基准测试中详细讨论这个问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面我们从 Pulsar 入手，详细探究各系统的持久性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Pulsar 的持久性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Pulsar 提供各级别的持久性保证，可将数据复制到多个位置，并将数据 fsync 到本地磁盘。Pulsar 拥有两种持久性模式（即上文所述的同步和异步）。用户可以根据使用场景自定义设置，单独使用某一模式，或组合使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Pulsar 利用筏等效、基于仲裁的复制协议来控制复制的持久性。通过调整 ack-quorum-size 和 write-quorum-size 参数可以调整复制持久性模式。表 7 列出了这些参数的设置，表 8 列出了 Pulsar 支持的持久性级别。（Pulsar 复制协议和共识算法不属于本文探讨范围，我们会在后续的博客中深入探讨该领域。）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"位置配置设定持久性模式复制 ackQuorumSize = 1 | 异步 | 同步 本地 （默认）journalWriteData = truejournalSyncData = true | 同步 journalWriteData = truejournalSyncData =false | 异步 journalWriteData = falsejournalSyncData =false | 异步 "}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"表 7：Pulsar 持久性配置设置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" | | | | | | | | | |
| | | | | | | |
| | | | | |
| | | | |
| | | | | | | | | | | | |
| | | | | | | | |
| | | | | | | | |
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"摘要："},{"type":"text","text":"如今，为开发新产品和服务，许多公司纷纷开始关注实时数据流应用程序。企业必须首先了解不同事件流系统的优势和差异，才能选出与其业务需求最匹配的技术。基准测试是各企业比较和衡量不同技术性能的一种方法。为了使该测试有参考价值，必须准确开展测试，并输出准确的信息。遗憾的是，总有诸多因素会影响基准测试的准确性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Confluent 最近开展了一次基准测试，对比 Kafka、Pulsar 和 RabbitMQ 的吞吐量和延迟差异。Confluent 博客显示，Kakfa 能够以“低延迟”实现“最佳吞吐量”，而 RabbitMQ 能够以“较低的吞吐量” 达到 “低延迟”。总体而言，基准测试结果显示 Kafka 在“速度”方面无疑更胜一筹。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka 技术成熟完善，但当今众多公司（从跨国公司到创新型初创公司）还是首先选择了 Pulsar。在近期举办的 Splunk 峰会 conf20 上，Splunk 公司首席产品官 Sendur Sellakumar 对外宣布，他们决定用 Pulsar 取代 Kafka："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“...我们已把 Apache Pulsar 作为基础流。我们把公司的前途压在了企业级多租户流的长期架构上。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"-- Splunk 首席产品官 Sendur Sellakumar "}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很多公司都在使用 Pulsar，Splunk 只是其中一例。这些公司之所以选择 Pulsar，是因为在现代弹性云环境（如 Kubernetes）中，Pulsar 能够以经济有效的方式横向扩展处理海量数据，不存在单点失效的问题。同时，Pulsar 具有诸多内置特性，诸如数据自动重平衡、多租户、跨地域复制和持久化分层存储等，不仅简化了运维，同时还让团队更容易专注于业务目标。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"开发者们最终选择 Pulsar 是因为 Pulsar 这些独特的功能和性能，让 Pulsar 成了流数据的基石。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"了解了这些情况后，还需仔细研究 Confluent 的基准测试设置和结论。我们发现有两个问题存在高度争议。其一，Confluent 对 Pulsar 的了解有限，这正是造成结论不准确的最大根源。如不了解 Pulsar，就不能用正确的衡量标准来测试 Pulsar 性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其二，Confluent 的性能测试基于一组狭窄的测试参数。这限制了结果的适用性，也无法为读者提供不同工作负载和实际应用场景相匹配的准确结果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了向社区提供更准确的测试结果，我们决定解决这些问题并重复测试。重要调整包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"我们调整了基准测试设置，包含了 Pulsar 和 Kafka 支持的各持久性级别，在同一持久性级别下对比两者的吞吐量和延迟。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我们修复了 OpenMessaging 基准测试（OMB）框架，消除因运用不同实例产生的变量，并纠正了 OMB Pulsar 驱动程序中的配置错误。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"最后，我们测量了其他性能因素和条件，例如分区的不同数量和包含 write、tailing-read 和 catch-up read 的混合工作负载，更全面地了解性能。 "}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"完成这些工作之后，我们重复了测试。测试结果显示，对于更接近真实工作负载的场景，Pulsar 的性能明显优于 Kafka，而对于 Confluent 在测试中所应用的基本场景，Pulsar 性能与 Kafka 性能相当。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下各部分将重点说明本次测试得出的最重要结论。在 StreamNative 基准测试结果章节，我们详细介绍了测试设置和测试报告。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"StreamNative 基准测试结果概要"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 在与 Kafka 的持久性保证相同的情况下，Pulsar 可达到 605 MB\/s 的发布和端到端吞吐量（与 Kafka 相同）以及 3.5 GB\/s 的 catch-up read 吞吐量（比 Kafka 高 3.5 倍）。Pulsar 的吞吐量不会因分区数量的增加和持久性级别的改变而受到影响，而 Kafka 的吞吐量会因分区数量或持久性级别的改变而受到严重影响。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"