蘑菇街千億級消息Kafka上雲實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Kafka憑藉其高吞吐、高可靠等特性在實時數據或流式數據架構中扮演着重要角色，受到了衆多企業用戶的青睞。但是隨着雲時代來臨，公有云廠商紛紛推出消息隊列服務，很多用戶也逐漸從自建消息集羣過渡到使用雲上消息隊列服務。本文將以蘑菇街Kafka服務遷移上云爲例，闡述騰訊雲消息隊列CKafka如何對用戶產生價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Kafka  簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Kafka官網用這樣一句話描述最新版本的Kafka：A distributed streaming platform。即分佈式流式計算平臺，並對其做了如下闡述："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies."}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"譯文：Kafka®用於構建實時數據管道和流式應用程序。它具有水平可伸縮性、容錯性、超快速性，可在數千家公司中投入生產。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"升級到2.0+的Kafka給自身加了一層定義，即流計算平臺。但是在企業級使用場景下，Kafka還是被經常當作數據管道來使用，履行消息隊列的基本職能。其典型的使用場景如下:  "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據管道和系統解耦。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異步處理和事件驅動。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流量削峯。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事務消息和分佈式事務的最終一致性。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/64\/641f3d48e242162f0acae12d2bbc8605.jpeg","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"自建Kafka集羣的痛點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Kafka的搭建方式簡單方便，且其性能高效穩定，很多企業用戶選擇自建Kafka集羣。但這樣看似完美的可行方案背後卻有一個隱型風險：當業務的消息數據量到達一定程度後，自建的消息隊列集羣就會引發各種各樣的問題，那麼如何解決問題呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們都知道Kafka入門簡單，進階卻有一定的門檻。解決問題的研發人員需要具備紮實的計算機功底（熟悉計算機網絡、IO等），並且對Kafka的底層原理、各種配置參數項等具有深刻理解，可以進行Kafka集羣參數調優，快速處理突發故障、恢復集羣抖動和動態進行集羣擴縮容等。正因如此，引發了一些問題：企業一方面需要投入更多的人力、物力成本，另一方面需要時刻監控集羣的健康狀況，及時排除問題以保障業務的穩定運行。所以自建Kafka集羣雖然簡單，但需要承擔日益加重的研發和運維成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"蘑菇街上雲背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"蘑菇街的業務場景和軟件架構決定了它對Kafka有着強大的依賴，作爲電商領域的佼佼者，其消息總量達到了日均千億條，生產峯值帶寬達每秒GB級別。其主要的業務場景爲分佈式大數據處理場景，如廣告、交易、安全、離線處理等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在意識到自建Kafka集羣的痛點後，爲了保證數據的安全性和集羣的穩定性，蘑菇街選擇使用雲上消息隊列服務CKafka。"},{"type":"text","marks":[{"type":"strong"}],"text":"CKafka不僅支持多可用區容災，還可以幫助客戶實現冷熱數據分離，解決頻繁讀取磁盤IO瓶頸，爲業務的穩定運行提供良好的保障。"},{"type":"text","text":"接下來我們來分析闡述CKafka是如何做到可用區容災和高性能的集羣服務器IO。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"集羣跨可用區容災方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/35\/350f6c2f7d99138da0e0cbb06b314c06.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Kafka消息系統中，客戶端感知服務端最核心的操作就是生產和消費。跨可用區容災的目標是：當一個可用區發生故障(如火災，斷電等)時，能夠做到客戶端無感知的進行生產和消費，保證業務的穩定運行。而滿足可用區容災需要在技術層面解決如下問題（以上面示意圖爲例）："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分區副本的跨可用區分佈，即保證每個分區的副本分佈在不同可用區。比如，當集羣跨上海二區和四區兩個可用區時，分區有四個副本，則需要保證每個可用區都分佈兩個副本；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka強依賴Apache Zookeeper，當Zookeeper不能正常提供服務時，Kafka集羣也會受到影響，故實現kafka的跨區容災，也要實現zookeeper的跨區容災。Apache Zookeeper和Kafka 一樣，具有跨區容災的能力。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Broker節點的IP對客戶端需要透明化。即客戶端不能感知Broker的地址。這樣當後端Borker故障，切換機器IP發生改變時，客戶端無感知，依然可以正常運行。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決上述問題需要下面4個技術方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1. 透明可漂移的Broker節點IP"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼Broker的節點IP和端口需要對用戶端透明呢？我們先來看如下一段代碼："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"Properties props = new Properties();\n props.put(\"bootstrap.servers\", \"192.168.10.10:9092,192.168.10.11:9092,192.168.10.12:9092\");\n props.put(\"acks\", \"all\");\n props.put(\"key.serializer\", \"org.apache.kafka.common.serialization.StringSerializer\");\n props.put(\"value.serializer\", \"org.apache.kafka.common.serialization.StringSerializer\");\n Producer producer = new KafkaProducer<>(props);\n for (int i = 0; i (\"my-topic\", Integer.toString(i),\n Integer.toString(i)));\n producer.close();"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是一段最簡單的Kafka Produce代碼。192.168.10.10:9092、192.168.10.11:9092、192.168.10.12:9092是三臺真實Kafka Broker 的IP和端口獲取Server端Metadata信息，開始進行生產消息操作。我們來設想一下如下的情況："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當其中一臺192.168.10.10機器故障無法恢復時，我們重新啓動了另外一臺Borker，比如192.168.10.13:9092提供服務。此時就需要通知所有客戶端，將Kafka地址從: \"192.168.10.10:9092,192.168.10.11:9092,192.168.10.12:9092\" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改爲 \"192.168.10.13:9092,192.168.10.11:9092,192.168.10.12:9092\"。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若IP配置硬編碼到客戶端代碼中，則需要修改代碼，然後打包併發布。"},{"type":"text","marks":[{"type":"strong"}],"text":"由於服務端調整而導致客戶端修改配置、重啓，這簡直是災難！"},{"type":"text","text":"那要怎麼解決這個問題呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決問題的思路就是: "},{"type":"text","marks":[{"type":"strong"}],"text":"Virtual IP Address"},{"type":"text","text":"。如下圖所示，我們會在每臺提供的Broker之前掛載一個四層的Virtual IP(VIP)和Virtual Port(VPORT)，用戶通過訪問VIP和VPORT來訪問實際的Broker服務。如10.0.0.1:9092對應的是真正的Broker服務192.169.10.10.9092。這樣就達到了實際Broker IP對用戶透明化的目的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3b\/3b16836b95df2c52a0ee84650cc66959.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那什麼是漂移呢？服務需要做到跨可用區容災。即我們提供的Virtual IP  Address能夠在可用區之間進行切換的，當該可用區故障，該VIP可以迅速切換漂移至另一個可用區，繼續提供服務。那麼該VIP應該是可以訪問所有可用區的。如下圖，當上海可用區2發生故障後，Virtual Ip Service迅速自動切換到上海可用區1可用的broker實例，保證客戶端的正常使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3c\/3cd22fbbb9f3c358b4fa575375a45464.jpeg","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 分區副本的跨區分佈"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原生的Kafka按照同一個可用區的副本不能分配在同一臺機器上的原則，進行副本隨機分配。副本分佈邏輯是無感知可用區。即當集羣裏面哪臺broker有空閒的空間，就將副本分佈在Broker上。則有可能將同一個partiton的分區分佈在同一個分區。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上面的跨可用區Virual IP切換示意圖所示，當創建一個3個Replication（副本）的Partition時，很有可能該Partition的Replication都落在了上海可用區2。如果此時上海可用區2發生故障，那麼該Partition就不能正常提供服務，直接影響業務。怎麼解決這個問題呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CKafka會在broker上添加可用區標記，當發現客戶創建的主題是跨可用區主題時，會將"},{"type":"text","marks":[{"type":"strong"}],"text":"同一個分區的副本分配在多個可用區"},{"type":"text","text":"，保證一個可用區故障時分區仍然有存活的副本。通過修改Kafka源碼的分區分配邏輯，添加了可用區標記邏輯，根據需求將不同的Replicatiton分配到不同的Broker上。而這些Broker則屬於不同的可用區。實現原理如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先來看一下Zookeeper上的節點\/broker\/topics\/test-topic的內容，內容如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"{\"version\":1,\"partitions\":{\"0\":[10840,10839],\"1\":[10838,10840],\"2\":[10839,10838]}}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這段內容意思是：test-topic這個主題有0、1、2三個分區，0分區分佈在broker[10840,10839]上，1分區分佈在broker[10838,10840]，依次類推。所以，只需要修改該內容的生成邏輯就可以控制Partiton的分佈，即可實現該邏輯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. Zookeeper的跨區部署"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"被Kafka強依賴的Zookeeper組件，它也需要跨區部署保證其可用性。首先來看一下Zookeeper的選舉策略：半數以上的節點都同意後才能當選leader，如果是偶數節點可能導致票數相同的情況，會使leader選取失敗，最終導致集羣失效。另外當Zookeeper集羣故障節點數超過半數時，Zookeeper集羣將無法正常工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由Zookeeper分佈式一致性算法的特點，可以得出一個結論：假如每個zone部署一個zk節點，zk要支持n區容災(同時掛掉n個區的zk節點)，需要部署2n+1個分區才能保證Zookeeper的分區可用。即在n=1的情況下，需要部署3個可用區，才能保證zookeeper集羣的單可用區可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4. Broker配置優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據設計方案，在不同的可用區部署Broker時，需要調整一些參數。這些參數保證了服務跨區容災的最大可用性。需要修改如下三個配置:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"unclean.leader.election.enable=true\nmin.insync.replicas=1\noffsets.topic.replication.factor=3"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這三個配置什麼意思呢? 依次來看一下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"unclean.leader.election.enable"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方描述:Indicates whether to enable replicas not in the ISR set to be selected as leader as a last resort, even though doing so may result in data loss。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解釋：該字段默認值爲False。默認情況下leader不能從非ISR的副本列表裏選擇；因爲在非ISR副本列表裏選擇leader，很有可能會導致部分數據丟失。既然這樣，那爲什麼還要打開這個字段呢？因爲在很異常情況下，比如ISR內的副本都不可用了，此時如果該字段設置爲False，服務會直接掛掉；如果該字段設爲True，即允許從非ISR列表中選擇leader，那麼服務儘管有可能丟失數據，卻依然可以繼續使用。所以這個參數必須參考業務特性來決定是否打開。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"min.insync.replicas"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方描述：When a producer sets acks to \"all\" (or \"-1\"), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解釋：該字段默認值爲1。上述英文翻譯爲：表示當在acks=-1時，最少有一個Replica進行確認回執，才確認數據寫入成功。這個參數在集羣搭建時，爲了保證數據的完整性，經常會被改爲2。這裏改爲1的原因是：在只有一個副本在工作 、其他都掛掉的極端情況下，保證客戶端能夠正常提供服務。如果設置爲2，當只有一個副本在工作的時候，就會出現生產端一直生產失敗的情況，會影響業務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"offsets.topic.replication.factor"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官網描述:The replication factor for the offsets topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解釋：該值默認爲1。表示kafka的內部topic consumer_offsets副本數。當該副本所在的broker宕機，consumer_offsets只有一份副本，該分區宕機。使用該分區存儲消費分組offset位置的消費者均會收到影響，offset無法提交，從而導致生產者可以發送消息但消費者不可用。所以需要設置該字段的值大於1。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"集羣IO壓力優化方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自建消息集羣的用戶常常會遇到一個問題：在流量峯值時，集羣IO壓力很大，用戶只能通過擴容來暫時解決問題。但這畢竟是權宜之計，爲了幫助用戶真正解決該問題，騰訊雲CKafka團隊對客戶服務器端的各項指標及業務場景進行了深入分析。我們發現集羣的IO壓力佔比最大的是磁盤讀壓力。但是爲什麼磁盤讀壓力大呢？我們首先來看一下Kafka底層的磁盤存儲設計原理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1. Kafka磁盤存儲設計原理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka的磁盤存儲設計可以用三個詞來概括：磁盤順序讀寫、Page Cache和零拷貝。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"磁盤順序讀寫"},{"type":"text","text":"：即Kafka數據的寫入和讀取是順序的。而根據局部性原理，在實際測試中，磁盤順序寫入和隨機寫入的性能比相差最大可達6000倍。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Page Cache"},{"type":"text","text":"：它是Kafka能夠實現順序讀寫的關鍵技術。另外，它也是操作系統主要實現的一種磁盤緩存，用來減少磁盤的 I\/O 操作。具體做法是把磁盤中的數據緩存到內存中，把對磁盤的訪問變爲對內存的訪問。Page Cache中的數據會按照一定的策略更新到磁盤。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"零拷貝"},{"type":"text","text":"：將數據直接從磁盤文件複製到網卡設備中，而不需要經由應用程序之手。這樣做大大提高了應用程序的性能，減少了內核和用戶模式之間的上下文切換。對 Linux 操作系統而言，零拷貝技術依賴於底層的 sendfile() 方法實現。對於 Java 語言，FileChannal.transferTo() 方法的底層實現也是 sendfile() 方法。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 爲什麼服務器讀壓力大？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/82\/8251c23c41c93df257ea4bf4e705daf1.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面的存儲原理圖來分析：理論上集羣的讀壓力不應該這麼大，因爲大部分的讀壓力應該命中Page Cache，不應該再從磁盤裏面讀取。然而實際情況中確實存在大量的磁盤讀取行爲。經過分析，客戶存在多個業務消費同一份消息的業務場景，根據消費的實時性可以將消息消費者行爲劃分兩類：實時消費者和離線消費者。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實時消費者：對數據實時性要求較高，需要採用實時消費消息的方式。在實時消費的場景下，Kafka會利用系統的page cache緩存，生產消息到broker，然後直接從內存轉發給實時消費者，磁盤壓力爲零。通常稱上述操作爲熱讀，常見的業務場景有廣告、推薦等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"離線消費者：又名定時週期性消費者，消費的消息通常是數分鐘前或是數小時前的消息。而這類消息通常存儲在磁盤中，消費時會觸發磁盤的IO操作。通常稱其爲冷讀，適合報表計算、批量計算等週期性執行的業務場景。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在消息量非常大的情況下，實時和離線消費者同時消費一個集羣，會導致兩個問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實時消費者受到離線消費者影響：由於離線消費者消費，導致落盤數據和實時數據會頻繁的換入換出內存，直接影響實時業務的實時性，增加實時業務的響應時延；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"離線數據會導致繁重的磁盤IO操作：當離線任務讀取的數據量非常大時，會觸發磁盤的高IO，磁盤的IO util 甚至達到100%，影響集羣的穩定性。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. 優化之道：冷熱數據分離方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對用戶集羣中存在的數據冷讀和熱讀並存問題，我們認爲將集羣的數據進行冷熱數據分離是當前較優的解決方案。而在不改變生產端行爲的情況下，怎麼對冷熱數據進行分離呢？騰訊雲CKafka推出了基於開源Kafka Connector的數據同步服務來解決上述問題。架構圖如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/93\/930946db77c8af974d906542bc8182cd.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"broker集羣被拆分爲實時集羣和離線集羣。兩個集羣分別負責同時引導離線業務消費離線集羣。CKafka 在兩個集羣中間添加了connector集羣。connector集羣將離線業務訂閱的消息（按照主題維度同步）從實時集羣同步到離線集羣中，connector集羣實時進行數據同步，和實時消費者保持一致。這樣操作不僅對磁盤IO沒有影響，也不會對其他的實時消費者造成影響。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"CKafka對業務的價值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CKafka提供高吞吐性能、高可擴展性的消息隊列服務。在性能、擴展性、業務安全保障、運維等方面具有超強優勢，讓用戶在享受低成本、超強功能的同時，免除繁瑣運維工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"頭圖：Unsplash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者：張曉宇, 許文強"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：https:\/\/mp.weixin.qq.com\/s\/89zOy63MjDfyJnLhY8CkUA"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：蘑菇街千億級消息Kafka上雲實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"來源：騰訊雲中間件 - 微信公衆號 [ID：gh_6ea1bc2dd5fd]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"轉載：著作權歸作者所有。商業轉載請聯繫作者獲得授權，非商業轉載請註明出處。"}]}]}