{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天的內容分享將主要包含以下四個方面："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"介紹京東到家的訂單履約業務背景"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在業務背景的基礎上說明訂單的底層存儲方案"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於存儲方案的數據異構設計與實踐"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"面向複雜度的架構設計方法論"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡單來看，面向複雜度的架構設計方法論與上面3個部分沒有直接關係，把這部分內容放到文章中來講，主要是因爲數據異構本質上也是解決了軟件的寫入複雜度問題。在這個基礎上，我們向上抽象一層，來討論一下面向複雜度的架構設計方法論。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、京東到家訂單履約業務背景"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6f\/6fc3617ff74bc59dd5667ee1c55b1632.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從用戶提交訂單到服務履約系統，我們大致經歷了支付、下發商家、商家確認、訂單打印、揀貨、下發物流、配送、妥投等環節。這是一個基本的新零售履約流程。這裏，我標藍了一些流程。比如：下發商家、訂單打印等環節。主要是因爲這些環節是我們要和商家交互的功能點，當我們把訂單下發給商家時，首當其衝的環節是商家要確認這個訂單，並且開始履約流程。但是，在我們的實際業務中，商家在大促期間往往會出現履約瓶頸，忙到看不到我們下發的訂單，甚至不忙的時候也會看不到我們的訂單已經下發到他們的系統中，商家需要一個提示功能。這也就是我們的提示音需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提示音需求需要不斷的查詢底層存儲ES，並提示給商家有訂單到達了，需要他們去履約，如果商家沒有看到，就不斷查詢，不斷提示。就是這樣的一個循環查詢量級，在大促期間，訂單量級增大，查詢量級增大。基本上每次大促都會把我們的ES查到CPU飆高，甚至出現不可用的情況。爲了保護履約系統，我們做的臨時方案是做一個功能開關，在大促期間對提示音功能降級。可是這樣的降級並不是我們想要的。因爲最終商家還是收不到提示。導致履約質量下降。於是我們就面臨一個問題“存儲組件無法支撐大促時提示音業務的查詢請求量。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、底層數據源的職責分工"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d4\/d4f009bf65bba287fade744db1584ba8.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要解決我們面臨的查詢量級問題，就必須首先分析一下底層的存儲方案。以上，是我對到家訂單履約系統底層存儲的一個整體概括。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.Redis"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Redis在履約系統中主要承載的一個職責是worker跑批任務的存儲和查詢。因我們在系統中大量運用了跑批任務來實現最終一致性的一個設計，而Redis的Zset結構正好滿足了這樣的需求，將時間作爲分值，不斷的提供近期任務的查詢是Redis充當的根本職能。這裏解釋一下Redis爲什麼沒有承載過多的查詢職能。Redis雖然性能更好一些，但是，在數據量和查詢複雜度上，沒有ES支持的好，關鍵點是我們的查詢條件複雜度是比較高的，所以，Redis沒有承載過多的查詢職能。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.MySQL"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL在履約系統存儲中的職能是持久化存儲訂單數據，這裏主要還是使用其強大的事務機制，以保障我們的數據是正確寫入的。這是其他的兩個組件所不支持的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從履約流程上來看，我們將數據做了冷熱分離，熱點數據是我們在履約中的訂單(也就是未完成的訂單)，而完成的訂單，由於其使用率不會太高，所以，我們稱之爲冷數據。這樣的一個拆分也就是上圖中對應的業務庫和歷史庫。業務庫是熱庫，而歷史庫則是冷庫。這樣的一個冷熱分離思想，使我們的單庫單表數據量級維持在千萬級別。從而避免了對應的分庫分表複雜度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從部署架構上看，我們對業務庫進行了大量的主從分割。其中biz slave是我們的業務庫從庫，它也會承載一些履約中的訂單查詢職能。接下來的big data slave集羣則是大數據抽數據用做統計分析。最後的delay slave設置延遲一定時間消費binlog則是爲了防止master被誤操作而兜底的。比如有人錯誤執行了刪除db的命令，這樣的一個延遲消費的機制就可以利用binlog進行兜底回滾。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.Elasticsearch"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES在數據存儲中承擔了幾乎所有的查詢職責，這主要取決於它支持複雜查詢，並有天然分佈式的特點。在數據量複雜度解決方案上，避免了MySQL分庫分表的複雜度。這裏我們一共有3個ES集羣。其中HOT ES和Full ES也是進行了冷熱分離，這樣對我們的查詢流量進行拆分。有助於保證履約系統的穩定性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而第三套集羣Remind Elastic Cluster則是爲了解決我們上述提示音的問題。在有提示音集羣之前，我們所有的提示音查詢流量都是打到熱集羣的。也正是這樣的一個訪問量需求，導致了我們的熱集羣時有發生CPU飆高，接口響應緩慢，卡頓業務線程。所以，我們對熱集羣進行了進一步的拆分，於是就正式提出了提示音單獨集羣的方案。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、寫入複雜度問題"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7b\/7b0c642dc6e498094c0aace7952c7bed.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當確定冗餘一套提示音集羣以後，我們面臨的問題就是上述這樣的一個寫入複雜度問題，從圖上來看，我們在拆分這套集羣之前，訂單中心每次操作一次訂單寫入。面臨的是3個數據源的寫入工作，這對研發人員是非常不友好的，維護難度過大。於是，我們就開始考慮用異構中間件的方式來去寫入這套ES數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異構中間件的優勢是屏蔽了數據同步的複雜度，但是隨之而來的是數據寫入鏈路可靠性、及時性等問題。而且，數據傳輸本身一般都具有高可用的需求，之前高可用在業務應用上，因爲業務應用的集羣方式本身是計算高可用的。但異構中間件則要在這高可用、可靠性、及時性三個維度上滿足我們的要求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、數據異構產品選型"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d6\/d6835cbefd2f23eddc10c166b1905ccf.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上述分析之後，我們也陸續調研了一些異構產品，在數據類型支撐上沒有太大差別，常用的存儲組件，這些異構中間件都是支持的。所以，我們更在意以上3個指標。社區活躍度代表了後續的維護性以及開源產品快速的問題響應，可用性方面的需求是非常強烈的，最終採用Canal的根本原因還是在學習成本和熟練度上。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、Canal簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏簡單說一下我對於Canal的理解，以便於後續有意向應用Canal的同學有一個簡單的瞭解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3b\/3bb879cab90c27daf4506cb28cabddae.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Step1 Load&Store："},{"type":"text","text":" Connection從Zookeeper 獲取到當前消費的binlog filename和position信息。隨後將該信息附帶到dump協議裏，mysql master開始推送binlog數據。Binlog經過Parser解析投遞到Sink，Sink則承載了過濾消息的作用，過濾掉沒有訂閱的binlog事件，最終把消息存儲到Store中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Step2：Send&Ack："},{"type":"text","text":"用任務worker的方式，不斷掃描store，最終將store中的數據發送到目的地，目的地可以是具體的存儲，也可以是mq產品。圖中，我用了kafka也主要是因爲我們的實踐方案。投遞消息完成之後將消息ACK給Store組件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Step3：Update MetaInfo："},{"type":"text","text":"這個時候數據雖然發送了。但是，我們的元信息binlog的filename和position仍然沒有更新，在這個操作上，Canal仍然採取了異步的方式去同步該信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Canal這種異步通信的設計要求你的系統必須具備可回溯、重試、冪等、延遲特點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上，是我對整個Canal的一個理解，圖中的兩個HA，後續將會和大家說到。這裏，我講Canal的工作的角色、運作規則都是從一個4R視角來說明的，這是爲了後續來講複雜度方法論的時候大家也好理解一些。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"六、提示音功能基於Canal數據異構實踐"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/75\/75e6c8281f099783bc4feaa0d5f076f0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提示音異構生產部署方式如上圖。我們部署了兩臺Deployer用於數據傳輸的高可用。同時把消息投遞到了kafka，利用adapter的集羣部署進行批量消費，插入到提示音集羣的ES中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在順序性保障上採用了訂單id hash的策略，保證在partition上是有序的。這樣也就保證了在業務操作上是整體有序的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在鏈路上採用kafka來傳輸，主要還是應對大促期間binlog數據量級的特點，保證插入到ES之前有緩衝buffer的一個作用。這也是直連方式的弱點，直連方式在大數據量短時間寫入時，對目的地存儲組件有可能會造成瞬間的大量插入，從而損耗目的地存儲組件的資源，可能影響到業務使用。但是，長鏈路也有數據延遲的缺點，如果對數據時效要求比較高的業務。還是建議用直連方式來搭建對應的異構方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在META Manager上使用Zookeeper來存儲，與Deployer的HA形成有效配合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"問題一：（網絡環境問題）kafka不可用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0c\/0c070192442cb0f110c25a77de6fdad1.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在實踐中，我們遇到第一個比較有代表性的問題是kafka集羣不可用，直接導致ES數據斷層，從而影響到商家的履約體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，kafka集羣所在的網絡環境和機器主機發生問題，deployer的store數據存滿，直接導致delay了8個小時。提示音沒有提示，也會有電腦端的管理系統同步訂單，但是需要人工刷新，所以，過了很久我們才發現這樣一個問題。緊急把訪問切到之前的ES熱集羣，之後，我們重新把kafka服務部署到可用狀態，數據雖然慢慢追上了，但是原來在kafka中沒有被adapter消費的一部分數據卻丟掉了，這主要還是因爲設置的kafka落盤頻率問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/36\/36654543b75e902dd31c2534ffdc7606.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"丟數據在數據異構的需求中是不可容忍的事情，索性這次事故基本上鎖定了丟數據的原因，所以，我們將Zookeeper中的jouralName和position設置到對應的事故之前的位置，將數據重新跑到ES中，至此問題解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總結一"}]}]}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/43\/43326dbbec2fb5cb8514f6bb5a39bd53.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總結二"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a8\/a8e182280d2ff869322ec114c929fd56.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，總結以上兩點，數據異構的實踐在問題監控、報警、及時降級方面是非常重要的。希望這樣的總結經驗能夠讓大家少走彎路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"問題二：Deployer故障，自動HA"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/42\/423222dc53dbf76c815bb419feb6af58.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Deployer機器發生故障，系統自動HA到備機，任務得以繼續消費。總起來說，問題二並沒有給我們的業務帶來任何的損耗，但是，還是比較經典的一個案例。這主要反應出來，對於數據異構這樣一個需求。它的鏈路上所有環節，基本上都是有高可用的要求的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d7\/d7922acc62d9de90cdead79e3858b437.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Canal一共提供兩種HA，其中Deployer的HA是靠Zookeeper的臨時節點和重試機制實現的，而Mysql的HA則是靠一個單獨的線程不斷的Detect來實現的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是MySQL的HA，只能用GTID的模式，這是因爲Mysql master和slave的binlogfile name、position是不一樣的。如果用master的binlogfilename和position去slave發送dump協議，這會出現無法匹配的問題。但是GTID是全局有序的，這也就保證了Mysql的HA只在GTID模式下才可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ce\/ce6697a4ef643a0cea2470096a7843f0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"談到高可用，提出上述總結。這裏我與大家互動了一個問題：“單機器部署兩臺Canal實例是否算是高可用？”答案是：“不算高可用，原因是單機部署了兩臺Deployer，但是機器如果故障，兩臺Deployer均不可用。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"問題比較有代表性，也有一些同學掉進了坑裏。這裏我與大家一起回顧一下高可用的範圍：多機器、多機房、多地區、多國家。範圍越大，高可用自然越是穩定。但是帶來的成本和數據傳輸要求也越高，一般都是根據業務量級和重要程序進行取捨的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總結三"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/db\/dbb72e5c3eb42b128144058e4438f762.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是我在數據異構中的一些經驗教訓。下面我們將問題向上抽象，聊一聊面向複雜度的架構設計方法論。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"七、面向複雜度的架構設計方法論"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.4R模型"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/03\/03f087fc228437f0e445ca1c0070d1ef.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家是否發現，我在和大家聊Canal或是到家的數據異構方案時，更多的都是以角色、關係、規則這種描述方法。相信大家也不是第一次碰到這種描述方式，在很多的架構中，都是這樣的一種描述方法。就像上圖中，說到的Parser、Sink、Store。這些角色的職責是什麼？他們是如何配合完成Canal這樣的一個產品功能的呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4R模型本質上就是一個視角，它是Rank、Role、Relation、Rule這4個單詞的首字母構成，它強調了描述方法、也強調了我們要用這樣的視角來看待我們的系統。這樣整體來看，系統會更加清晰和簡潔。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.區分複雜度"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ec\/ecab787a914f8ba3164c344d9fe91b10.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖，將複雜度問題分爲技術方向和業務方向兩個部分，其中灰色的部分，一般都有一些開源軟件來幫我們解決，比如Dubbo、Spring、Canal等。而紅色的部門正是我們日常工作中所不能避免的複雜度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些複雜度問題，如果平時不加以重視，忽視掉的複雜度問題最終則會演變成爲不可維護的技術債務，最終打掉系統的可維護性，只能重新推倒重來了。很多重構行爲都是因爲複雜度的忽視累積而成的後果。所以，學會如何區分複雜度就是比較重要的點了。比如這次Canal的數據異構，同時面臨了數量級複雜度和寫入複雜度兩個。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.複雜度的架構設計環"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0c\/0c2c88e2ea68eb5a8823958868fdf975.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同樣，面向複雜度的架構設計方法論，最終會歸結到業務實現上。下面描述一下具體的步驟含義："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1-需求："},{"type":"text","text":"產品同學提出需求描述，或一句話需求、或完善的PRD文檔"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2-判斷："},{"type":"text","text":"對需求進行判斷，需要什麼樣的數據量，什麼樣的峯值，是否要高可用等等，如果不能理解清楚，則找對應的需求人員不斷澄清，直到清晰爲止"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3-複雜度識別："},{"type":"text","text":"將需求精確化以後，對需求的複雜度問題進行識別，比如業務複雜度問題、數據量複雜度問題"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4-拆解到備選架構："},{"type":"text","text":"針對識別出來的複雜度設計出多個具體的架構方案。比如採用ES存儲數據屏蔽分庫分表的數據量複雜度、採用數據異構的方式寫入數據，從而屏蔽數據寫入的複雜度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"5-取捨："},{"type":"text","text":"對備選架構進行取捨，任何的架構方案都有好的一面和壞的一面。在不同的時間都有不同的選擇，這裏建議大家從簡單、合適、演進三個架構原則來進行方案取捨,選擇最適合自己的那一套方案。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"6-架構方案4R："},{"type":"text","text":"用4R視角來設計系統分層、角色、關係、規則。以這種視角設計出來一套抽象模型"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"7-實現需求："},{"type":"text","text":"將4R架構模型實踐即可。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7b\/7b1c28ba813f33652dc4b9bafb7877d4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同樣，將本需求的一個架構設計環案例呈現給大家。(由於部分設計有保密性，4R此處用Canal4R代替)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上，就是我和大家分享的全部內容啦，謝謝大家!"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：訂單表中，如果有一些商品id，那麼同步到ES中也是id嗎，不會關聯出name打成寬表存到ES嗎?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A1："},{"type":"text","text":"具體的映射字段需要在Adapter中配置映射即可，存入到ES中的情況也與配置的映射是直接且唯一關係。是否寬表要在實際應用中把控字段的個數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2：Canal部署deploy主從和canal-adapter有沒有遇到官方的bug？有，改動了哪些？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A2："},{"type":"text","text":"遇到過Column not match的異常.具體看Canal的TSDB來解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3：這套複雜度方法論如何落地到實際應用？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A3："},{"type":"text","text":"需要對系統進行4R視角拆分、識別複雜度類型並按照架構設計環的方式來評定需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4：平時的Canal有消息延遲嗎？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A4："},{"type":"text","text":"有一定延遲的，binlog的數量、網絡等因素，都會造成一定的延遲，所以，建議異構還是要建立在業務數據可延遲的基礎上的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5：我主要用canal-adapter讀取Kafka中的binlog日誌然後寫到數據庫中，Kafka中有多個表的日誌，我rdb目錄下的yml文件只配置了一個表的爲什麼其他的表也會同步？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A5:"},{"type":"text","text":"Yml的作用是配置映射關係，具體的過濾職能在Deployer的Sink配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q6：異構數據是直接同步原表嗎，還是做了關聯？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A6: "},{"type":"text","text":"做了關聯，直接在Adapter中配置對應映射關係即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q7：請問爲什麼不直接增加熱集羣的節點和分片，而是重新建一套ES集羣呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A7: "},{"type":"text","text":"這裏主要還是一個數據拆分的思想，如果通過提高配置來解決訪問量問題，那麼，隨着業務量級增加，流量混在一起，對應的ES集羣流程會呈現不可評估的情況。本質上還是一個數據存儲職責的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q8：如何保證Zookeeper的高可用？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A8: "},{"type":"text","text":"Zookeeper本身就是高可用的，如果想在機房或異地方面做高可用，建議做主備同步、多集羣部署等手段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q9：新集羣的查詢請求峯值是多少?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A9: "},{"type":"text","text":"大約2000-4000 QPS。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q10：怎麼把握冗餘尺度呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A10: "},{"type":"text","text":"冗餘的維度在機器、機房、地區、國家是不斷增加的。維度越大，對應的高可用方案越可靠，但是，對應的費用以及實現複雜度也會變高。因爲這種冗餘方案肯定會有數據copy。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"嘉賓介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"張磊，"},{"type":"text","text":"京東到家高級研發工程師。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"8年+軟件研發經驗，曾先後就職於鏈家地產、互動吧、寺庫網等公司，任研發人員或團隊leader，在解決業務系統設計落地方面擁有豐富經驗；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現就職於達達-京東到家，主要負責訂單履約、金額拆分、計費相關業務域的研發設計工作。業餘時間主攻軟件複雜度優化。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/juMj7eUKbFKX0xbyQ8781g","title":"xxx","type":null},"content":[{"type":"text","text":"大促突圍：京東到家基於Canal的數據異構設計"}]}]}]}
大促突圍：京東到家基於Canal的數據異構設計
{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天的內容分享將主要包含以下四個方面："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"介紹京東到家的訂單履約業務背景"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在業務背景的基礎上說明訂單的底層存儲方案"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於存儲方案的數據異構設計與實踐"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"面向複雜度的架構設計方法論"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡單來看，面向複雜度的架構設計方法論與上面3個部分沒有直接關係，把這部分內容放到文章中來講，主要是因爲數據異構本質上也是解決了軟件的寫入複雜度問題。在這個基礎上，我們向上抽象一層，來討論一下面向複雜度的架構設計方法論。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、京東到家訂單履約業務背景"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6f\/6fc3617ff74bc59dd5667ee1c55b1632.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從用戶提交訂單到服務履約系統，我們大致經歷了支付、下發商家、商家確認、訂單打印、揀貨、下發物流、配送、妥投等環節。這是一個基本的新零售履約流程。這裏，我標藍了一些流程。比如：下發商家、訂單打印等環節。主要是因爲這些環節是我們要和商家交互的功能點，當我們把訂單下發給商家時，首當其衝的環節是商家要確認這個訂單，並且開始履約流程。但是，在我們的實際業務中，商家在大促期間往往會出現履約瓶頸，忙到看不到我們下發的訂單，甚至不忙的時候也會看不到我們的訂單已經下發到他們的系統中，商家需要一個提示功能。這也就是我們的提示音需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提示音需求需要不斷的查詢底層存儲ES，並提示給商家有訂單到達了，需要他們去履約，如果商家沒有看到，就不斷查詢，不斷提示。就是這樣的一個循環查詢量級，在大促期間，訂單量級增大，查詢量級增大。基本上每次大促都會把我們的ES查到CPU飆高，甚至出現不可用的情況。爲了保護履約系統，我們做的臨時方案是做一個功能開關，在大促期間對提示音功能降級。可是這樣的降級並不是我們想要的。因爲最終商家還是收不到提示。導致履約質量下降。於是我們就面臨一個問題“存儲組件無法支撐大促時提示音業務的查詢請求量。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、底層數據源的職責分工"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d4\/d4f009bf65bba287fade744db1584ba8.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要解決我們面臨的查詢量級問題，就必須首先分析一下底層的存儲方案。以上，是我對到家訂單履約系統底層存儲的一個整體概括。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.Redis"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Redis在履約系統中主要承載的一個職責是worker跑批任務的存儲和查詢。因我們在系統中大量運用了跑批任務來實現最終一致性的一個設計，而Redis的Zset結構正好滿足了這樣的需求，將時間作爲分值，不斷的提供近期任務的查詢是Redis充當的根本職能。這裏解釋一下Redis爲什麼沒有承載過多的查詢職能。Redis雖然性能更好一些，但是，在數據量和查詢複雜度上，沒有ES支持的好，關鍵點是我們的查詢條件複雜度是比較高的，所以，Redis沒有承載過多的查詢職能。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.MySQL"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL在履約系統存儲中的職能是持久化存儲訂單數據，這裏主要還是使用其強大的事務機制，以保障我們的數據是正確寫入的。這是其他的兩個組件所不支持的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從履約流程上來看，我們將數據做了冷熱分離，熱點數據是我們在履約中的訂單(也就是未完成的訂單)，而完成的訂單，由於其使用率不會太高，所以，我們稱之爲冷數據。這樣的一個拆分也就是上圖中對應的業務庫和歷史庫。業務庫是熱庫，而歷史庫則是冷庫。這樣的一個冷熱分離思想，使我們的單庫單表數據量級維持在千萬級別。從而避免了對應的分庫分表複雜度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從部署架構上看，我們對業務庫進行了大量的主從分割。其中biz slave是我們的業務庫從庫，它也會承載一些履約中的訂單查詢職能。接下來的big data slave集羣則是大數據抽數據用做統計分析。最後的delay slave設置延遲一定時間消費binlog則是爲了防止master被誤操作而兜底的。比如有人錯誤執行了刪除db的命令，這樣的一個延遲消費的機制就可以利用binlog進行兜底回滾。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.Elasticsearch"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES在數據存儲中承擔了幾乎所有的查詢職責，這主要取決於它支持複雜查詢，並有天然分佈式的特點。在數據量複雜度解決方案上，避免了MySQL分庫分表的複雜度。這裏我們一共有3個ES集羣。其中HOT ES和Full ES也是進行了冷熱分離，這樣對我們的查詢流量進行拆分。有助於保證履約系統的穩定性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而第三套集羣Remind Elastic Cluster則是爲了解決我們上述提示音的問題。在有提示音集羣之前，我們所有的提示音查詢流量都是打到熱集羣的。也正是這樣的一個訪問量需求，導致了我們的熱集羣時有發生CPU飆高，接口響應緩慢，卡頓業務線程。所以，我們對熱集羣進行了進一步的拆分，於是就正式提出了提示音單獨集羣的方案。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、寫入複雜度問題"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7b\/7b0c642dc6e498094c0aace7952c7bed.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當確定冗餘一套提示音集羣以後，我們面臨的問題就是上述這樣的一個寫入複雜度問題，從圖上來看，我們在拆分這套集羣之前，訂單中心每次操作一次訂單寫入。面臨的是3個數據源的寫入工作，這對研發人員是非常不友好的，維護難度過大。於是，我們就開始考慮用異構中間件的方式來去寫入這套ES數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異構中間件的優勢是屏蔽了數據同步的複雜度，但是隨之而來的是數據寫入鏈路可靠性、及時性等問題。而且，數據傳輸本身一般都具有高可用的需求，之前高可用在業務應用上，因爲業務應用的集羣方式本身是計算高可用的。但異構中間件則要在這高可用、可靠性、及時性三個維度上滿足我們的要求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、數據異構產品選型"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d6\/d6835cbefd2f23eddc10c166b1905ccf.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上述分析之後，我們也陸續調研了一些異構產品，在數據類型支撐上沒有太大差別，常用的存儲組件，這些異構中間件都是支持的。所以，我們更在意以上3個指標。社區活躍度代表了後續的維護性以及開源產品快速的問題響應，可用性方面的需求是非常強烈的，最終採用Canal的根本原因還是在學習成本和熟練度上。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、Canal簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏簡單說一下我對於Canal的理解，以便於後續有意向應用Canal的同學有一個簡單的瞭解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3b\/3bb879cab90c27daf4506cb28cabddae.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Step1 Load&Store："},{"type":"text","text":" Connection從Zookeeper 獲取到當前消費的binlog filename和position信息。隨後將該信息附帶到dump協議裏，mysql master開始推送binlog數據。Binlog經過Parser解析投遞到Sink，Sink則承載了過濾消息的作用，過濾掉沒有訂閱的binlog事件，最終把消息存儲到Store中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Step2：Send&Ack："},{"type":"text","text":"用任務worker的方式，不斷掃描store，最終將store中的數據發送到目的地，目的地可以是具體的存儲，也可以是mq產品。圖中，我用了kafka也主要是因爲我們的實踐方案。投遞消息完成之後將消息ACK給Store組件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Step3：Update MetaInfo："},{"type":"text","text":"這個時候數據雖然發送了。但是，我們的元信息binlog的filename和position仍然沒有更新，在這個操作上，Canal仍然採取了異步的方式去同步該信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Canal這種異步通信的設計要求你的系統必須具備可回溯、重試、冪等、延遲特點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上，是我對整個Canal的一個理解，圖中的兩個HA，後續將會和大家說到。這裏，我講Canal的工作的角色、運作規則都是從一個4R視角來說明的，這是爲了後續來講複雜度方法論的時候大家也好理解一些。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"六、提示音功能基於Canal數據異構實踐"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/75\/75e6c8281f099783bc4feaa0d5f076f0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提示音異構生產部署方式如上圖。我們部署了兩臺Deployer用於數據傳輸的高可用。同時把消息投遞到了kafka，利用adapter的集羣部署進行批量消費，插入到提示音集羣的ES中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在順序性保障上採用了訂單id hash的策略，保證在partition上是有序的。這樣也就保證了在業務操作上是整體有序的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在鏈路上採用kafka來傳輸，主要還是應對大促期間binlog數據量級的特點，保證插入到ES之前有緩衝buffer的一個作用。這也是直連方式的弱點，直連方式在大數據量短時間寫入時，對目的地存儲組件有可能會造成瞬間的大量插入，從而損耗目的地存儲組件的資源，可能影響到業務使用。但是，長鏈路也有數據延遲的缺點，如果對數據時效要求比較高的業務。還是建議用直連方式來搭建對應的異構方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在META Manager上使用Zookeeper來存儲，與Deployer的HA形成有效配合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"問題一：（網絡環境問題）kafka不可用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0c\/0c070192442cb0f110c25a77de6fdad1.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在實踐中，我們遇到第一個比較有代表性的問題是kafka集羣不可用，直接導致ES數據斷層，從而影響到商家的履約體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，kafka集羣所在的網絡環境和機器主機發生問題，deployer的store數據存滿，直接導致delay了8個小時。提示音沒有提示，也會有電腦端的管理系統同步訂單，但是需要人工刷新，所以，過了很久我們才發現這樣一個問題。緊急把訪問切到之前的ES熱集羣，之後，我們重新把kafka服務部署到可用狀態，數據雖然慢慢追上了，但是原來在kafka中沒有被adapter消費的一部分數據卻丟掉了，這主要還是因爲設置的kafka落盤頻率問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/36\/36654543b75e902dd31c2534ffdc7606.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"丟數據在數據異構的需求中是不可容忍的事情，索性這次事故基本上鎖定了丟數據的原因，所以，我們將Zookeeper中的jouralName和position設置到對應的事故之前的位置，將數據重新跑到ES中，至此問題解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總結一"}]}]}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/43\/43326dbbec2fb5cb8514f6bb5a39bd53.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總結二"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a8\/a8e182280d2ff869322ec114c929fd56.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，總結以上兩點，數據異構的實踐在問題監控、報警、及時降級方面是非常重要的。希望這樣的總結經驗能夠讓大家少走彎路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"問題二：Deployer故障，自動HA"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/42\/423222dc53dbf76c815bb419feb6af58.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Deployer機器發生故障，系統自動HA到備機，任務得以繼續消費。總起來說，問題二並沒有給我們的業務帶來任何的損耗，但是，還是比較經典的一個案例。這主要反應出來，對於數據異構這樣一個需求。它的鏈路上所有環節，基本上都是有高可用的要求的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d7\/d7922acc62d9de90cdead79e3858b437.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Canal一共提供兩種HA，其中Deployer的HA是靠Zookeeper的臨時節點和重試機制實現的，而Mysql的HA則是靠一個單獨的線程不斷的Detect來實現的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是MySQL的HA，只能用GTID的模式，這是因爲Mysql master和slave的binlogfile name、position是不一樣的。如果用master的binlogfilename和position去slave發送dump協議，這會出現無法匹配的問題。但是GTID是全局有序的，這也就保證了Mysql的HA只在GTID模式下才可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ce\/ce6697a4ef643a0cea2470096a7843f0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"談到高可用，提出上述總結。這裏我與大家互動了一個問題：“單機器部署兩臺Canal實例是否算是高可用？”答案是：“不算高可用，原因是單機部署了兩臺Deployer，但是機器如果故障，兩臺Deployer均不可用。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"問題比較有代表性，也有一些同學掉進了坑裏。這裏我與大家一起回顧一下高可用的範圍：多機器、多機房、多地區、多國家。範圍越大，高可用自然越是穩定。但是帶來的成本和數據傳輸要求也越高，一般都是根據業務量級和重要程序進行取捨的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總結三"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/db\/dbb72e5c3eb42b128144058e4438f762.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是我在數據異構中的一些經驗教訓。下面我們將問題向上抽象，聊一聊面向複雜度的架構設計方法論。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"七、面向複雜度的架構設計方法論"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.4R模型"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/03\/03f087fc228437f0e445ca1c0070d1ef.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家是否發現，我在和大家聊Canal或是到家的數據異構方案時，更多的都是以角色、關係、規則這種描述方法。相信大家也不是第一次碰到這種描述方式，在很多的架構中，都是這樣的一種描述方法。就像上圖中，說到的Parser、Sink、Store。這些角色的職責是什麼？他們是如何配合完成Canal這樣的一個產品功能的呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4R模型本質上就是一個視角，它是Rank、Role、Relation、Rule這4個單詞的首字母構成，它強調了描述方法、也強調了我們要用這樣的視角來看待我們的系統。這樣整體來看，系統會更加清晰和簡潔。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.區分複雜度"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ec\/ecab787a914f8ba3164c344d9fe91b10.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖，將複雜度問題分爲技術方向和業務方向兩個部分，其中灰色的部分，一般都有一些開源軟件來幫我們解決，比如Dubbo、Spring、Canal等。而紅色的部門正是我們日常工作中所不能避免的複雜度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些複雜度問題，如果平時不加以重視，忽視掉的複雜度問題最終則會演變成爲不可維護的技術債務，最終打掉系統的可維護性，只能重新推倒重來了。很多重構行爲都是因爲複雜度的忽視累積而成的後果。所以，學會如何區分複雜度就是比較重要的點了。比如這次Canal的數據異構，同時面臨了數量級複雜度和寫入複雜度兩個。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.複雜度的架構設計環"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0c\/0c2c88e2ea68eb5a8823958868fdf975.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同樣，面向複雜度的架構設計方法論，最終會歸結到業務實現上。下面描述一下具體的步驟含義："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1-需求："},{"type":"text","text":"產品同學提出需求描述，或一句話需求、或完善的PRD文檔"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2-判斷："},{"type":"text","text":"對需求進行判斷，需要什麼樣的數據量，什麼樣的峯值，是否要高可用等等，如果不能理解清楚，則找對應的需求人員不斷澄清，直到清晰爲止"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3-複雜度識別："},{"type":"text","text":"將需求精確化以後，對需求的複雜度問題進行識別，比如業務複雜度問題、數據量複雜度問題"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4-拆解到備選架構："},{"type":"text","text":"針對識別出來的複雜度設計出多個具體的架構方案。比如採用ES存儲數據屏蔽分庫分表的數據量複雜度、採用數據異構的方式寫入數據，從而屏蔽數據寫入的複雜度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"5-取捨："},{"type":"text","text":"對備選架構進行取捨，任何的架構方案都有好的一面和壞的一面。在不同的時間都有不同的選擇，這裏建議大家從簡單、合適、演進三個架構原則來進行方案取捨,選擇最適合自己的那一套方案。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"6-架構方案4R："},{"type":"text","text":"用4R視角來設計系統分層、角色、關係、規則。以這種視角設計出來一套抽象模型"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"7-實現需求："},{"type":"text","text":"將4R架構模型實踐即可。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7b\/7b1c28ba813f33652dc4b9bafb7877d4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同樣，將本需求的一個架構設計環案例呈現給大家。(由於部分設計有保密性，4R此處用Canal4R代替)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上，就是我和大家分享的全部內容啦，謝謝大家!"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：訂單表中，如果有一些商品id，那麼同步到ES中也是id嗎，不會關聯出name打成寬表存到ES嗎?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A1："},{"type":"text","text":"具體的映射字段需要在Adapter中配置映射即可，存入到ES中的情況也與配置的映射是直接且唯一關係。是否寬表要在實際應用中把控字段的個數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2：Canal部署deploy主從和canal-adapter有沒有遇到官方的bug？有，改動了哪些？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A2："},{"type":"text","text":"遇到過Column not match的異常.具體看Canal的TSDB來解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3：這套複雜度方法論如何落地到實際應用？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A3："},{"type":"text","text":"需要對系統進行4R視角拆分、識別複雜度類型並按照架構設計環的方式來評定需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4：平時的Canal有消息延遲嗎？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A4："},{"type":"text","text":"有一定延遲的，binlog的數量、網絡等因素，都會造成一定的延遲，所以，建議異構還是要建立在業務數據可延遲的基礎上的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5：我主要用canal-adapter讀取Kafka中的binlog日誌然後寫到數據庫中，Kafka中有多個表的日誌，我rdb目錄下的yml文件只配置了一個表的爲什麼其他的表也會同步？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A5:"},{"type":"text","text":"Yml的作用是配置映射關係，具體的過濾職能在Deployer的Sink配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q6：異構數據是直接同步原表嗎，還是做了關聯？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A6: "},{"type":"text","text":"做了關聯，直接在Adapter中配置對應映射關係即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q7：請問爲什麼不直接增加熱集羣的節點和分片，而是重新建一套ES集羣呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A7: "},{"type":"text","text":"這裏主要還是一個數據拆分的思想，如果通過提高配置來解決訪問量問題，那麼，隨着業務量級增加，流量混在一起，對應的ES集羣流程會呈現不可評估的情況。本質上還是一個數據存儲職責的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q8：如何保證Zookeeper的高可用？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A8: "},{"type":"text","text":"Zookeeper本身就是高可用的，如果想在機房或異地方面做高可用，建議做主備同步、多集羣部署等手段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q9：新集羣的查詢請求峯值是多少?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A9: "},{"type":"text","text":"大約2000-4000 QPS。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q10：怎麼把握冗餘尺度呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A10: "},{"type":"text","text":"冗餘的維度在機器、機房、地區、國家是不斷增加的。維度越大，對應的高可用方案越可靠，但是，對應的費用以及實現複雜度也會變高。因爲這種冗餘方案肯定會有數據copy。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"嘉賓介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"張磊，"},{"type":"text","text":"京東到家高級研發工程師。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"8年+軟件研發經驗，曾先後就職於鏈家地產、互動吧、寺庫網等公司，任研發人員或團隊leader，在解決業務系統設計落地方面擁有豐富經驗；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現就職於達達-京東到家，主要負責訂單履約、金額拆分、計費相關業務域的研發設計工作。業餘時間主攻軟件複雜度優化。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/juMj7eUKbFKX0xbyQ8781g","title":"xxx","type":null},"content":[{"type":"text","text":"大促突圍：京東到家基於Canal的數據異構設計"}]}]}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章