超3億活躍用戶的多活架構，數據同步與流量調度怎麼做？

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、多活業務架構"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、OPPO多活架構原則"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一，主線多活。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多活成本比較高的，雙活是兩倍，三活可能成本會低一些，但三活的難度更大。因此沒有辦法對所有業務進行多活，只能對主線做多活。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二，是保障多數用戶。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉個例子，系統有個充值的功能，充值功能本身是強一致的，完全不能允許任何的延遲或者是副本的讀。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是多活切換之後，只有少數用戶在切換的前幾分鐘有充值的，這部分用戶餘額可能沒有通過過去，只需要對這部分用戶進行服務降級，其他絕大多數用戶是可以使用完整的服務的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第三，數據分類，應用不同的CAP模型。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CAP定理不是針對的業務功能，比如說賬號、支付、登錄，CAP定理是對數據的要求。一個功能可能用到多個數據，數據本身的一致性、可用性、延遲的容忍是不一樣的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以需要對業務功能用到的數據進行分類，比如餘額數據、流水數據、日誌數據、個人資料數據……我們對每個數據進行一致性、可用性的需求分析，一致性要求很強，這個數據就選用同城高可用的數據庫服務。這個數據一致性要求不高、允許延遲，就可以選擇異地高可用的數據庫服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以這個業務來說不是整體使用一個CAP模型，在業務內部，因爲不同的數據分類，使用了不同的模型，因此業務有時候存在部分降級的情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第四，平臺業務SDK化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OPPO的業務比較多，比如瀏覽器、軟件商店、廣告務、音樂、視頻等非常多的業務，這些業務都用了平臺化的服務，比如評論系統、消息系統，還有賬號鑑權的系統等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OPPO公司的機房比較多，主要的就有好幾個機房，我們的上層業務是分佈在不同的機房裏面去，這對平臺業務來說就比較麻煩，上層業務可能只需要做雙活就行了，而平臺業務可能就要做七活、甚至八活，而且七、八個機房都要有讀和寫，難度就非常大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲解決這個問題，我們提出平臺業務進行SDK化思路，把這種平臺型業務，拆分成獨立的域名，從SDK開始拆分，這樣我們平臺業務只需要單獨做多活就行了，不需要在每個機房都提供讀寫的能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第五，數據最終一致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第六，我們的記錄日誌、流水，避免修改、計數操作。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、同城多活業務架構"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d8\/d8e5f10c13091843bc3355eeab9c0a77.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是典型同城多活的業務架構，應用層是完全無狀態的，隨便打流量。四層採用DPDK技術開發，七層包括Nginx和API網關兩個組件，Nginx只用來做SSL卸載、WAF防火牆，其他功能都是API網關來提供。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據層以主備爲主，寫流量只會寫到Master節點，但是讀的流量可以訪問slave節點，但是也不一定，看業務本身數據一致性要求，如果要求非常強一致的，我們的讀也只會指向Master節點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意，我們把Nginx和API網關都放到同一個容器中，兩隻之間採用進程間通信。這樣的好處是，我們擴容的時候，我們可以將整個七層同步去擴容，而不會存在某一層組件容量不足的情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外就是註冊中心，我們沒有使用 k8s本身的一個註冊功能，而是自己基於數據庫，實現了AP模型的註冊中心，保證註冊中心的跨機房高可用。同時註冊中心兼容Consul協議，從而更好的融入開源生態。多個k8s集羣的實例，都會註冊到統一的註冊中心裏面去。這個註冊的動作，是由發佈平臺完成的，好處是應用發佈的時候，發佈平臺可以提前摘掉流量，避免重啓影響服務的成功率。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、異地多活業務架構——單元化"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/df\/df0ca85ffa5b12ceb3cbb3b9475fda0e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異地多活，比較典型的架構是單元化，就是將用戶進行分片，將不同的用戶分片放到不同的機房裏面去，這樣可以做到一個完全的擴展，隨着用戶規模的增加，我們可以很容易去擴展機房的數量，這些都可以持續的去增加的，包括每個機房的容量也可能不一樣。比如有的機房大，有的機房小，我們可以調整每個機房存放的單元數量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏確實實現了多活，每個機房都有流量，每個機房也是讀寫，是完全的多活，但是單元高可用的問題如何解決，單元的歸屬機房故障了，如果把這個單元轉移另一個機房繼續提供服務。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、異地雙活業務架構"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5d\/5d6cccede2e5ce7691621f432ff33f88.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是我們使用較多的異地雙活架構，首先我們將用戶按照地域維度進行了一個單元劃分，比如說按照地域將用戶劃分爲七個大區單元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意這個單元劃分是用戶首次訪問服務的時候進行的，然後客戶端就保存了單元號，就不會產生變化了，所以用戶出差，換到因爲另外一個地域裏面去，它所屬的單元號我們是不會變化的，還是訪問單元歸屬的機房，這個時候可能就不是訪問最優的機房。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣的好處是，當一個用戶移動的時候，數據訪問就不會在兩個機房之間跳來跳去，避免雙向同步的數據衝突問題，很容易實施。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據層在兩個機房，都是完全全量的，兩個機房間數據是做雙向同步，沒有誰是主誰是從的區分，是完全對等的架構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶流量調度按單元進行，這樣可以保證一個用戶，他只會訪問其中一個機房，不會在南北兩個機房之間跳來跳去，就算是用戶出差也是如此，按照首次訪問服務時的地域來劃分的單元。只要我們的調度規則沒有變更的情況下，一個用戶他永遠只會在其中一個機房讀寫，這樣的好處就是，第一個可以避免我們的同步的衝突，第二個好處就是容忍了數據延遲的情況，比如說一個用戶他永遠是看到北方機房，南北之間數據同步的延遲日常情況下其實是感知不到的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個架構是非常簡單，只需要在客戶端網絡庫裏面做一些封裝，對用戶進行單元劃分，按單元進行流量調度就可以了，雙向同步比較好實施，延遲、衝突，這些問題都可以避免。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了地域之外，也可以按照賬號或者設備來劃分單元。按賬號或者設備劃分單元的好處是，如果按照地域劃分單元，在用戶刪除手機APP這種情況下，APP裏面保存的單元號就沒有了，下次訪問服務的時候就需要重新分配單元號，因爲地域可能和之前不同了，就可能分配到不同的單元號，按賬號或者設備劃分就沒有這個問題，重新分配還是原來的單元號。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面說到，南北機房的數據層都是全量的，一般情況下，按地域的劃分單元的模式，就算重新分配了單元號，也不影響數據的讀寫訪問。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5、異地雙活——評論系統案例"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/88\/887ab5ccf197899104e9aa2fe39ea271.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是平臺型業務-評論系統異地生活的案例，評論系統從SDK開始，就進行了域名拆分，避免了在業務域名所在機房內部去做跨機房的評論服務調用，影響服務的可用性和性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示，我們只對MySQL原始數據層做了南北雙機房同步，第二層的評論元數據表，還有第三層的一個Cache，這兩層實際上沒做同步的。兩個機房分別基於MySQL數據獨自去重建第二層的元數據表，第三層的Cache，以及重建其他的數據源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣好處就是，我們只有一個數據源做了南北機房的同步，就可以避免雙數據源同步的時候，兩個數據源之間會存在同步的進度不一致，從而兩個數據源之間的依賴關係出現問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉個例子，我們上面的評論表、點贊表這一層，最上面這一層做了同步以後，我們中間第二層如果也做了同步，然後第二層同步以後，兩個數據可能存在差異，比如說第一層同步快一點，第二層同步得慢一點，同樣是南方的用戶，他們看到這個數據之間的存在不匹配的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲用戶流量調度是按單元進行的，兩個機房的數據雖然有差異，有延遲，但是用戶感知不到的。一個用戶要麼看到南方機房，要麼看到北方機房，我們評論數量兩個機房有差異，點贊數量有差異，回覆數都有差異，但是無所謂，用戶是感知不到差異的。需要注意的一點，就是當多活切換的時候，用戶能感知到一個差異，但日常情況下用戶感知不到這個差異。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6、異地N活業務架構"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/16\/16ffa4ab1491c9f220ef8bbe4d91d39a.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是比較複雜異地N活業務架構。它基本的思路就是對用戶進行兩級的劃分。第一級按照設備和賬號劃分單元，其中單元裏面既有登錄的用戶，也有未登錄的用戶。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在第二級劃分的單元內部，我們再應用異地雙活的模式，或者是同城多活的模式，比如說左邊單元1，按照地域做第二級的劃分，把它劃分成南北兩個副本，既然是副本，肯定數據是全量的，是異地雙活模式，兩個副本數據做雙向同步，這種模式適用非強一致的業務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼強一致的業務怎麼辦呢？比如右邊的單元4，跨同城的兩個機房，單元內部採用同城多活的模式，就是共享跨機房高可用的數據層，是主備的的。這種模式適合強一致的業務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面說了單元內部主要兩種模式，第一種是異地雙活，雙向同步，主主模式，讀寫在本機房，然後做雙向同步；第二種是同城多活，主備的模式，跨機房共享主備切換的數據層。除此之外，單元內部還可以選擇主從，冷備等模式。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7、服務部署"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/13\/135272b1c4cc3a7a148b441c7ad640dd.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是服務部署架構。服務部署分爲幾大部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一部分是中心域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中心域主要是部署一些運營管理後臺，還有一些爬蟲，還有一些非常長尾的應用，但這些業務可能不太重要，也不需要做一個多活。中心域的讀寫都是在中心機房，然後把數據單向同步到其他單元機房。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二部分是全局域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全局域主要存放非單元分片維度的數據，比如評論、消息等。這些數據不能按統一維度進行拆分，需要全量的訪問，放到全局域的數據都是全量的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第三部分是單元域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存放按單元拆分的數據，比如用戶訂單、收藏、下載記錄等。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"8、服務路由"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/12\/1252c098944518df49fd9e03cd108e15.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶會先請求到API網關，API網關根據請求的單元號參數，判斷是是否訪問錯了機房，如果訪問錯了，就做重定向，或者跨機房轉發，用戶自己選擇的其中一種模式。轉發的模式比較依賴於兩個機房之間的專線的帶寬和穩定性，重定向模式機房之間的帶寬要求會低一些，客戶端重新發起請求，這兩個機房之間的網絡專線要求低一些。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面說到用戶首次請求的時候，會給客戶端分配一個單元號，這個單元號將會存儲起來，以後每次業務請求都會帶上這個單元號。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"請求到了單元內部，單元號會做一個全鏈路的傳遞，全鏈路傳遞是通過調用鏈來實現的，調用鏈可以把一些參數做全鏈路的傳遞。應用實例打上了單元號的標籤，微服務調用方通過單元號對實例進行篩選，防止請求打到其他單元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據訪問層要做一個兜底的操作，可能由於服務路由還是其他的一些原因，不小心訪問錯了單元，這個數據層有可能訪問錯，所以數據訪問層要做一個兜底，根據傳過來單元號，做拒絕或者轉發。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"9、用單元化解決業務擴展性問題"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/04\/04e73d646acb9e469201fdc3a36158a1.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單元化不只是可以用來解決多活的問題，也可以用來解決業務擴展性問題。在一個機房內部，如果服務1000萬用戶，他可能需要10個數據庫，服務1億個用戶，需要100個數據庫，如果100個數據庫讓每個應用實例都連上的話，連接數就太多了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以在一個機房內部也拆分多個單元，每個單元保證1000萬、2000萬左右的用戶，隨着用戶的增長，我們再將單元數量進行增加就行了，這樣就可以保證每一個單元內部的服務規模受控。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、多活數據同步"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、MySQL同城多活"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fb\/fb460ae68657713bc523b0cba763d1b5.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是MySQL同城多活架構，MySQL對外看上去是一個集羣，只有一個IP。我們需要解決的問題是：怎麼讓跨機房的集羣看到的是同一個IP？這裏就用到了Anycast技術，IP的作用可以理解爲域名，我們把一個 VIP用Anycast技術，將它路由到兩個機房，或者是三個機房。我們是路由到三個機房，然後就到了機房內部，再通過 ECMP協議將流量再分到多個四層負載均衡節點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過Anycast第一層路由到不同的機房，第二層的ECMP再路由到基於DPDK技術開發的四層負載均衡節點。這樣我們整個的數據庫對外看到的VIP就是同一個了，所有機房看到VIP都是同一個。利用Anycast和ECMP兩個技術，實現跨AZ共享VIP。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後是數據層，數據層我們現在是一主三從，然後需要2個以上slave同步成功，才能完成最終的成功。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL版本需要5.7以上，操作系統內核需要打一個 toa補丁，這樣經過四層負載均衡之後，MySQL Server才能拿到真正來源IP。因爲我們這邊要做一個IP白名單的授權，如果不打補丁，拿到的來源IP就是四層負載均衡的IP，就沒法做IP白名單授權了。當然top補丁有一個缺陷，就是隻能支持ipv4，這在內網使用問題不大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"底層採用了開源的MySQL拓撲管理組件，通過檢測我們數據庫節點的情況，然後做重新選組做切換，然後通知SLB改變後端指向，流量打到新的master節點，"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Anycast不是必須的，也可以用域名代替，但是域名有個問題，需要重新接連的時候纔會發起解析，所以域名切換的時候可能會切不乾淨。Anycast做切換是立即生效的，因爲這是路由協議的一個變更，馬上就能切過去，不存在解析不乾淨和生效不一致的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Anycast除了內網之外，外網也用的比較多，比如說谷歌上負載均衡器，它發佈的IP就是Anycast的IP，在公網環境下，在不同的地區路由到不同的一個真實地址，包括我們 DNS Server也是用Anycast去發佈的，在不同的區域，路由到就近的IDC，所以Anycast技術應用還比較廣泛。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、MySQL異地多活"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3f\/3fa08b063524d4dc549f9127ca5d2e7a.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是MySQL的異地多活架構，重點在於提升同步的性能，從源庫訂閱到數據以後，不是直接寫目標庫，而是先存起來，在目標機房部署中繼日誌模塊。這樣的好處是，我們可以在網絡上快速的傳輸過去，中繼日誌並行去寫目標庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個設計性能提升非常大，OPPO實際業務場景下，這個模式比訂閱後直接寫目標庫提升了幾倍。因爲引入了中繼日誌，就存在兩階段提交的問題。比如中繼日誌寫成功，但是中繼日誌寫目標庫沒有成功。這就存在數據一致性問題，需要用到兩階段提交。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有就是數據壓縮和加密，對數據的安全和同步性能也非常重要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後是多消費者支持，訂閱模塊會保存數據，每個訂閱方可以維持自己的消費位點，彼此之間沒有干擾，從而減少多訂閱方同步對 Source DB的壓力。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、MySQL訂閱——數據最終一致"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6a\/6ae949be8d9cec2c1b7029e546c0037b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以前面提到的評論系統爲例，數據同步只同步MySQL那一層，而其他的數據源Cache、MQ、ES、排序服務等，分別訂閱MySQL binlog重新構建。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原則上，我們儘量只同步底層的一份MySQL數據，其他數據源訂閱MySQL重建。前面說到，MySQL只需要訂閱一次，Jins程序自己存儲了一份數據到本地文件隊列，然後分別重放到Cache、MQ、ES等其他數據源，也可以多次重放數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果多數據源分別進行同步的話，多個數據源同步的進度是沒法保證協調一致的，必然有的數據源快，有的數據源慢，這有可能導致兩個數據源之間的關聯關係出現一些程序錯誤。所以我們儘量只同步一個數據源，再基於MySQL重建其他的數據源，避免進度不一致的問題。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、MySQL數據對比&修復"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f9\/f9fb54b843f8a8c84c8635dfbd9c679f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OPPO的業務場景，很多地方都非常依賴底層的 MySQL數據同步，兩個機房之間之間到底有沒有差異，是蠻重要的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此我們設計了一個獨立的MySQL比對修復工具，就執行上圖這樣一個SQL語句，通過這個SQL語句，對一段時間之內的所有數據算一個異或的值，通過異或值去比對兩個機房之間數據差異，如果比對有差異，我們再縮小比對範圍，逐步逼近到差異的記錄行，這個語句的執行效率還是蠻高的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是這個方案有個不足，要求我們數據庫裏面有一個時間戳的字段，程序會對比前一個週期內的所有記錄的異或值，判斷兩個機房之間數據是否有差異。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外一重要場景就是數據修復，因爲業務可能配置錯了數據庫、應用實例配置生效不一致，再比如A單元數據寫到B單元，這個時候需要修復數據，通過這個工具，把兩個數據庫不一致的數據行整理出來，然後人工做識別或者批量修復。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5、Redis多活"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a6\/a65bae787b10ce64510b19e117ca7892.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Redis同城多活的架構如上圖所示，我們在Redis Server上面做了一層代理，下層Redis Server沒有使用Redis cluster技術，代理將流量進行分片，分發到了不同的Redis Group裏面去，每個Group裏面就是普通的Redis主從。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主從之間採用binlog的同步，因爲Redis本身沒有binlog，我們把 AOF做了改造，把讓它變成binlog的這種格式，這裏改造的工作量不大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後代理也支持兩種模式，一種是重定向模式，一種是轉發模式。轉發模式就是寫主讀從，它只會把寫流量轉到了主機房裏面去，但是從機房是能讀的。重定向模式就不一樣，重定向模式是非常更強一致的，讀寫都只能在主機房。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面反覆提到，CAP是針對數據的，是指數據本身的延遲或者差異的容忍度，所以這兩種模式都需要支持，有的數據它就是要強一致，一定要到主庫裏面的去讀，但有的數據它允許從庫讀，允許延遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/15\/15078e6aedf3c51afe61973849930b43.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異地多活也很簡單，異地多活兩個機房各部署一個組件去訂閱同機房的Redis，訂閱Redis的binlog，訂閱的數據寫到MQ裏面去，兩個機房分別重放binlog，實現起來並不複雜。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後簡單說一下binlog的格式，裏面包括了命令、數據產生的機房、遞增的序號，還有一個時間戳。還需要注意的一點，Redis持久化RDB也要改造一下，RDB需要包含一個 binlog offset，binlog讀取偏移量，需要把它記下來，因爲主從顛倒的時候，我們訂閱程序要重新從offset開始繼續訂閱下面的命令。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、GSLB流量調度"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、Http DNS"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b3\/b399813a6ba72a9388675f105d4f5c69.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後講我們的GSLB流量調度，首先是爲什麼要使用Http DNS。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一個是防劫持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DNS劫持，DNS是多級緩存，部分環節存在解析劫持的情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DNS黑洞，這個大家可能遇到比較少，什麼叫DNS黑洞呢？就是運營商監控到某個域名有惡意的請求，封殺他的時候不小心擴大了封殺的範圍，我們已經出現過幾次這種情況，有時候某個地區甚至可以把整個cn頂級域名全封殺，這種封殺的範圍很大，稱之爲DNS黑洞。整個2020年已經發生過多次這種情況了，某個地域整個頂級域名都給你封殺掉，大家都解決不了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二個是快速生效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是DNS本身的多級緩存，這個時間不受控制，但它可能不是主要問題，更主要的問題是客戶端長連接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們還沒上Http DNS之前，業務使用了客戶端長連接，需要20分鐘甚至一個小時才能大部分流量調度走。主要的原因就在客戶端長連接，DNS做了變更以後，只有客戶端重新發起連接的時候，它纔會發重新發起解析，纔拿到新的IP，如果連接沒斷開，就一直不會轉移，所以這部分長連接用戶根本就切不走。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果是機房入口網絡故障還好，連接天然會斷開，如果是因爲業務自己的問題，需要把流量切走，這種情況下就會發現根本切不走，所以客戶端長連接是比較重要的問題。所以客戶端網絡庫需要處理一下，解析變更的時候，需要主動去關閉連接，但是傳統DNS，沒有解析變更的通知機制，不發起解析就不知道解析變更了，這裏就進入了循環了，需要仔細的思考一下流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第三個是精準調度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的DNS解析只能獲取到IP這一個參數，首先IP信息不準確，包括運營商歸屬、地域歸屬，都不是很準確，國外運營商特別多，情況更嚴重。現在IPv6也在快速的推廣，信息不準確的情況更爲嚴重。其次傳統DNS無法做到用戶維度、設備維度的解析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"最後是生效一致性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單元一旦發生調度以後，在單元內的所有用戶要同時調走，不能說一部分先調走，一部分後調走，這樣數據寫入就亂了，需要保證全體用戶生效的一致性。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、單元調度"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/08\/0820d6ef23dc8fb36afe19d2cbe64a69.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"下面講單元調度的主要流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一步: 劃分用戶單元"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"劃分用戶單元主要有三種模式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按設備劃分單元；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按賬號劃分單元；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按地域劃分單元。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏有個地方需要注意，我們爲了劃分單元，客戶端肯定要傳一些參數，如果按賬號劃分單元，需要傳賬號ID；按設備劃分單元，需要傳設備的IMEI,或者國內Android廠商推行的OpenID；按地域劃分單元很簡單，直接從IP裏面可以獲取，不用客戶端傳遞參數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲隱私合規要求，比如說海外業務，直接傳用戶的ID或設備信息，是違規的，因爲我們這個調度的域名它是一個獨立的域名，它不是業務本身的，這個域名很難跟用戶解釋，即使跟用戶簽了協議，因爲業務主體的不同，可能也不一定包含了這個域名，所以我們做了一個匿名化處理，設計了兩個新參數，一個叫ADG（匿名設備分組），一個叫AUG（匿名用戶分組）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將賬號ID和10萬取模的值定義爲AUG,設備ID與10萬取模的值定義爲ADG。通過這種方式，把設備和賬號分成10萬個桶，然後對桶分單元，比如說1~5000桶是單元1，5000~1萬桶是單元2。這樣我們就不用傳真實的設備ID和真實賬號。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二步：客戶端獲取單元號。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端首次訪問業務的時候要分一個單元號，這樣就算按地域劃分單元，基本上也不會出現變更，只要用戶的APP不被刪除，我們OPPO手機的好處就是，我們的APP是不怕被刪除的，我們的數據不會被清掉；但如果是一個外發的APP，可能就存在APP刪除，這個可以考慮用設備或者賬號分單元。獲取到單元號之後，就永久保存在客戶端。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第三步：客戶端解析域名IP。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"域名解析的時候會帶上單元號的參數，獲取這個單元對應的IP列表，然後客戶端緩存IP列表。需要注意的一點是緩存機制，建議根據網絡環境進行緩存，比如WiFi名稱，或者運營商的名稱，底層的緩存數據結構就是域名加上網絡環境的名稱。這樣的好處就是，用戶網絡切換的時候，比如說家裏面是WiFi，我們拿的是IP1，我們一出門，網絡環境變了，我們取出的緩存IP就是IP2，在每個網絡環境都是緩存最優的IP。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外一點需要注意是：我們爲每個單元還分配了一個單元域名，這是一個傳統DNS域名，主要是降級的時候使用。可以設想一下，如果我們沒有爲每個單元單獨分配一個傳統DNS域名，一旦降級的時候就會走到業務的主域名，而傳統DNS是不能攜帶任何參數的，無法做到按單元進行解析，用戶流量就全都亂了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以每個單元分配一個域名的好處就是，降級的時候只要降級到我們這個單元的域名，這樣大多數用戶解析結果還是準確的，不準確的一部分通過API網關重定向或者內部轉發，只要很少用戶需要走這個路徑，絕大多數用戶還是最佳的路徑。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第四步：客戶端重定向。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲調度過程當中還有一部分用戶在訪問舊的IP，我們是通過API網關，把新機房IP直接告訴客戶端，客戶端立即用新IP重試，並且異步去刷新解析，如果只是反饋一個狀態碼，告訴客戶端需要重新刷新解析，客戶端的總請求時間就會拉得比較長，這就是重定向模式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但除了重定向模式，還有轉發模式，但是轉發模式比較依賴機房之間的專線帶寬和穩定性，如果公司規模不是很大的話，機房之間的專線帶寬和穩定性可能趕不上公網，重定向模式可能更適合一些。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、單元調度注意事項"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d3\/d32a671886c403e365ef0f7610becab1.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據層聯動，舉一個用戶餘額充值的例子，這是非常強一致的，我們可以維護一個數據不一致用戶清單，比如說有用戶剛剛進行了充值，這個數據還沒在各個機房達成一致，機房調度的時候，只是這一部分用戶需要進行服務降級，其他用戶還可以繼續提供完整的服務。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、域名解析刷新時機"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/22\/22695095ed4a4dffd6027a794ca16cd4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來講域名解析刷新時機。因爲HttpDNS是直連解析的，不像傳統DNS有多級的緩存，如果我們還沿用傳統DNS的 TTL方式來刷新解析，這個TTL就不能設置的太短，太短了HttpDNS Server的壓力非常大。TTL設置過長又不能滿足業務快速恢復的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以域名解析的及時刷新依賴另外兩種途徑，第一種途徑是失敗。我們請求一個服務，要麼連接錯誤，要麼響應內容出現錯誤，比如說我們響應了500，或者其他我們認可的一個響應值（客戶端可以自己定一個規則），我們訪問失敗的時候，就需要立即去刷新一下域名解析，因爲請求失敗的時候可能需要做一個機房調度，不管是業務後端出現了問題，或者是連接不上，這種情況都需要做機房調度，需要客戶端刷新解析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二種途徑是指令，如果是因爲我們帶寬不足，做活動，或者其他原因的，需要把流量切走，這時我們可以通過API網關下發指令，下發指令也是隨着API網關的正常的業務請求，響應Header帶下去，不是單獨的通道，也不是通過Push推送。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣我們就可以兜底，要麼會請求失敗，會立即刷新解析。要麼請求成功，響應header就會攜帶指令。所以用戶一定能走到失敗和指令其中一條路徑。因此我們做了調度變更以後，用戶一定會刷新，不再依賴TTL了，過期時間可以設置非常長，這樣我們絕大多數請求，都不會發生真正的解析請求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通常情況下，傳統DNS有2%~3%的解析失敗率，還是挺高的。通過這種方式，我們就可以把解析成功率做到99.5%以上，日常情況下甚至能做到接近100%。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5、調度生效一致性"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/df\/dfffa110f9fe9bc513e1d619249d93ec.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面講一講調度生效的一致性，當我們的客戶端降級到傳統DNS的時候，就會解析到錯誤的機房，在調用生效過程當中，也會訪問到舊的機房，所以我們在API網關會做一個攔截，因爲每個請求都帶上了單元號，API網關就可以判斷這個請求是否請求到了正確的機房，如果請求錯了機房，API網關把請求定向單元當前歸屬的機房。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定向用戶請求有兩種模式，一種是轉發模式，API網關直接轉發到新機房的業務後端實例。另一種是重定向模式，API網關在響應header攜帶了重定向指令，以及新機房的IP（避免客戶端多一次的請求），客戶端立即重試新IP。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"轉發模式需要消耗較多的機房專線帶寬，重定向模式的總體時長更高，業務可以自由選擇兩種模式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解析刷新採用並行跑馬的模式，客戶端會並行請求兩個HttpDNS Server和一個傳統DNS，三個請求同時發出去。如果HttpDNS Server請求成功，哪個先到就用哪個，如果兩個HttpDNS Server請求都失敗，就使用傳統DNS解析結果。因爲每個單元都分配一個傳統域名，所以傳統DNS解析結果和HttpDNS解析結果也基本是一致，只有極少數用戶會解析錯誤，API網關重定向一次以後也能糾正過來。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6、調度決策大腦"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/8f\/8f519fc3f1e5420608d1fb996c8dc45b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"調度決策大腦會收集很多路的原始監控數據，比如客戶端調用鏈的數據、外網撥測平臺的數據、機房網絡監控的數據等等，多路數據彙總到決策大腦裏，進行比對分析，得出故障的結論。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"調度決策大腦一定要依賴多路監控數據源，因爲單路數據的質量無法保證，比如可能會出現撥測用例配置錯誤、網絡監控數據丟失等，所以單路數據都是不可信的，需要多路數據源做交叉的比對，過濾抖動、防止誤判。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"調度決策大腦最終會輸出一個指令，指令只會告訴你故障類型，比如：機房故障、運營商線路故障、機房之間網絡（DCI)故障，或者是容量不足、業務自身出現了問題等。業務自身出現問題，比如業務的數據庫故障，也需要切到另外的機房去。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"決策指令同時發到兩個地方，既要發給接入層，也要發給數據層，爲什麼需要這樣呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假設我們同城兩個機房之間，專線出現了故障，兩個機房的數據庫肯定達不成一致，同步不過去了。這個情況下，假設我們的數據庫選主B機房，而接入層保留A機房， A機房的數據庫完全寫不進去，即使寫進去也是錯誤的，這裏我們要保證數據層和接入層兩邊選擇的機房要一致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以這種專線故障情況下，我們是調度決策大腦來通知，做統一的決策，同時通知接入層、數據層做聯動，選擇同一機房，這個主機房的選擇是事先配置好的，它不是由我們剛剛說的Raft組件來解決的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7、調度效果"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/45\/4534e7a8ae1daacd43c4f18da7328219.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是我們9月份做過的一次機房調度的效果，基本上做到分鐘級（實際上是秒級的）的生效，是很陡的一個曲線。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，給大家總結一下今天分享的內容："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/11\/1122255f55d69aefe03f6b0fb04e338a.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":">>>>"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：Http DNS也有緩存的吧？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A1 ："},{"type":"text","text":"對，剛剛提到我們Http DNS緩存時間非常長，緩存了一週的時間，而且緩存的時候是根據環境來緩存的，就是按照 WiFi名稱、運營商的名稱來做緩存，這樣網絡切換的時候可以拿到最優的IP。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存的時間非常長，是因爲域名解析的刷新，是不依賴緩存過期的，如果能請求成功，API網關在響應Header就會帶上調度指令，如果請求失敗客戶端也會主動去刷新解析。因此解析的刷新，是不依賴緩存過期時間的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2：同城多活網絡是怎麼配置的？兩個機房使用相同的ip地址，還是不同的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A2 ："},{"type":"text","text":"對於跨機房高可用的數據庫來說，用戶看到的是同一個IP，第一層使用Anycast路由到機房，第二層使用ECMP路由到多個四層負載均衡節點，單個四層負載均衡的流量扛不住，四層負載均衡是一個集羣，通過ECMP實現流量分發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多餘入口流量來說，前面架構圖可以看到，接入層在兩個機房從四層、七層都是獨立的，接入層有2組出口IP，如果其中一個機房運營商線路出現問題，根據調度決策系統的指令，自動停止該運營商線路的IP解析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3：老師能介紹一下多活帶來什麼業務收益嗎？是什麼契機促使 OPPO開始做異地多活？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A3 ："},{"type":"text","text":"OPPO業務多活的三個核心訴求是成本、擴展、容災。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"成本是指業務總體技術運營成本，包括基礎設施的資源成本、研發成本，還包括業務中斷的成本、品牌和口碑的成本；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"擴展是因爲業務規模過大，一個服務需要調用數百個三方實例、一個數據庫被數百個實例連接、一個服務需要連接幾十個數據庫，這就需要對用戶進行分片，縮小業務規模，自然演進到單元化多活的架構；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容災一方面是極端情況下用戶數據可靠性保障的需求，另一方面還是業務過於複雜、處理的鏈路很長，總有一些意想不到情況的發生，頻率還挺高，問題定位到恢復的時間達不到公司RTO的要求。機房內部共享了運營商線路、DNS、SNAT防火牆、負載均衡、K8S集羣、註冊中心、監控等等資源，而機房之間是相對隔離的環境，同時出問題的概率大爲降低。在業務出現無法自動恢復的故障時，先切換機房恢復業務，然後再從容定位問題根因。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4：隨着業務發展啓用多個訂閱時，如何減少對數據庫的壓力？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A4 ："},{"type":"text","text":"我們從數據庫源庫訂閱出來以後，先落地到本地文件隊列，然後多個訂閱方可以維持自己的同步位點，所以對於源數據庫來說，只會有一次訂閱。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5：請問同城雙活方案MHA manager部署在哪個數據中心？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A5 ："},{"type":"text","text":"我們這裏不是MHA，我們用的是一個開源的Raft組件，部署在同城的3個機房，通過Raft組件檢測數據庫的狀態、觸發切換。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q6：Http DNS和Local DNS的區別是什麼？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A6 ："},{"type":"text","text":"Http DNS走的是HTTP協議，客戶端直連解析，沒有運營商的多級緩存。Local DNS就是運營商的DNS，成功率低，還有劫持、黑洞等問題，而且這兩年黑洞頻率是越來越高了，前幾年基本上很少出現黑洞情況。傳統DNS劫持情況現在好一些了，像移動端的接口劫持相對來說會少一些，H5的劫持多一點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Http DNS就是依賴HTTP協議做解析，但這個壓力會比較大，因爲Http DNS沒有多級緩存，所有請求都到我們的機房，所以刷新機制的設計就非常關鍵，前面一個章節詳細介紹解析刷新的時機。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HttpDNS還有一個好處，因爲是自定義的協議，可以傳遞其他參數，比如設備信息、賬號信息，這樣才能夠實現按用戶單元進行解析、調度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q7：能否制定統一的用戶單元劃分規則？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A7 ："},{"type":"text","text":"這個問題比較好，我們最開始也是想這樣子的，我們有云服務、廣告、信息流、音樂、視頻等業務，起初也想整個公司使用一套單元劃分規則，這樣業務之間可以做到單元內封閉調用，避免跨機房的調用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終的方案，業務之間沒有使用同一套單元劃分規則。主要原因是：比如說有個業務他經常會做活動，做活動的時候他需要將一部分用戶調度走，如果全公司用一套規則的話，所有業務都要跟着調度走，其他業務是不同意的。所以我們是每個業務自己制定單元劃分規則。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那這裏怎樣解決業務之間跨機房調用呢？前面說到了平臺型業務SDK化，上層業務之間本身沒有強依賴，音樂、軟件商店、視頻之間本身沒有強依賴，他們主要是依賴平臺型服務，如賬號、評論服務、消息中心、推送服務等。這些平臺型業務我們最開始也是提供機房內部API去給其他業務器調用，這就導致我們的平臺型服務在每一個機房都要去部署，每個機房都要提供讀寫功能。所以我們將平臺型域名拆分出來，從SDK就開始就和業務域名分開，平臺型自己做多活。當然平臺型業務無法做到100%的SDK化拆分，平臺型服務的部分數據也需要單向同步到各機房，提供本地查詢的服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q8：Redis日誌是哪個開源組件做到的來的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A8 ："},{"type":"text","text":"Redis binlog是OPPO自己修改的，基於AOF修改，簡單說一下binlog的格式，裏面包括了命令、數據產生的機房、遞增的序號，還有一個時間戳。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相關的組件今年會開源出去，OPPO微服務體系ESA Stack、存儲已經對外開源，可以搜索一下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q9：MQ數據同步了，如果切換，在A機房寫入了，但是還沒消費，此時切到B機房，B機房咋知道從那個點開始消費呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A9 ："},{"type":"text","text":"OPPO主要使用RocketMQ定製版本，節點跨同城AZ部署即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"嘉賓介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"羅代均，"},{"type":"text","text":"OPPO安第斯系統資深工程師。負責OPPO後端體系建設，包括API網卡、用戶流量調度、微服框架、調用鏈路跟蹤等，經歷了OPPO用戶從千萬級到億級的增長曆程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/DKmHORPEa5r-YFmeUhXtbA","title":"xxx","type":null},"content":[{"type":"text","text":"超3億活躍用戶的多活架構，數據同步與流量調度怎麼做？"}]}]}]}