金融IT保命丹：多端支付強一致性架構設計實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在線下使用現金交易的時候，我們可能都會認爲抹個零頭、少幾毛幾塊都是不大的問題，但在平臺上，1分錢計算錯誤都會釀成悲劇。相信從事金融相關係統的同學都有感觸，在處理計算金額邏輯的時候，大家都會十分謹慎，會多次複查代碼，"},{"type":"text","marks":[{"type":"strong"}],"text":"準確性、一致性，是支付系統的首要指標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作爲互聯網公司，“快”是核心必要指標。"},{"type":"text","text":"尤其我們作爲以實時性需求的o2o（Online To Offline）電商公司，整個訂單的生命週期不到1小時，如果支付延遲幾分鐘，履約的質量就會下降，甚至會出現用戶需求減少的情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以架構的整體層面上，在保證了系統的安全穩定性的同時，我們要儘可能保證系統的高性能運轉，"},{"type":"text","marks":[{"type":"strong"}],"text":"解決互聯網的“快”和金融行業的“穩”之間的矛盾。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面，介紹一下本次分享的主要內容："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一、支付平臺整體能力介紹"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二、如何安全保障金額正確性"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三、高可用架構的一些實踐及思路"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、支付平臺整體能力介紹"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、支付平臺的整體架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是整個支付平臺的全貌。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/25\/25783143486204065340981b0c84b0c9.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲平臺系統，應該具備：產品通用能力、個性化可配置。目前接入支付平臺有幾十條業務線，主要包括到家業務、騎士業務、快送業務等，根據不同的業務形態，我們提供了5種主要的支付產品："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"收銀臺（具備多種支付能力）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直連支付（直接喚醒支付場景）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"代扣服務（週期性扣款）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"代付（好友代付等）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"協議支付（便快捷支付、免密支付等）"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支付系統核心功能主要是：支付和退款。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如在一些預售場景下，爲業務線提供定金支付這種2階段支付能力（如定金支付）；退款主要是全款、部分退款、以及提供人工退款服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了核心功能，我們還提供支付"},{"type":"text","marks":[{"type":"strong"}],"text":"營銷能力"},{"type":"text","text":"，進一步提升支付的轉化，如根據用戶所屬區域進行支付引流，支付券產品、免息產品、滿減等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在開發定金、和營銷產品之前，支付系統被設計爲一個訂單，在產品形態上很單一，僅認爲訂單爲固定金額，因此開發這兩個需求的時候改動了比較大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前主流支付能力大概有多階段支付、大額支付場景、組合支付、支付參與營銷等能力，所以有開發支付系統需求的同學，在最開始最好提前預留一些設計。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們目前基本接入了所有主流支付方式（微信支付、支付寶、京東支付、京東白條等），尤其是現在各類小程序當道，各業務線都需要支持在不同小程序中發展業務，但受到平臺支付方式的限制，就需要我們接入更多的支付方式，例如百度支付、頭條支付等，都需要接入。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當業務被嵌入到各種各樣的流量渠道入口，我們也要根據不同的渠道支持不同的支付方式，如App支付、H5支付、刷臉支付等。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、支付能力的快速接入"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們接下來看一下支付能力的快速接入的一個流程："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7c\/7c292e8953dff6c61474a0e418e69412.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"設計流程的主要目標："},{"type":"text","text":"屏蔽接入第三方支付平臺的複雜度，爲業務提供便捷接入的支付的能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"整體交互邏輯是："},{"type":"text","text":"用戶下單之後，業務線生成生訂單的同時請求支付系統，返回攜帶加密後的收銀臺連接，業務前端渲染收銀臺H5連接，之後的用戶操作都直接與支付系統直接交互，不用再經過業務線。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/4a\/4afeecbb84b8cbee3783276bf0838ed0.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"左上圖是我們經常使用的收銀臺（支付頁面），包括訂單的基本信息、隨機減活動和微信的引流活動等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"右上圖是我們支付平臺和微信的交互邏輯，整體上還是比較複雜的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、配置化支付方式"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖我們目前的個性化配置，目前我們最小力度是支持按照不同渠道進行配置，比如我們最核心的商超業務渠道有幾十個，根據不同的終端適當屏蔽掉一風控能力減弱的支付方式，或者在某些特別的終端按照業務的要求配置指定的支付方式，每種相同的支付方式，根據具體的業務線或具體的業務配置不同的商戶號，不同的商戶號在第三方平臺的費率，收款賬戶都是不同的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2e\/2eae848248b71874d0215096c756c498.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、支付單的生命週期"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是一筆支付單的生命週期："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/32\/32ad49948481831116a1830f224be78e.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的多端支付場景是指："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一： 由於我們無法感知用戶喚醒sdk後的操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以我們不能限制用戶的支付行爲，一個訂單可以從多個手機多個渠道使用相同或不同的支付方式、不同的人同時對一筆進行支付。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二：支付、退款跨越多個端。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支付跨越多個機構，就會經過支付平臺、第三支付平臺、銀行等。支付和退款是一個天然異步場景，這是不可抵抗的因素。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"那我們如何做到多端支付支付金額的強一致呢？"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、如何安全保證金額正確性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ec\/ecab1e977a3f1c675f6a58c466527990.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在我們的臺賬信息中會記錄了 4個金額字段：應收、實收、應退、實退。在收到支付通知的時候，我們會在收到支付通知的時候進行一次對賬。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"判斷：應收+應退=實收+實退"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"保持了這個等式，我們就可以正確的計算每筆次金額的變動。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然我們使用了一個簡單的公式來保障多端併發下金額修改的正確性，但是由於金額的頻繁改動。我們是否可能出現邏輯上的bug呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果給用戶少退了，第一時間就可以得到用戶的反饋，及時修正bug，並補償給用戶；但如果給用戶多退一些錢，很可能用戶不會產生反饋，我們自己也沒發現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以我們的系統底線保障是："},{"type":"text","marks":[{"type":"strong"}],"text":"確保不會產生多退款。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/52\/529207319baf3653450358bf8e54532b.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼我們要通過什麼手段，來保證實際收的錢應該大於或等於應該收的錢？這時，我們要儘可能通過不變量和不需要加工的數據來驗證變量。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不變量：訂單金額；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不需要加工的數據：業務申請的退款。業務一旦申請退款，校驗通過就會插入數據庫。這裏是業務產生的退款，退款可能會有人工退款、多支付退款等，但是這些我們不用關心。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們需要關心的是："},{"type":"text","marks":[{"type":"strong"}],"text":"訂單金額-業務產生的退款，"},{"type":"text","text":"也就是我們至少收到多少錢，如果和實收相等，則認爲沒有問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"那麼我們如何保證實收就是正確的呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"繼續使用實收和支付平臺對賬，其實就可以進一步確保實收沒問題，我們需要對賬每一筆正向支付交易和逆向支付交易產生的金額記錄，且對賬至少需要2種機制來相互保障。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、高可用架構的一些實踐及思路"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、高可用的分類"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5c\/5cc93c26641b3c185327c384c4b0ba4d.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高可用分爲三部分："},{"type":"text","marks":[{"type":"strong"}],"text":"業務上的高可用、容災上的高可用、架構上的高可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支付是一個不可抗拒的跨端異步場景，還要抵抗網絡帶來的不確定因素。對於一筆銀行轉賬，大家在心裏是有預期的，即使實時轉賬，大家也會等待一段時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是對於在線支付，用戶支付完以後，用戶很理所當然的想到應該看到訂單是已支付狀態，而不是待支付狀態，延遲增加到一定時長，客服就找到研發頭上。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、高可用-支持實時性保障"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/33\/3390e23859acede61cc40e9195446f10.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"線上場景還有以下特徵："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"配送的時效性。"},{"type":"text","text":"一筆訂單生命週期就不到1小時，所以在支付上我們不能延遲，不能像銀行轉賬這麼慢。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"高併發。"},{"type":"text","text":"大家也知道銀行等金融機構比較有錢，所以他們在做活動的時候，這些活動的力度不亞於一些商品秒殺場景。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"營銷限量。"},{"type":"text","text":"這是非常重要的一點，如果用戶享受支付優惠，但是最終由於支付通知的延遲、服務器負載較高的情況下未能成功處理支付通知，那麼用戶就會要求索賠營銷優惠了。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"線下場景比線上場景更加複雜，有以下特徵："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"即時性。"},{"type":"text","text":"大家去的線下商店，基本都支持收銀臺在線支付，所以在排隊的情況下，就需要商家及時完成顧客的支付請求。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"網絡環境不可控。"},{"type":"text","text":"在不同的位置，其網絡信號就存在着不確定性。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"羣體性。"},{"type":"text","text":"線上主要是平臺和支付平臺的網絡交互，但是線下還涉及到了商家，整個支付環節也沒這麼流暢。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們需要做的就是儘可能快的讓訂單支付完成，或者在某種支付有問題的時候，第一時間下線這種支付方式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們過去的做法是通過暴力從數據庫反查待支付的訂單，但是對數據庫壓力還是比較大的，還要單獨寫個任務表，後來改寫爲基於事件通知機制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們使用JMQ隊列作爲事件管道，但由於不同場景觸發的反向查詢時機不同，所以我們不能對所有對待支付訂單進行無差別對待，因此就受限於JMQ的特性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前不支持個性化延遲消費消息，所以我們的策略是申請多個隊列，並按照不同的延遲level，放入隊列中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/29\/29881dec5dedc4614adbfd3f052a9f60.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是我們整體查詢補償的設計，"},{"type":"text","marks":[{"type":"strong"}],"text":"反向查詢主要由以下幾種場景："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一種是支付喚醒。"},{"type":"text","text":"由於用戶需要輸入密碼，我們考慮到需要用戶參與，進行多次間隔3秒到重試之後，如果還沒支付結果則放到更大時間間隔到重試leave中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二種是協議支付、代扣類場景。"},{"type":"text","text":"這種支付方式用戶無需輸入密碼，所以我們選擇更低延遲到消費隊列、或無延遲隊列中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第三種是訂單取消"},{"type":"text","text":"（這是大家比較容易忽視的）。由於反向查詢在一定的次數之後會放棄，不然會很佔用資源；但如果一筆訂單取消了，那麼也有可能會因爲支付延遲導致訂單取消，所以我們就會最後查詢一次。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第四種是支付通知"},{"type":"text","text":"（最重要的一種）。支付通知場景我們會同步進行查詢結果，主要是爲了防止僞造通知，但是增加了一次外網交互，超時的可能還是很大的，僞造通知是極端場景。所以在超時之後會暫時信任本次通知，繼續交給反查隊列，繼續對這筆通知進行驗證。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、高可用-應用部署隔離"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是我們高可用部署的一個架構："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0e\/0e6dcc787342d4bdb41a763799d7741e.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們根據ToC和ToB業務請求，對服務器部署上做了資源的傾斜。確保業務相互不影響。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ToC一般是正向交易場景，RT要求比較高；ToB場景對時效基本沒有要求，在某些業務場景下，會存在集中性的大批量退款申請、退款流程的事務也比較大，ToB就針對一些任務worker比較消耗cpu。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的目標是儘量避免非業務耗時導致的RT升高，而導致RT升高的因素有：池化資源不夠（http請求線程、rpc處理線程、數據庫線程、以及http連接等）、cpu資源搶佔、GC 導致的業務線程等待等。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、高可用-多級本地緩存"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2c\/2c9f910d24f415fe52f32256348baf85.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再分享一個最近剛做的業務："},{"type":"text","marks":[{"type":"strong"}],"text":"商品支付的營銷需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個新興業務需求整體的特點是：參與商品不到百萬級的一個量級，但是調用量大，峯值調用量會超過10萬qps、RT苛刻5ms。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但也由於這是剛新起的業務，產生的業務價值收益有不確定性，所以沒打算通過機器去抗量，所以我們把業務請求直接請求到Redis。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是redis需要較多的副本才能扛超高併發，避免大量無效請求。並且增加了內存基本的緩存，使用了布隆過濾器（Bloom Filter），仍然會把cpu打高，通過門店的過濾把cpu降到最低，所以我們最後會通過caffeine來做熱點sku緩存。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7f\/7fa0e510a2e80eff8e453d6e6ed88082.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是我們線上的一個效果圖，Redis的利用率已經達到了97%，這完成是布隆過濾器來決定的，效果還是比較明顯的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5、高可用-監控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們再來舉一個例子："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"某晚上，手機一陣震動，打開報警一看，報警很明顯。紅色框是垂直類-支付渠道層的報警，而且都是apple Pay導致的報警，那麼基本大概率影響支付方式是apple Pay。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綠色框是水平維度監控，顯示了我們的影響功能：支付、查詢。同理報警沒有顯示的業務線，就跟這些具體業務場景沒關係。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6d\/6de94c685ef4dda1dc1a6cde17c7b028.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以，做一個平臺的系統，監控是非常重要的，有句名言："},{"type":"text","marks":[{"type":"strong"}],"text":"“沒有度量就沒有管理”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們跑在線上生產環境中的每一個服務，也需要管理，我們需要管理它們的運行情況，所以就需要我們建立的完整指標反饋監控系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"機器層面的監控包括："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"機器維度：系統指標：cpu、負載、內存；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡指標：tcp連接數、丟包數、tcp重傳；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磁盤指標：磁盤使用率和磁盤繁忙度；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容器指標：關注線程數；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用：軟件異常。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"應用層面的監控包括："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系統異常：基礎組建異常（數據庫、Redis）、RPC異常"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務異常：業務異常的捕獲主要是爲了捕獲業務線的一些非法出入參。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"非預期邏輯：主要一些沒想到的一些邏輯場景。通過自定義監控。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Bug(Jex接入)：上線一陣時間後。大家可以去搜搜excpetion、error關鍵詞。總有意想不到的收穫。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"業務監控維度包括"},{"type":"text","text":"（作爲平臺類系統，最重要的是結合水平維度+垂直維度劃分系統報警情況）："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"水平維度：支付、退款、營銷、通知"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"垂直維度：業務方、渠道平臺"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":">>>>"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：延遲隊列使用什麼框架實現的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A1 ："},{"type":"text","text":"使用京東的JMQ隊列。按照隊列進行消費延遲。RocketMQ支持消息級別的延遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2 : 如何避免單重複支付呢？或者避免重複退款呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A2 ："},{"type":"text","text":"支付是一個不可抗拒的多支付場景。在接收到支付通知的時候，要做對賬，如果是多支付，就進行退款。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3 : 商品價格類型是用float還是decimal？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A3 ："},{"type":"text","text":"支付系統最好使用bigDecimal，因爲和支付平臺交互單位都是元。其他交易系統儘可能使用long類型，分作爲單位。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4 : 各個階段對賬，如果有差錯，怎麼處理？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A4 ："},{"type":"text","text":"程序設計越複雜，bug的可能性就會越多，所以要儘可能通過一些不變對量和不可修改的數據進行底線保障，至少需要2種金額校驗相互保障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5 : 如果應收50，A付款30，B付款40 你這個情況咋退款呢？退款給誰呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A5 ："},{"type":"text","text":"不會出現。支付金額至少是50，即便多支付，也是100 或者150。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q6 : sku的緩存怎麼做的，直播說到是基於Redis的，那數據庫和緩存的同步呢？比如下單後扣減庫存呢？或更新呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A6 ："},{"type":"text","text":"我們前面有提到過，整體是基於redis的。但是redis需要較多的副本才能扛超高併發，避免了大量無效請求。增加了內存基本的緩存，使用了布隆過濾器，但是仍然會把cpu打高，通過門店的過濾把cpu降到最低，最後通過caffeine來做熱點sku緩存。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q7 : 持久化數據庫用的哪個產品？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A7 ："},{"type":"text","text":"MySQL。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鄭志成，"},{"type":"text","text":"京東到家高級工程師，2015年加入京東到家。負責交易系統（提單、支付）以及基礎系統（Api網關、定位、地址）等開發工作,通過深入到業務，搭建合理的業務架構。目前主攻降低軟件複雜性設計、構建高可用系統方向。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/tlL9jJohvXFdfWEsxTSAVQ","title":"xxx","type":null},"content":[{"type":"text","text":"金融IT保命丹：多端支付強一致性架構設計實踐"}]}]}]}