富途證券因數據中心故障導致交易中斷，創始人發文道歉

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"10月9日凌晨，互聯網券商富途出現故障，用戶無法登錄App進行交易，甚至出現了資產清零的狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"9日下午，富途證券發佈相關說明並致歉。富途表示，事故原因爲“運營商機房電力閃斷導致的多機房網絡故障”，公司已於第一時間聯繫運營商進行修復，並在2小時內陸續恢復核心服務。後續富途表示將進一步提升多運營商多機房的網絡容災能力，提升核心服務的穩定性，避免類似問題的再次發生。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"富途證券一向以“技術”爲豪，和傳統券商不同，富途沒有線下營業部，用戶只通過線上開戶進行交易。但全部業務依託線上通道完成，就意味着富途對技術的依賴比傳統券商高。其創始人兼董事長李華是騰訊第18號員工，同時也是QQ產品重要參與者和騰訊視頻創始人之一，領導過騰訊視頻產品設計和開發。在李華看來，富途之所以能彎道超車，花3年到達傳統券商10多年積累才能達到的高度，技術是最重要的一環。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"10月11日中午，技術出身的富途創始人李華再發2000字長文道歉和回覆，引起市場關注。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有關係統容災的問題，李華表示，富途的系統是有做容災設計的，從行情到交易，從服務器到交易網關到網絡傳輸都有做雙路或多路的冗餘設計。不同的子系統設計會有所不同。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以行情爲例，單向傳輸爲主、對時延的敏感度也不是那麼高，富途很早就作了多區域多IDC的容災設計；尤其像美股行情，涉及到越洋傳輸，爲避免中斷，富途選擇了全球頂級的兩家行情供應商分別提供行情源，分別從美國、香港多地多點接入，當這些都不可用時，富途還保留了富途美國IDC直傳的能力。不考慮其他的冗餘設計，光是因爲行情源的冗餘，富途一年增加的成本過千萬港元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"李華指出，在實時熱備的多路冗餘交易系統的設計上會面臨着兩種選擇。一是較差的交易性能更大的訂單延時但更好容災能力的跨IDC多路冗餘方案，二是更好的交易性能較小的訂單提交延時單一IDC的多路冗餘方案，但IDC本身會成爲故障的單點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這也間接導致了一定要做出選擇。在李華看來，考慮到IDC的建設標準，IDC的大級別事故是罕見的，尤其是在電力故障方面。經過綜合推演之後，富途選擇了更好性能的方案二，也因此留下了IDC的單點故障隱患。這次事故恰恰就是IDC出了問題，而且是最不應該出現問題的電力系統出了問題，不間斷電源和柴油發電機都沒能發揮應有的作用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"附：李華迴應全文"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"10月9日凌晨1點26分，事故發生後不少客戶at我，有批評、有建議、有鼓勵，由於9號早晨還要去出差，會有幾個小時在飛機上，就沒來得及一一回復。不論如何都要謝謝你們，因爲你們，我才覺得富途所作的事情格外有意義，我們可以去努力和改善的地方還有非常多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"首先我要向大家鄭重及誠懇地道歉：真的很對不起，讓你們失望了，我們虛心接受所有的批評和建議，並會立即着手相應的改進。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"雖然幾次影響大、耗時長的事故都與不可控的外部依賴有着直接的關係，但給到客戶的感受都是一樣的，那就是富途的服務不可用了。因此，我們責無旁貸，也不會把應該我們解決的問題推到外部，只會看在我們可控的範圍內如何可以做得更好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"1、末日期權價值歸零的補償問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有購買了末日期權因故障未能及時平倉導致價值歸零的客戶在問是否會補償，從週末開始針對這類客戶我們的客服已經在逐一聯繫，會根據具體的情況溝通對應的補償方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"2、有關係統容災的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"首先可以確定的是，富途的系統是有作容災設計的，從行情到交易，從服務器到交易網關到網絡傳輸都有作雙路或多路的冗餘設計。不同的子系統的設計會有所不同。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"富途的客戶中不乏技術達人，這次事故後，不少有技術背景的客戶針對系統的容災給了各種的建議，尤其是有關多區域多IDC的容災建議。在此表示感謝！"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以行情爲例，單向傳輸爲主、對時延的敏感度也不是那麼高，我們很早就作了多區域多IDC的容災設計；尤其像美股行情，涉及到越洋傳輸，爲避免中斷，我們選擇了全球頂級的兩家行情供應商爲我們分別提供行情源，分別從美國、香港多地多點接入，當這些都不可用時，我們還保留了富途美國IDC直傳的能力。不考慮其他的冗餘設計，光是因爲行情源的冗餘，一年增加的成本過千萬港元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"交易系統比較特殊，對時延有着非常高的要求。所有的多路冗餘熱備系統都存在時延大小和數據一致性的衝突；物理位置越分散，比如跨IDC、跨區域，爲確保數據一致性，時延就會越大。跨IDC、跨區域的數據一致性的時延問題好解決嗎？不好解決。因爲在我們生活的這個宇宙，光速或電子流動的速度每秒鐘30萬公里就已經是極限，這也意味着在理論狀態下物理位置每間隔300公里的網絡傳輸就會存在1毫秒的時延；這還僅僅是在理論狀態下，實際上如果加上光電轉換、加上路由跳轉，300公里的傳輸能作到不超過5毫秒已是優秀。那麼5毫秒算長嗎？5毫秒從一般的認知來看也不算長，但對於數據一致性來說就算很長了，光一個網絡傳輸一來一回就是10毫秒；就具體的交易場景來說，你可能啥也沒作，只是因爲熱備的數據一致性的需要，網絡傳輸延時加上各類寫操作的延時和同步確認的延時，你的訂單還沒有提交出去可能就消耗掉了幾十毫秒，且這還只是理論值，實際上有機會更多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"基於上邊的描述，在實時熱備的多路冗餘交易系統的設計上會面臨着兩種選擇。一是較差的交易性能更大的訂單延時但更好容災能力的跨IDC多路冗餘方案，二是更好的交易性能較小的訂單提交延時單一IDC的多路冗餘方案，但IDC本身會成爲故障的單點。有客戶問，你們是不是不願意投入啊，纔會有這個事故發生；看到這裏，你會發現其實不是投入的問題，是選擇的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"考慮到IDC的建設標準，IDC的大級別事故是罕見的，尤其是在電力故障方面。經過綜合推演之後，我們選擇了更好性能的方案二作爲我們的系統設計，也因此留下了IDC的單點故障隱患。這次事故恰恰就是IDC出了問題，而且是最不應該出現問題的電力系統出了問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"供電網絡一個幾秒鐘的電壓抖動，IDC一堆網絡IT設備跟着關機或重啓，實在是難以想象，說好的不間斷電源和柴油發電機去哪了？不間斷電源和柴油發電機竟然都沒能發揮應有的作用，要知道電力保障是一個IDC之所以是IDC的最基礎能力。另一方面也暴露了我們的系統在這種情況下的脆弱。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這次事故的恢復時間以小時計，給我們的教訓和啓發都非常大。兩害相權取其輕，相對於小時級的故障時間，假如我們可以接受一個分鐘級的故障時間，那麼在方案二的基礎上是不是可以有一個兼顧交易性能低訂單延時又支持跨IDC的準熱備方案呢？答案是有的，接下來我們就會對這裏作進一步的研究和推進。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"3、有關資產顯示的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這次事故讓我看到了我們在產品設計上的一些欠周到。事故發生時，有客戶發現自己的持倉和資產數據都沒了，這讓人感到非常驚悚，馬上就有人在牛牛圈上問“富途是不是捲款跑路了？”。實際情況是因爲故障導致了牛牛app跟後臺數據的斷開；既然只是斷開，那前端app的表現爲何是作清空的處理？顯然以最後可以正常顯示的數據快照繼續展示會是更好的實現方案；雖然數據不會作實時更新了，但給到人的心理感覺會安定很多。在這裏我要給受到驚嚇的客戶們道歉，接下來也會在app相關的表現上作出改進。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以上是我想要重點回復的三個問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這次事故值得我和團隊們總結和反思地方非常多，教訓和警示也都非常深刻。我們不會去做無意義的辯解，立足當下作好改進會來得更重要；我們也從未因爲富途有了一點體量和成績就感到自大和自滿，只期望通過我們的持續努力不辜負您們的信任。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"再次道歉並感謝！"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]}]}