如何將數億MySQL數據無縫遷移到MongoDB？

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在好大夫在線內部，S3系統負責各業務方操作日誌的集中存儲、查詢和管理。目前，該系統日均查詢量數千萬次，插入量數十萬次。隨着日誌量的不斷累積，主表已經達到數十億，單表佔用磁盤空間400G+。S3是業務早期就存在的系統，當時爲了簡單快速落地，使用了Mysql來存儲，隨着業務的不斷增長，同時也要兼顧性能和可擴展性，到了必須要重新選型的時候了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新項目命名爲：LogStore。"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"目標"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、安全性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"S3系統在設計之初，沒有按業務系統考慮數據隔離，而是直接採用 key(系統 + 類名 + id) + 有限固定字段 + 序列化value 的方式進行存儲，這種方式顯然不便於後續集羣拆分和管理。LogStore系統要在邏輯上進行數據區域劃分，業務方在接入時要指定app進行必要的權限驗證，以區分不同業務數據，進而再進行插入和查詢操作。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、通用性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"S3主要提供一種3層結構，採用MySQL固定字段進行存儲，這就不可避免的會造成字段空間的浪費。LogStore系統需要提供一種通用的日誌存儲格式，由業務方自行規定字段含義，並且保留一定程度的可查詢維度。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、高性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"S3系統的QPS在300+，單條數據最大1KB左右。LogStore系統要支持當前QPS 10倍以上的寫入和讀取速度。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、可審計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要滿足內部安全審計的要求，LogStore系統不提供對數據的更新，只允許數據的插入和查詢。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5、易擴展"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"LogStore系統以及底層存儲要滿足可擴展特性，可以在線擴容，滿足公司未來5年甚至更長時間的日誌存儲需求，並且要最大化節省磁盤空間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"方案選型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了達成改造目標，本次調研了四種存儲改造方案，各種方案對比如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/resource\/image\/18\/c0\/1825bc742e3f0174b0274ff1ee8ccbc0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、我們不合適—分庫分表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}