Qunar 數據庫遷移程序

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1. 概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文主要介紹 Qunar DBA 部門運維平臺中的自動化遷移程序。自動化遷移程序，主要實現 DBA 日常工作中常見的運維操作，如：過保機器替換，集羣架構升級，數據庫版本升級等，並且支持流程可配置，可以滿足各種定製化的運維場景。使用程序代替人爲操作，實現操作的規範化，提升工作效率的同時，能夠避免各種由於人爲操作不當引發的故障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2. 背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"DBA 工作繁重，日常操作繁瑣"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲 DBA，真的很忙，每天都有很多事情要處理，備份和恢復、監控狀態、集羣搭建與擴容、數據遷移高可用和性能優化等，除此之外，還要面對開發的各種需求和疑問，有時候一個需求，一個問題甩過來，可能半天時間就沒了，有時候還得扮演下救火隊員的角色。因此，急需自動化程序來解放 DBA 的雙手，讓我們有精力去做一些更有意義的事情。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"新人入職培訓成本高，週期長"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新人入職，需要進行較長時間培訓，瞭解公司架構、操作規範等，至少一個月時間才能夠進行線上基本操作，一些複雜操作甚至需要更長時間的學習和理解，纔有信心對生產環境進行變更。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"遷移操作流程複雜，人爲操作具有不確定性，易引發故障"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人是感性的動物，儘管我們制定了各種操作規範，當我們去按照文檔操作時，特別是在個人狀態不佳的時候，難免會出現紕漏。有時因爲忘記改某個參數，忘記執行某條命令，可能就會引發線上故障。Qunar2019 由於人爲操作導致的故障損失較大，而這些故障使用自動化程序都可以避免。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3. 自動化遷移實現"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動化，目的在於使用程序替代人爲操作，因此我們最早先做了集羣的安裝部署，是用的最多的場景。部署不是單單幾條命令就完事了，公司內部架構比較複雜，涉及到的組件多，包括：集羣搭建（主從半同步、PXC 集羣），哨兵集羣，配置中心，zookeeper 等；所以流程比較複雜。我們按照規範文檔，一個個步驟的實現，然後串聯起來，形成一個完整的部署流程。每次部署通過平臺添加任務接口，放到後臺異步執行即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動化部署開發完成後，新集羣部署的問題解決了，按照這個思路，其他日常的操作同樣也能實現，於是開始構思實現更多的場景。但是，在開發過程中，我們遇到了幾個問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據遷移週期較長，需要開發介入，比如：重新發布程序，修改客戶端連接方式等"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"涉及到高危操作，主從切換、割接等，需要擇機進行"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這就導致了不能像部署一樣，把操作步驟組合起來丟到後臺執行就可以了。需要整個流程能夠支持隨時暫停和隨時調度，我們把一個複雜流程拆分爲多個獨立的子任務，每個子任務能夠單獨調度執行，通過後臺拼裝子任務來定義各種遷移流程。按照這種方式，先後實現了幾種在 Qunar 常用的遷移場景：過保機器替換、MMM 集羣數據庫版本升級、MMM 升級 PXC 集羣等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實現了一些常用的場景，但日常的遷移需求遠遠不止於此，稍微一變，後臺定義的模板就無法滿足需求，就得重新開發一個模板，非常不友好。所以爲了滿足更多的需求，我們實現了在 Web 端自定義遷移流程，每個操作人可以自行組合遷移子任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 程序整體架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d8\/74\/d834a808160a31972e525c53d020f274.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3.1.1 任務池"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"獨立的任務，可以理解爲某一個運維操作，每個任務可以是一條簡單命令，例如啓動數據庫、關閉數據庫，也可能是一系列操作。但步驟拆分的越細，複用程度就會越高，可應用在更多的場景。使用 SaltStack 實現對服務器的遠程操作"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3.1.2 任務模板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"後臺定義模板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"早期由各個子任務定製組合成的常用遷移流程，每次操作僅需要配置遷移參數即可，整個耗時過程無需值守。目前已實現了 Qunar DBA 常見的運維場景；比如：集羣版本升級、架構升級、過保替換、部署、卸載等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Qunar 目前使用情況如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MMM 版本升級：Qunar 內部有不少 MySQL5.5 的 MMM 老集羣，均已完成升級，每套節省時間約1小時"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集羣部署：人工需要30分鐘，自動化程序約3分鐘，目前已使用自動化部署集羣超過180套"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"過保機器替換：4年內過保機器超過400臺，使用自動化程序遷移過程無需人爲值守，節省超過90%的時間"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單實例備份遷移：主要適用集羣的資源整合，機器之間數據庫實例遷移、配置中心上下線操作等一鍵操作"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"卸載集羣：由於集羣組件多，涉及操作步驟複雜，需要操作 zk、哨兵、git 等，自動化可一鍵完成"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MMM 升級 PXC 架構：目前 Qunar DBA 正在全公司範圍內推行數據庫架構升級，即將廢棄老的架構 MMM，用 PXC 集羣替換。第二季度已完成內部超過60%的 MMM 集羣架構升級，共計80套，升級過程與業務解藕，支持服務滾動發佈，無需割接、停服，僅涉及一次讀寫切換，對業務影響降低到秒級；純升級時間由3小時變爲20分鐘，並且實現零故障升級"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"用戶自定義模板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲支持更多遷移場景，支持通過 Web 頁面自行配置。與固定模板不同之處在於參數之間的傳遞，固定模板因爲由後臺代碼寫死，預先知道每一個步驟需要哪些參數，通過程序傳入即可。而自定義流程不能預先知道每一步之間參數如何傳遞，因此我們增加了一個參數池，用來存儲所有輸入以及輸出的參數，後續任務只要定義了輸入參數的名稱，在參數池中都可以找到對應參數，找不到的情況，則說明任務流程配置不合理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開發一個新的流程的成本非常低，流程大致如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"確認遷移過程的大致操作及所需要的子任務（如果任務池沒有所需要的，則需要按照樣本實現新的子任務功能） 例：QMHA 新增一個讀節點，所需流程如下 機器初始化實例——> 備份並還原源端數據 ——> 創建主從複製 ——> 檢查主從複製延時 ——> 配置中心添加並上線新節點"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"平臺編輯任務流程"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/e0\/65\/e0f01531924a5f021e142b642a34ff65.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null}}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"確認任務是否高危操作，是否需要人爲調度"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"參數填寫"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0b\/a6\/0b48e48c4323dbe868d893a7a39427a6.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null}}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"提交任務，或保存爲模板，可供下次使用"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3.1.3 遷移任務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"遷移任務由 web 頁面配置管理，任務詳情中，包括：日誌、快照(參數)、流程、管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"管理頁面，支持隨時暫停\/手動調度\/標記任務已執行等"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/30\/a7\/30ab1ddb8bebbc7112703a6c817f22a7.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流程頁面，記錄每一步執行時間以及狀態"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/61\/da\/619c457af967ecf5db14291bdc6fbcda.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"快照，記錄遷移參數"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7a\/7b\/7a3d4695cd81853e7b5fe9ae3f36ab7b.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日誌，實時打印執行信息"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/04\/53\/04aa051fa84e964e4be71cd34ecd0b53.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3.1.4 定時任務\/手動調度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定時任務會每10秒查詢一次任務表，取出滿足執行條件的任務放入執行隊列中，等待任務被調度執行；手動調度是由操作人在頁面點擊按鈕，調用接口進行任務調度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3.1.5 執行隊列"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用 celery 任務隊列，支持多線程，多個流程併發執行；只需將任務放入隊列，程序自行調度，執行結果會反饋到具體執行的任務步驟中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/b8\/5b\/b86ac01b8a300eb053e84aaf6b85105b.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 使用工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開發工具：Python3.6 + Tornado + Celery 遠程命令調用：SaltStack"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4. 總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動化遷移程序，將冗長而複雜的遷移流程，通過拆分組合的方式，使用程序實現，對於 DBA 來說，僅需要配置任務的時間，無需登錄機器進行復雜的操作，大大提升了運維效率，並且在很大程度上避免了人爲故障的發生，真正實現了線上環境操作的規範化和自動化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Qunar DBA 部門正在推進架構升級，即將廢棄老的 MMM 架構，使用 PXC 代替。目前已經使用自動化程序完成超過60%的 MMM 集羣架構升級。PXC 集羣採用 namespace 的方式，對客戶端屏蔽了數據庫實例真實 ip 地址，這對後續數據庫資源池管理提供了可行性，而自動化遷移程序的實現，也爲後續資源池管理，實例遷移整合提供了基礎工具。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"頭圖"},{"type":"text","text":"：Unsplash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者"},{"type":"text","text":"：雷孝龍 - 去哪兒網DBA"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文"},{"type":"text","text":"："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/P_7iZKKbOoKfdc8XyPfGVw","title":"","type":null},"content":[{"type":"text","text":"Qunar 數據庫遷移程序"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"來源"},{"type":"text","text":"：Qunar技術沙龍 - 微信公衆號 [ID：QunarTL]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"轉載"},{"type":"text","text":"：著作權歸作者所有。商業轉載請聯繫作者獲得授權，非商業轉載請註明出處。"}]}]}