一個人如何遷移40TB SQL服務器數據庫？

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的HAProxy1日誌又叫流量日誌，當前存儲在兩臺SQL Server上，一臺在紐約，一臺在科羅拉多州。在2019年初，我們有約4.5年的數據，總計約38TB。最初，數據庫設計爲每天只存一個表。也就是說在2019年初，單個數據庫中大約有1600個表，其中包含多個數據庫文件（因爲數據文件的大小限制爲16TB）。每個表都有一個集羣化的列存儲索引，其中包含100-400百萬行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我必須將數據從現有的每日表中移到一個新結構中——每個月一個表。這需要同時在NY（紐約）和CO（科羅拉多州）服務器上完成，並且數據是存儲在機械硬盤上的，所以遷移會非常緩慢而痛苦。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，遷移工作只能用現有服務器上僅剩的一點磁盤空間來完成。這些服務器有一個44TB的機械硬盤分區，而我們已經用掉了36-38TB不等。所以，我得這麼做："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將數據遷移到新格式"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"刪除舊數據"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"壓縮舊的數據庫文件"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不斷重複"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我以爲只要幾個月就能搞定，沒想到，最後整個項目花了11個月的時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一些要點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先提一些項目要點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我是完全遠程工作的，所以無法從本地計算機運行任何進程。一切任務都需要在一臺永久在線的計算機上執行，因此出現任何VPN故障都不行。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我需要在兩臺單獨的機器上完成工作。我當時是分別在兩個數據中心的SQL Server上進行遷移的，所以需要在兩個地點各準備一臺機器，以避免網絡速度問題。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們在Stack Overflow中有用於各種用途的跳轉盒（jump box），在NY和CO各有一個，非常適合遷移工作。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舊數據庫仍在實時生產環境中運行，這意味着在我移動數據時，每天都會添加一個新表。換句話說，我的遷移對象是動態的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫處於"},{"type":"codeinline","content":[{"type":"text","text":"simple recovery"}]},{"type":"text","text":"狀態，因此我們不必處理事務日誌。我們的備份是原始的源日誌文件，而兩地的SQL Server（NY和CO）是彼此的副本。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}