30萬行代碼的平臺升級：給跑着的汽車換輪胎

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#333333","name":"user"}},{"type":"strong"}],"text":"本文最初發佈於"},{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"Mahmoud Hashemi的個人"},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#333333","name":"user"}},{"type":"strong"}],"text":"博客，經原作者授權由InfoQ中文站翻譯並分享。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"2020年可謂反覆無常。儘管一切都超出了人們的控制，但隨着時間的推移，我發現自己把越來越多的時間地投入到一件感覺唾手可及的事情中：爲我幫助構建的大型企業級Web應用程序"},{"type":"link","attrs":{"href":"https:\/\/simplelegal.com\/","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"SimpleLegal"}]},{"type":"text","marks":[{"type":"italic"}],"text":"設計一個面向未來的解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"現在已經完成了，這次平臺升級很容易就可以在我最複雜的項目中名列前茅，此時此刻，最幸福的結局。幸福是要付出代價的，但是藉助一些恰當的方法，代價可能不會像你想的那麼高。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將"},{"type":"link","attrs":{"href":"https:\/\/simplelegal.com\/","title":null,"type":null},"content":[{"type":"text","text":"SimpleLegal"}]},{"type":"text","text":"的主要產品，一個30萬行的Django-1.11-Python 2.7-Redis-Postgres-10代碼庫，移植到Django 2.2-Python 3.8-Postgres-12技術棧，如期完成，而且沒有發生重大站點事件。這感覺很棒。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲這個項目的技術主管，它看起來是什麼樣子？對我來說，是這樣的："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a7\/a7936f3e65682bf40d198be21b54bc02.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但作爲工程總監，它的成本是多少？"},{"type":"text","marks":[{"type":"strong"}],"text":"3.5年的開發時間，每行代碼只需要2美元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我對這個結果感到特別自豪，因爲在這個過程中，我們也大大提高了網站和開發過程本身的速度和可靠性。現在，該產品有了一個光明的未來，已經準備好在銷售徵求建議書和合規調查問卷上大放異彩了。最重要的是，你不必擔心怎樣委婉地告訴潛在客戶，他們將使用的是不受支持的技術。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡而言之，這是一筆巨大的、穩健的投資，而且已經取得了回報。如果你來這裏只是爲了看看我們自己對這項工作的估計，那就是上面這些了。這篇文章是介紹如何讓你的團隊達到同樣的結果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"故事開始於2013年，剛剛從YC孵化出來的SimpleLegal爲一家新成立的SaaS法律技術公司做了所有正確的決定：Python、Django、Postgres和Redis。在典型的初創公司模式中，在技術不成障礙的情況下，功能是第一位的。軟件包只是順帶升級。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到2019年，這條技術跑道的終點已經臨近。雖然Python 2可能得到了來自不同供應商的擴展支持，但在2021年，Django 1 CVE補丁的志願者已經非常少了。Web框架成了風險較大的攻擊面，所以是時候償還我們的技術債務了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"開端"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，我們在2019年第4季度開始了Tech Refresh平臺升級計劃。其目標是：升級技術棧，同時仍然提供新特性，就像給跑着的汽車換輪胎。我們要小心謹慎，而那需要時間。以下是一些長期項目的基本原則："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"任何每週工作10小時以上的項目都應該每週花30分鐘進行同步。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"每次定期會議都應該有記錄。把它放在邀請函裏。使用項目日誌記錄進度、阻礙因素和決策。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"這是一場馬拉松，不是短跑。要避免在晚上、週末和假期工作。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們從一個計劃草圖開始，經過開放地討論，最終只有一半正確。有一些早期的猜測成功實現："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"轉到"},{"type":"link","attrs":{"href":"https:\/\/github.com\/jazzband\/pip-tools","title":null,"type":null},"content":[{"type":"text","text":"pip-tools"}]},{"type":"text","text":"，並根據廣泛的變更日誌分析解除依賴關係。識別不兼容py23版本的包。（儘管我們已經轉向"},{"type":"link","attrs":{"href":"https:\/\/github.com\/python-poetry\/poetry","title":null,"type":null},"content":[{"type":"text","text":"poetry"}],"marks":[{"type":"color","attrs":{"color":"#41accd","name":"user"}}]},{"type":"text","text":"。）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"在CI中加入行覆蓋率報告。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"改進內部測試框架，讓開發者可以快速編寫測試。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面有更多相關內容。其他的計劃就不那麼現實了："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"在6個月內將CI行覆蓋率從大約60%提升到95%。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"在三個月內並行轉換app程序包。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"利用美國節假日（感恩節、聖誕節、新年）期間的低流量時間，在2021年之前逐步切換到新應用。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們年輕！雖然我們天真，但至少我們知道有很多工作要做。爲了分擔這項工作，我們尋找、僱傭並培訓了三名敬業的海外開發人員。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"導向問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即使新增了開發人員，到2020年中期，我們越來越認識到，95%的覆蓋率就是在做夢，更不用說100%了。全部覆蓋可能是最佳實踐，但3個半開發人員沒法做到這樣的覆蓋範圍。我們做了有價值的測試，甚至發現了以前的Bug，但如果我們堅持這個計劃，Django 2最終將成爲一個2022年的項目。70%，我們決定修改目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們意識到，對於大多數站點來說，CI比大多數用戶更敏感。所以我們專注於測試影響最大的代碼。怎麼纔算影響大？1）失敗了最易被察覺的代碼；2）最難重試的代碼。通過查看流量統計數據、批處理作業計劃和詢問支持人員，你可以在一週內構建出高影響代碼清單。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大約80%的代碼庫都不在這個高流量\/高影響列表中。那80%該怎麼辦呢？利用錯誤檢測和快速修復。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"轉換Sentry的角色"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f7\/f7c2752008eb562abd16aedce60c3018.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創業生活的一個好處是，嘗試新工具很容易。我們在SimpleLegal所採用的一種做法是，把每5個周的最後一週（即20%的時間）留給開發人員，讓他們專注於開發過程本身。即使是最好的廚師也不能在髒亂的廚房裏做出五星級的食物。這是我們改進工作的方法，最終加快了交付速度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這樣一個時期，有人想出了一個天才的主意，使用"},{"type":"link","attrs":{"href":"https:\/\/sentry.io\/","title":null,"type":null},"content":[{"type":"text","text":"Sentry"}]},{"type":"text","text":"將專門的錯誤報告添加到系統中。在一兩天內，我們就有了一個網站，你可以訪問並獲取堆棧跟蹤。這非常神奇，但直到Tech Refresh計劃開始我們才意識到，雖然集成只需要一天的開發時間，但完全採用卻需要團隊幾個月的時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"你看，在一個成熟但快速運轉的系統上增加Sentry意味着一件事：噪音。我們的網站一直在出錯。大多數錯誤是不可見的，也沒有妨礙用戶使用，有些用戶已經悄悄學會了如何處理長期存在的網站怪癖。很快，我們的開發人員就學會了把Sentry當作調試信息的存儲庫。2019年，Sentry事件本身並不值得認真對待。2020年，情況發生了變化，負責將平臺無縫升級的團隊需要把Sentry變成另一種東西：響應性網站質量工具。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們是怎麼做到的呢？第一步，通過"},{"type":"link","attrs":{"href":"https:\/\/docs.sentry.io\/product\/sentry-basics\/guides\/getting-started\/#-how-many-projects-should-i-create","title":null,"type":null},"content":[{"type":"text","text":"以下最佳實踐"}]},{"type":"text","text":"增強流入Sentry的數據："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"將產品拆分成"},{"type":"link","attrs":{"href":"https:\/\/docs.sentry.io\/product\/sentry-basics\/guides\/getting-started\/#-how-many-projects-should-i-create","title":null,"type":null},"content":[{"type":"text","text":"單獨的Sentry項目"}]},{"type":"text","text":"。這包括前端和後端。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"標記版本。不要用分支來標記開發環境部署，這會導致Releases UI混亂。添加一個單獨的分支標籤用於搜索。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"把環境分開。這對於定向報警至關重要。Sentry客戶端環境是通過域約定和Django的"},{"type":"link","attrs":{"href":"https:\/\/docs.djangoproject.com\/en\/3.1\/ref\/contrib\/sites\/","title":null,"type":null},"content":[{"type":"text","text":"sites框架"}]},{"type":"text","text":"來配置的。爲了便於理解，這裏有一個基線，我們使用這些環境："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"生產環境：當前正式版本。DevOps監控。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"沙箱環境：當前正式版本（部分公司會做下一次發佈）。供用戶測試變更使用。DevOps監控。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"演示\/銷售環境：上一個正式版本。主要是內部流量，但在前景演示時外部也可見。DevOps監控。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","text":"金絲雀環境：下一個正式版本。也稱爲過渡環境。內部流量。Dev監控。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":8,"align":null,"origin":null},"content":[{"type":"text","text":"ProdQA環境：當前正式版本。內部用於重現技術支持問題。Dev監控。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":9,"align":null,"origin":null},"content":[{"type":"text","text":"QA環境：Dev分支、dev發佈、內部流量。未監控調試數據。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":10,"align":null,"origin":null},"content":[{"type":"text","text":"本地測試\/CI環境：默認不發佈到Sentry。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當問題最終被正確標記並且可以搜索之後，我們使用Sentry新增的"},{"type":"link","attrs":{"href":"https:\/\/docs.sentry.io\/product\/discover-queries\/","title":null,"type":null},"content":[{"type":"text","text":"Discover工具"}]},{"type":"text","text":"每週導出問題，並對遺留錯誤進行優先級排序。我們首先關注的是對於非內部人類用戶高可見的生產錯誤。具體查詢是："},{"type":"codeinline","content":[{"type":"text","text":"has:user !transaction:\/api\/* event.type:error !user.username:*@simplelegal.*"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將其分爲4類：快速修復（小漏洞）、快速錯誤（將一個含糊的500錯誤轉變成某種形式的可操作的400錯誤）、"},{"type":"link","attrs":{"href":"http:\/\/agiledictionary.com\/209\/spike\/","title":null,"type":null},"content":[{"type":"text","text":"Spike"}]},{"type":"text","text":"（比較大的漏洞，需要研究）和Silence（使用Sentry的忽略功能）。在6周的時間裏，每週事件量由每週超過2500次下降到了不到500次。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過進一步的努力，每週的事件量已經少於100次，並且分散在幾個問題上，對於一個精益團隊來說，這非常容易管理。雖然“Sentry Zero”是最理想的，但我們實現並維持了響應流的真正目標，這在很大程度上要歸功於"},{"type":"link","attrs":{"href":"https:\/\/sentry.io\/integrations\/slack\/","title":null,"type":null},"content":[{"type":"text","text":"Slack集成"}]},{"type":"text","text":"。我們的團隊不再從支持團隊那裏獲取服務器錯誤信息。事實上，現在，當客戶遇到麻煩時，我們會告訴他們，而我們已經有了一個處理中的工單。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"和支持團隊建立緊密的聯繫非常重要。在上面的策略中，我們嵌入了比真實用戶更敏感的CI。雖然完美很誘人，但要求企業用戶有一點耐心也是可以的，前提是支持團隊已經做好了準備。每週都和他們同步，這樣驚喜就少了。如果他們幹勁十足，你也可以教他們一些Sentry基礎知識。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"新徵程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/05\/05783d1d0f878419369cea675b3ad316.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着噪音的消除，我們已準備好快速行動。以下是我們在做出這些改變時積累的一些經驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"訴諸事務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用得當，回滾可以使錯誤看起來像從未發生過，這是快速修復策略的完美補充。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"真正的原子請求"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"把操作儘可能地放入事務中。打開"},{"type":"link","attrs":{"href":"https:\/\/docs.djangoproject.com\/en\/3.1\/topics\/db\/transactions\/#tying-transactions-to-http-requests","title":null,"type":null},"content":[{"type":"text","text":"ATOMIC_REQUESTS"}]},{"type":"text","text":"（如果沒打開的話）。但是，有些請求所做的不僅僅是更改數據庫，比如它們會發送通知，將後臺任務入隊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在SimpleLegal，我們重新設計了架構，將所有副作用（除了日誌記錄）推遲到成功返回響應時。中間件可以提供幫助，但我們主要是通過將Redis隊列切換到基於PostgreSQL的任務隊列\/代理來實現的。這種配置可以確保，如果發生錯誤，事務將被回滾，任務不會進入隊列，用戶將得到一個乾淨的失敗。我們在Sentry中定位故障，切換到舊站點進行消除，他們下一次重試就會成功。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"事務性測試設置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事實證明，事務性對我們的測試策略來說也很關鍵。SimpleLegal早已超過了Django原始的fixture系統。大多數測試都需要複雜的Python設置，這使得編寫測試和運行測試都很慢。爲了加快編寫和運行的速度，我們將整個測試會話封裝到一個事務中，然後，在運行任何測試用例之前，我們設置了示例性的基本狀態。測試用例使用這些基本狀態作爲"},{"type":"link","attrs":{"href":"https:\/\/docs.pytest.org\/en\/stable\/fixture.html","title":null,"type":null},"content":[{"type":"text","text":"fixture"}]},{"type":"text","text":"，並在每個測試用例之後回滾到基本狀態。詳情請參閱"},{"type":"link","attrs":{"href":"https:\/\/gist.github.com\/mahmoud\/10f6b6b0a9c5860030693357124131df","title":null,"type":null},"content":[{"type":"text","text":"contest.py摘錄"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"有些最佳實踐並不適合你"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"軟件場景的差別如此之大，知道哪些建議不適合你是一門藝術。以下是我們親身瞭解到的各種死衚衕。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"命名空間的運用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到代碼被劃分成模塊、包、Django應用等的方式，把它們作爲工作單元可能很有誘惑力。開始時不要這樣。代碼劃分可能非常隨意，很難知道你何時就進入了一個有風險的思路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假如有自動重構，就像在"},{"type":"link","attrs":{"href":"https:\/\/portingguide.readthedocs.io\/en\/latest\/","title":null,"type":null},"content":[{"type":"text","text":"2to3轉換"}]},{"type":"text","text":"中一樣，首先要按轉換類型進行移植。這樣，你只需要查看一個命令和受影響的路徑列表。另外，自動修復必須遵循一種模式，這意味着更多的人可以修復重構導致的錯誤。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"蓋"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"覆蓋率工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/57\/57cc3e421391def8f4e55d38a17bdba3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"覆蓋率對我們來說是好壞參半。顯然，覆蓋率優先策略是站不住腳的，但對優先級劃分和狀態檢查，它仍然有用。就單次變更來說，我們發現覆蓋率工具有些不可靠。我們從來沒有弄清楚爲什麼覆蓋率的作用有不確定性，我們得出了這樣的結論:“像codecov這樣的現成工具可能並不是針對我們這種規模的monorepos。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在撞上覆蓋率牆的過程中，我們研究了其他許多關於覆蓋率的解釋。對我們來說，“路由覆蓋”（即每個URL至少有一個集成測試）和“模型表示覆蓋”（即每個模型對象都有一個有用的文本表示，可以用於Sentry調試）比行覆蓋優先級高得多。如果有更多的時間，我們會希望圍繞這些構建工具，甚至是圍繞基於在線分析的覆蓋率統計，從而優先考慮流量最高的路由，而不僅僅是流量最高的代碼行。如果你聽說過這些方法，我們很想和你討論一下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"扁平化數據庫遷移"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從表面上看，減少需要升級的文件數量似乎是合理的。事實證明，扁平化"},{"type":"link","attrs":{"href":"https:\/\/docs.djangoproject.com\/en\/3.1\/topics\/migrations\/","title":null,"type":null},"content":[{"type":"text","text":"遷移"}]},{"type":"text","text":"是一種消除文件的低收益策略。更改歷史遷移文件結構會使上線過程變得複雜，而升級沒有扁平化的遷移文件則很簡單。更不用說，如果只是想要加速CI，你可以像我們在"},{"type":"link","attrs":{"href":"https:\/\/openedx.atlassian.net\/wiki\/spaces\/AC\/pages\/23003228\/Everything+About+Database+Migrations#EverythingAboutDatabaseMigrations-SquashingMigrations","title":null,"type":null},"content":[{"type":"text","text":"Open edX平臺"}]},{"type":"text","text":"上所做的那樣："},{"type":"link","attrs":{"href":"https:\/\/github.com\/edx\/edx-platform\/blob\/66f0f9891f00994f77604a51dbb29736aa605fa8\/scripts\/reset-test-db.sh#L75","title":null,"type":null},"content":[{"type":"text","text":"建立一個基本的DB緩存，每隔幾個月檢查一次"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事實證明，"},{"type":"link","attrs":{"href":"https:\/\/sedimental.org\/awesome_python_applications.html#goal-1-a-better-development-cycle","title":null,"type":null},"content":[{"type":"text","text":"你可以從開源應用程序中學到很多東西"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"慢慢適應新技術棧"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你有多個應用程序，請使用相對比較小也比較簡單的應用程序來試驗更改。幸運的是，我們有一個獨立的應用，它的測試運行速度更快，這讓我們能夠更緊湊地瞭解開發循環。同樣地，如果你有多個生產環境，則從影響最小的一個環境開始推出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"把CI作業複製到新的技術棧中，它們都會失敗，但要剋制住把它們標記爲可選項的衝動。相反，構建一個包含所有測試及其當前測試狀態的單文件清單。我們爲測試運行程序"},{"type":"link","attrs":{"href":"https:\/\/docs.pytest.org\/en\/stable\/","title":null,"type":null},"content":[{"type":"text","text":"pytest"}]},{"type":"text","text":"構建了一個小擴展，它基於狀態清單文件批量跳過測試。然後，ratchet：取消並修復測試，更新文件，檢查測試是否通過，然後重複。這比遍佈代碼庫的"},{"type":"link","attrs":{"href":"https:\/\/docs.pytest.org\/en\/latest\/skipping.html#skipping-test-functions","title":null,"type":null},"content":[{"type":"text","text":"pytest標記"}]},{"type":"text","text":"裝飾器更方便和可掃描。詳情請參閱"},{"type":"link","attrs":{"href":"https:\/\/gist.github.com\/mahmoud\/10f6b6b0a9c5860030693357124131df","title":null,"type":null},"content":[{"type":"text","text":"contest .py摘錄"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"上線試運行"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在2020年第四季度，我們增加了基礎設施，以在相同的數據庫支持下並行運行新舊站點。我們進入了這樣一個循環，使流量到達新技術棧，構建一個需要修復的Sentry問題隊列，然後關閉它，並跟蹤時間。使用新技術棧大約120個小時後，經過晝夜不停地策略性擴展，組織已經建立起足夠的信心，我們可以在最關鍵的時間讓站點繼續運行：在月初的週一和週二。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"唯一的問題是"},{"type":"link","attrs":{"href":"https:\/\/www.zdnet.com\/article\/aws-outage-impacts-thousands-of-online-services\/","title":null,"type":null},"content":[{"type":"text","text":"AWS在感恩節周的宕機"}]},{"type":"text","text":"。此時我們已經提前完成了計劃，並且對快速修復工作流建立起了足夠的信心，不再需要最初的假日測試窗口。爲此，我們感謝了很多人。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們一直用快速修復的方法，直到我們完成。“完成”不是指新系統沒有錯誤，而是指流量在新系統上時事件比舊系統少。然後，繼續修復，並開始安排時間刪除腳手架。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7b\/7b1ef7b0ebac8d8b34b99a8bdb4b6ce7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"後記"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以，一旦你使用了Django、Python、Linux和Postgres當前的LTS版本，任務就完成了，對吧？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"謝天謝地，技術債務從不會到0。雖然按期更新並更換核心技術不是一件小事，但用閃亮的部件替換生鏽的部件並不會改變設計。架構技術債務——抽象中的錯誤，包括缺乏抽象——可能會帶來更大的挑戰。這些問題的解決方案並不能在項目之間完全推廣，但它們確實會受益於這個最新的、無錯誤的基礎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於所有希望更換輪胎的項目，我們希望這次回顧能夠幫助你在未來幾年充滿信心地、務實地改進技術棧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看英文原文："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/sedimental.org\/tech_refresh.html","title":null,"type":null},"content":[{"type":"text","text":"Changing the Tires on a Moving Codebase"}]}]}]}