如何處理分佈式系統中的緊急情況？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"本文最初發表於 gitconnected 博客，經原作者 Gregory Pabian 授權，InfoQ 中文站翻譯並分享。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"由於發生意外或遭受攻擊，任何分佈式系統都可能會遇到緊急情況。即便是像執行腳本這樣看似無害的操作，也可能會對整個平臺造成嚴重破壞，最終導致用戶無法使用。本文作者分享了軟件開發者在緊急情況發生之前爲系統做好準備的一些方法。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"免責聲明：我不是軟件安全專家，而是一名全棧軟件開發者，本文中的所有信息都源於我的開發經驗。我希望討論主要的可能性以準備和處理緊急情況，但我不打算就正確和錯誤的做法提出看法。至於如何設計一個安全的分佈式系統，這可能需要用到資深安全工程師的專業知識。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"應急預案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我相信系統設計者可能會考慮準備一個應急預案，至少要知道如何保護一個分佈式系統不受內外威脅。我從各種 OWASP（即 Open Web Application Security Project，開放式 Web 應用程序安全項目，是一個在線社區，在 Web 應用安全領域提供免費的文章、方法、文檔、工具和技術。）資源中學到了很多關於網絡應用安全的知識。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在我看來，應急預案至少應該包括對下列問題的回答："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"誰可以啓動應急預案，以及有哪些權利；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"應急預案實施期間將發生什麼情況，在哪裏發生；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"如何纔能有人啓動應急預案，將會持續多久。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"本文假定系統管理員可以使用 CLI 以系統上的最高權限啓動應急預案，該措施的調用將無限限制多個後臺的某些功能。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"時間框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這個應急預案可在這兩個時間框架下生效："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"過去和未來：系統會根據指定的過去和所有未來事件改變其行爲；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"未來：系統會根據所有未來事件改變其行爲。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當然，並非所有的分佈式系統都支持逆行性行爲變更。在使用長時間運行事務的系統中，遵守這樣的轉換要求執行補償性事務，這樣才能明顯消除原始過去事務的影響。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"解決方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我將所有可能的解決方案分爲三組：無狀態協議（如 HTTP）中阻止訪問的方案、基於消息的系統（如 Kafka）中阻止訪問的方案，以及回滾事務的方案。只要它們在所設計的系統中實際發揮作用，系統架構師就可以選擇應用這些解決方案中的一些，甚至是並行的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"無狀態協議中阻止訪問"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"阻止特定的訪問令牌（例如 JWT）可以說是通過限制特定用戶對資源的訪問來解決緊急情況的最精確的方法。若攻擊者已獲得訪問令牌，並且可能將其用於非法用途，則系統可應用此解決方案。根據暴露程度的不同，此解決方案可以影響到一個或多個後端。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"系統也可以組織訪問特定的訪問方法（例如 HTTP 端點或 RPC）。如果系統管理員不知道特定攻擊者的訪問令牌，或者由於最新的部署，端點無法正常工作，那麼他們可能會選擇此解決方案。在選擇用這種方法解決問題之前，我認爲人們應該考慮一下鎖定整個功能的後果，以及如何與最終用戶溝通這個選擇。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在阻止某些服務中的某些訪問方法無效的情況下，可以考慮關閉該服務。在分佈式系統中，這相當於禁止在相關服務實例中向負載均衡器轉發請求，或者終止所有以上實例。不管選擇哪種解決方案，都會對系統造成嚴重的影響，因爲很多功能已經不能工作了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"最後，系統管理員可以決定關閉整個系統（或者至少所有後端）。關閉分佈式系統的精確算法在很大程度上取決於其基礎設施和使用的冗餘。若有人考慮到這一可能性，則應考慮到後果，因爲除了某些可能在離線模式下工作的前端功能外，系統將會停止工作。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"基於消息的系統中阻止訪問"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"系統管理員可以禁止在基於消息的系統內共享特定的消息。有些情況下，消息代理可以應用此類策略，而有些情況下，這一工作屬於後端。這樣的規則，讀者可以將其視爲判定函數，它接受一個參數、一條消息，然後返回一個布爾值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"另外，人們可以選擇阻止特定的主題或通道分發消息。它會間接地影響依賴這種通信方式的後端。此外，人們也可以禁止某種服務通過某一主題或通道收發消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"最後，有人可能會決定阻止整個消息管道。與關閉整個系統一樣，這種解決方案對系統內的通信產生巨大影響，並可能導致多個功能失效。軟件開發者在出現這種情況之前，應該弄清楚哪些功能可能會崩潰。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"回滾事務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在後端向關係數據庫提交某些事務之前，系統可能會回滾它們。這類規則的應用要求對後端架構進行有效性檢查，並確保後端能夠及時接收策略變更信息。此解決方案可以作爲取消已啓動事務的示例。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於使用依賴注入的後端，一個基於緊急情況的限制列表可以駐留在所有請求或消息處理程序重用的組件中。人們可以使用全局變量實現類似的結果，但是我不贊成這種方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"處理程序可以運行一個檢查函數，以驗證對於當前評估的請求是否存在某些限制。當系統使用緊急編排時，接受緊急通告的後端需要與系統的其他部分通過無狀態請求或消息管道共享信息。若後端無法執行此操作，則將阻止應急響應的傳播。系統設計者應該找到避免這個障礙的方法，例如，讓前一個服務等待下一個服務確認信息處理，然後繼續進行處理。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在本文的開頭，我已經指出，由於不同的系統使用不同的模式進行後端到後端的通信，所以在分佈式系統中處理緊急情況的方法多種多樣。我認爲，嘗試將危機應對措施融入開發中的系統中，或許可以爲系統設計提供許多參考。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Gregory Pabian，全棧軟件開發者，熱愛構建產品。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"原文鏈接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"https:\/\/levelup.gitconnected.com\/emergencies-in-distributed-systems-9f93c6d75600"}]}]}