Uber：模型性能調優

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 使用機器學習模型來支持關鍵的業務決策。一種機器學習模型要經過多次實驗迭代才能進入生產階段。在實驗階段，數據科學家或機器學習工程師將探索添加特性、調優參數以及運行離線分析或回溯測試。在保證生產中高模型質量的同時，我們通過強化平臺來減少這一階段的人力和時間。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"超參數優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/5f\/5f56ae2b7e38a1c88dacd4556df60526.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 1：基於 TURBO 的超參數優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於維數災難，在大型超參數搜索空間中的搜索成本很高，即使對於能夠平衡探索與利用並通過後驗結合信念更新的 貝葉斯優化 技術也是如此。超參數的重要性和選擇對於幫助降低搜索空間的複雜性具有重要意義，因爲超參數對模型性能有很大的影響。結果表明，對於像 XGBoost 這樣的基於樹的模型來說，最大樹深、最大樹數、學習率和最小子重等的調整通常會會產生更好的性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣做的目的是主動顯示已知類別問題的適當參數，從整體上節省時間。除了優化 二值分類問題 的 AUC或 迴歸分析 問題的 均方誤差（MSE）之外，我們還公開了額外的目標函數，以便數據科學家和機器學習工程師可以爲他們的問題選擇合適的優化目標。爲避免 過擬合，我們在優化函數中引入一個懲罰項，來捕捉訓練和測試性能之間的差異。要加速對超大數據集的 超參數搜索，如果我們在新的試驗中沒有發現明顯的改進，則允許超參數優化研究提前停止。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"學習曲線與提前停止"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在訓練期間，通過繪製模型或估計器對評估數據集進行多次迭代，學習曲線 通常被用來評估和診斷機器學習模型。它分析了模型從額外的迭代訓練中獲得的增量性能收益，並就 偏差 - 方差權衡 問題提供了深入的見解。從圖 2 中可以看到，對增量學習過程的知識使用戶能夠對模型的擬合作出明智的判斷，並在每一次實驗的基礎上急切地應用 提前停止。超參數優化設置中，在一個實驗中存在多個同時進行的實驗，提前停止也可以跨實驗聯合應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/9a\/9a27f14e26eafb64951dfb5dd6480825.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 2：使用提前停止節省可能的計算時間"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據每次實驗結果，利用學習曲線可以估算出數百個相互作用達到穩定狀態前的收斂性，在一般的超參數優化設置中，經過大量實驗，每個實驗的效率增益與實驗的數量成比例。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/b1\/b1b0f03dcc4d55b1be37cb8e8c7f1d12.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖 3：屬於同一超參數優化工作的所有試驗的學習曲線。紅色邊界框突出顯示曲線，根據早期表現與其他正在進行的試驗相比，這些曲線不太可能成爲最佳候選者"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"儘管我們可以將提前停止應用於單個實驗，以修剪多餘的迭代（如圖 3 所示），但是通常可以通過在所有同時進行的實驗中共享信息來獲得額外的空間。在一個實驗的所有實驗中聯合使用學習曲線信息，可以讓我們估算該實驗將優於其他實驗的概率。諸如 ASHA（"},{"type":"text","marks":[{"type":"strong"}],"text":"A"},{"type":"text","text":"synchronous "},{"type":"text","marks":[{"type":"strong"}],"text":"S"},{"type":"text","text":"uccessive "},{"type":"text","marks":[{"type":"strong"}],"text":"H"},{"type":"text","text":"alving "},{"type":"text","marks":[{"type":"strong"}],"text":"A"},{"type":"text","text":"lgorithm，異步連續減半算法）等技術，利用由分佈式實驗執行的中央調度器促進的順序貝葉斯優化，可以聯合優化整個研究，使整個超參數調整過程更加迅速。在接下來的博文中，我們將詳細介紹我們的自動超參數優化服務架構，以及在生產中採用黑盒優化服務的結果。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"特徵變換"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 的機器學習模型利用了豐富的地理空間特徵，通常需要用特徵工程來提取有意義的信號。一種方法是將經度和緯度離散化，但是這會產生非常高的基數（cardinality）特徵。將經度和緯度變換爲不同的嵌入空間，然後從嵌入空間中提取出地理空間信號。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於樹的模型執行分段線性函數近似（piecewise linear functional approximation），這種方法無法很好地捕捉複雜的非線性交互作用。在基於樹的模型訓練過程中，我們沒有使用內核技巧，而是自動加入頂級複合特徵，重要性低的複合特徵將被刪除。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"額外策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 的業務數據有很強的時間因素。行程信息被攝入並按照日期字符串劃分到數據湖中。多數 Uber 的機器學習模型都是在日期範圍內訓練的。舉例來說，使用更廣泛的日期範圍使模型能夠以周或月爲週期來捕捉。增加數據的日期範圍是提高模型準確性的常見方法，這也是一個關鍵的最佳實踐。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，我們也看到一些情況，日期範圍的增加會導致模型的準確性降低。出現這種情況的原因之一是，更大的範圍會放大異常值的影響。在這種情況下，我們檢查數據並在 DSL 中進行補償。另外一種可能是由於預測的行爲會隨着時間而改變。因此，對訓練數據進行時間加權衰減可以起到一定的作用。通過使用 XGBoost 列加權特性和 DSL 轉換來完成此任務。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"支持最佳實踐的工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用模型調優的最佳實踐需要進行大量的工作和跟蹤。我們稱這一過程爲“金礦開採”。金礦開採始於克隆客戶的 Michelangelo 項目。這樣可以避免在客戶的生產環境中運行實驗性訓練作業。每個實驗性運行都涉及許多容易出錯的步驟，包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"仔細設置（例如，延長日期範圍意味着改變訓練 \/ 測試拆分，如果它是基於日期的）。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用相關的啓發式方法（例如，設置超參數搜索範圍）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更新計算資源（例如，分區模型需要更多的工作器或並行性，以避免更高的延遲）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"記錄實驗結果並確定當前的最佳模型"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們已經將上述步驟、模型探索和模型分析自動化爲一個工具，以加快金礦開採的過程。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建和維護高性能的模型是一個迭代的過程。Michelangelo 提供了大量的功能目錄，如超參數優化、提前停止、特徵變換等，可在模型開發和調優階段使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Michelangelo 還提供了執行全面重新調優的工具，包括模型架構搜索、確定特徵協同和冗餘性等。最終的結果是，我們已經能夠爲 Uber 部署的機器學習模型提供一個迭代調整和一次性全面調優的框架。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Joseph Wang，Uber 機器學習平臺團隊軟件工程師。負責特徵存儲、實時模型預測服務、模型質量平臺和模型性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Michael Mui，Uber 機器學習平臺團隊高級軟件工程師。負責分佈式訓練基礎設施、超參數優化、模型表示和評估。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Viman Deb，Uber 機器學習平臺團隊高級軟件工程師。負責黑盒優化服務、Uber 定製貝葉斯優化算法以及 Michelangelo 的超參數優化工作流。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Anne Holler，Uber 機器學習平臺團隊機器學習框架 TLM 前僱員。負責機器學習模型的標識和管理，以及訓練和離線服務可靠性、可伸縮性和調優。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文鏈接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/tuning-model-performance\/","title":"","type":null},"content":[{"type":"text","text":"https:\/\/eng.uber.com\/tuning-model-performance\/"}]}]}]}