性能優化：空調能耗節能的強化學習探索之路

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"導讀：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ICT行業一直以來都被公認爲“能耗大戶”，據權威機構預測2025年ICT行業耗電約佔全球電力的20％，貢獻全球碳排放量的3.5％以上，成爲實現“碳中和”目標的主要阻礙和重要影響因素。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從作爲ICT行業主要參與者的運營商領域來看，近些年隨着數字化轉型的加速和對算力需求的增長，數據中心和基站的建設運營成本和能耗持續增加，以國內某省份運營商數據中心機房爲例，省管機房每年空調電費支出高達七八千萬，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"降低10%以上","attrs":{}},{"type":"text","text":"的能耗即可節省千萬級別的電費支出，因此如何通過科技化的手段推動行之有效的節能實踐，成爲需要運營商持續關注的重要目標。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"空調製冷側能耗成爲整體能耗的關注焦點","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過分析基站和數據中心兩個主要的能耗組成來看，空調製冷側能耗佔比尤爲突出。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在基站能耗中，目前空調側能耗消耗最多，佔比56%左右，主設備佔比32%左右。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0d/0d0f165c58f228893ae76bf4cfb19c54.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在機房能耗中，IT佔43%，空調約40%以上，其餘電源、照明佔17%左右。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b50c79426464565545c4b0194a4fe39.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於空調系統結構和理論複雜，目前大多運維人員只是長時間或者季節性的對冷源側設備參數和末端空調溫度進行單參數的管理控制，簡單粗放。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"空調節能手段各異，仍需尋求新技術突破","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近年來，新技術飛速發展爲空調系統的智能管控帶來許多新的思路和方法，致力於通過對空調精細化的管理和控制節省空調側的能耗支出。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如PID控制方式，是利用比例、積分、微分計算出控制量進行控制的，設計簡單、成本低廉，但是空調系統作爲典型的高度非線性、耦合性、時變性、不確定性的複雜多變量系統，這種傳統控制方式很難通過簡單的比例控制取得理想的運行效果，並且在實際應用過程中，每個機房的空調狀況、業務承載和環境換化差異巨大，傳統控制模型也無法快速適配所有機房情況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前比較成熟的方法是通過對利用環境、空調和能耗等歷史數據建立AI預測模型，預測未來空調和環境的能耗值，然後通過貪婪算法等優化求解算法求得在未來能耗值下空調的最優控制參數，這種方式嚴重依賴於模型質量，一旦模型與建築或空調系統真實情況存在偏差，控制策略質量將無法得到保障。即便初始模型效果較好，但仍面臨數據和模型更新的問題，導致模型的魯棒性較差。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此面對複雜的環境變化、專業的空調系統、衆多的控制參數，需要有新的AI技術，能夠對空調系統的監測和運行數據進行分析，找出空調系統自身運行最本質的特性，在保證環境安全的前提下最大程度的挖掘空調節能潛能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"浩鯨基於強化學習的空調能耗節能之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"強化學習是解決策略優化問題的一種重要方法，特別適用於解決高度非線性、不確定性的複雜多變量系統，具有實時、免模型和在線自學習的特點，能夠在缺少控制系統模型的情況下實現控制器的自適應優化，極大降低對先驗知識的依賴。近些年來在自動駕駛、機器人等實際工程優化中逐漸開始廣泛探索應用。強化學習智能體根據周圍環境產生控制動作，動作下發後周圍環境會產生新的狀態和狀態變化後的動作獎勵，從而利用新狀態和獎勵優化智能體的模型，下發更合理的動作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a5/a5fa8cf5a747b9af7a4f51c169e17334.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於空調能耗節能問題，最終目標是通過合理的空調控制優化在保證基站或者機房環境安全前空調的能耗較少，所要關注的也是動作、環境和狀態之前的關係。因此也適用於強化學習的模型思想。我們也在實際應用過程中，基於強化學習不斷探索實踐和突破，走出了浩鯨空調節能的新道路。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/57/57c858c5ff2bcfba18de9aecfc3b359f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"夯實數據之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"強化學習來講，儘管是免模型的自學習方法，即對先驗知識的依賴性較弱，但是仍需要空調和其他相關狀態數據的連續採集去得到動作下發前後的反饋才能進行模型不斷優化，因此對數據時間和空間的維度要有一個清楚的認知才能更好的定義模型的參數和狀態。以數據機房空調節能爲例，在數據層面，梳理定義了機房的五大數據體系，在數據對接、數據類型、數據粒度上形成標準和建議。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a4/a4684f20335169a6edeb1634887752f9.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"開拓技術之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的強化學習一般是在離散的場景下，動作空間和樣本空間都很小，而實際任務往往比較複雜，有着很大的狀態空間和連續的動作空間，深度神經網絡則可以面對高維且連續的狀態自動提取複雜特徵，浩鯨智慧能耗算法團隊通過分析空調系統特點，採用深度Q強化學習模型將深度學習的感知能力和強化學習的決策能力相結合，可以適用於複雜的空調控制場景，並且模型更加合理、高效、快速收斂。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以機房風冷空調控制流程爲例：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6e/6e07d6688840c31ddbb07881f7d9d87a.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用對抗Q網絡強化學習的方式進行控制器的自學習，通過設定獎勵和反饋，探索和學習的機制讓控制器能夠自適應環境變化。在指令下發階段，系統實時收集傳感器溫溼度等環境狀態，通過e貪心策略進行動作選擇並執行動作。如在該動作執行週期內室內溫度超過預警值時，將降低空調設置值直至降至溫度下限，並重新開始週期採樣狀態。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法創新點如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1） 將數據採樣與監控模塊和強化學習控制模塊相結合。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採樣與監控模塊實現機房環境狀態週期查詢採樣和異常數據監控，強化學習控制模塊進行深度Q學習網絡的訓練和最優能耗的空調設定溫度生成。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2） 引入經驗回放和e-greedy貪婪策略提升模型訓練效果。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智能體與環境交互得到的訓練樣本存儲到經驗池。訓練時，每次從經驗池中選取小批量的樣本，通過梯度下降法更新網絡參數，經驗回放能夠打破樣本的相關性，使得訓練模型更穩定。在動作選擇上引入e-greedy貪婪策略，即有1-e的概率選擇平均獎勵最高的動作，剩下的e概率會隨機選擇一個動作作爲探索機制，防止進入局部最優。在實驗前期，由於缺乏經驗數據，e概率會被設定爲較大的值，增加探索性。隨着迭代時間的增長，e會逐漸減小，從而增加效果的穩定性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ae/ae53298c2545cdfa380fc6b95ad800ff.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"保障安全之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不管是機房還是基站，空調的系統安全和環境的安全都是影響業務生產關鍵因素，稍有偏差可能會導致不可預估的嚴重後果，因此無論採用哪種控制手段或者再先進的AI模型，首要的是保證系統安全問題和及時的應急策略。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於強化學習是通過真實環境交互進行訓練，因此在算法前期會有隨機探索動作的發生，爲了保證動作下發的安全性，通過引入專家經驗，將各類工況條件下的系統控制變量的合理區經驗和安全約束條件結合到模型訓練過程中進行動作空間的約束和警報閾值的應急處置，最大限度保證環境安全和穩定。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9d/9d910e5101f055d39553133227383739.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"浩鯨實戰","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今年初在浩鯨科技數據中心實現基於強化學習的風冷空調控制模型的實踐應用，通過空調實時動態控制下發、空調實時功率及機房溫度變化監測，使得機房的能源利用效率有明顯提升。上線兩個月後，空調能耗和PUE（機房總能耗/IT設備能耗）都有所改善：單日空調能耗平均降低了15.2%, PUE爲1.44平均降低了5.6%；單週空調能耗平均降低了13.6%, PUE爲1.48，平均降低了4.5%。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bd/bda9f9939732b5d47fedd59e9f144ebc.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"空調實時控制下發參數","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ae/aef20bb98e2f861e92dfe1f340a86fe4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"空調實時功率及機房溫度變化趨勢","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/97/97fbd8b4a2ef931555b09cc2c8e5f735.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"空調能耗趨勢","attrs":{}}]}]}