性能优化：空调能耗节能的强化学习探索之路

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"导读：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ICT行业一直以来都被公认为“能耗大户”，据权威机构预测2025年ICT行业耗电约占全球电力的20％，贡献全球碳排放量的3.5％以上，成为实现“碳中和”目标的主要阻碍和重要影响因素。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从作为ICT行业主要参与者的运营商领域来看，近些年随着数字化转型的加速和对算力需求的增长，数据中心和基站的建设运营成本和能耗持续增加，以国内某省份运营商数据中心机房为例，省管机房每年空调电费支出高达七八千万，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"降低10%以上","attrs":{}},{"type":"text","text":"的能耗即可节省千万级别的电费支出，因此如何通过科技化的手段推动行之有效的节能实践，成为需要运营商持续关注的重要目标。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"空调制冷侧能耗成为整体能耗的关注焦点","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过分析基站和数据中心两个主要的能耗组成来看，空调制冷侧能耗占比尤为突出。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在基站能耗中，目前空调侧能耗消耗最多，占比56%左右，主设备占比32%左右。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0d/0d0f165c58f228893ae76bf4cfb19c54.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在机房能耗中，IT占43%，空调约40%以上，其余电源、照明占17%左右。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b50c79426464565545c4b0194a4fe39.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于空调系统结构和理论复杂，目前大多运维人员只是长时间或者季节性的对冷源侧设备参数和末端空调温度进行单参数的管理控制，简单粗放。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"空调节能手段各异，仍需寻求新技术突破","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近年来，新技术飞速发展为空调系统的智能管控带来许多新的思路和方法，致力于通过对空调精细化的管理和控制节省空调侧的能耗支出。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如PID控制方式，是利用比例、积分、微分计算出控制量进行控制的，设计简单、成本低廉，但是空调系统作为典型的高度非线性、耦合性、时变性、不确定性的复杂多变量系统，这种传统控制方式很难通过简单的比例控制取得理想的运行效果，并且在实际应用过程中，每个机房的空调状况、业务承载和环境换化差异巨大，传统控制模型也无法快速适配所有机房情况。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前比较成熟的方法是通过对利用环境、空调和能耗等历史数据建立AI预测模型，预测未来空调和环境的能耗值，然后通过贪婪算法等优化求解算法求得在未来能耗值下空调的最优控制参数，这种方式严重依赖于模型质量，一旦模型与建筑或空调系统真实情况存在偏差，控制策略质量将无法得到保障。即便初始模型效果较好，但仍面临数据和模型更新的问题，导致模型的鲁棒性较差。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此面对复杂的环境变化、专业的空调系统、众多的控制参数，需要有新的AI技术，能够对空调系统的监测和运行数据进行分析，找出空调系统自身运行最本质的特性，在保证环境安全的前提下最大程度的挖掘空调节能潜能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"浩鲸基于强化学习的空调能耗节能之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"强化学习是解决策略优化问题的一种重要方法，特别适用于解决高度非线性、不确定性的复杂多变量系统，具有实时、免模型和在线自学习的特点，能够在缺少控制系统模型的情况下实现控制器的自适应优化，极大降低对先验知识的依赖。近些年来在自动驾驶、机器人等实际工程优化中逐渐开始广泛探索应用。强化学习智能体根据周围环境产生控制动作，动作下发后周围环境会产生新的状态和状态变化后的动作奖励，从而利用新状态和奖励优化智能体的模型，下发更合理的动作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a5/a5fa8cf5a747b9af7a4f51c169e17334.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于空调能耗节能问题，最终目标是通过合理的空调控制优化在保证基站或者机房环境安全前空调的能耗较少，所要关注的也是动作、环境和状态之前的关系。因此也适用于强化学习的模型思想。我们也在实际应用过程中，基于强化学习不断探索实践和突破，走出了浩鲸空调节能的新道路。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/57/57c858c5ff2bcfba18de9aecfc3b359f.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"夯实数据之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"强化学习来讲，尽管是免模型的自学习方法，即对先验知识的依赖性较弱，但是仍需要空调和其他相关状态数据的连续采集去得到动作下发前后的反馈才能进行模型不断优化，因此对数据时间和空间的维度要有一个清楚的认知才能更好的定义模型的参数和状态。以数据机房空调节能为例，在数据层面，梳理定义了机房的五大数据体系，在数据对接、数据类型、数据粒度上形成标准和建议。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a4/a4684f20335169a6edeb1634887752f9.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"开拓技术之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"传统的强化学习一般是在离散的场景下，动作空间和样本空间都很小，而实际任务往往比较复杂，有着很大的状态空间和连续的动作空间，深度神经网络则可以面对高维且连续的状态自动提取复杂特征，浩鲸智慧能耗算法团队通过分析空调系统特点，采用深度Q强化学习模型将深度学习的感知能力和强化学习的决策能力相结合，可以适用于复杂的空调控制场景，并且模型更加合理、高效、快速收敛。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以机房风冷空调控制流程为例：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6e/6e07d6688840c31ddbb07881f7d9d87a.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"采用对抗Q网络强化学习的方式进行控制器的自学习，通过设定奖励和反馈，探索和学习的机制让控制器能够自适应环境变化。在指令下发阶段，系统实时收集传感器温湿度等环境状态，通过e贪心策略进行动作选择并执行动作。如在该动作执行周期内室内温度超过预警值时，将降低空调设置值直至降至温度下限，并重新开始周期采样状态。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法创新点如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1） 将数据采样与监控模块和强化学习控制模块相结合。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"采样与监控模块实现机房环境状态周期查询采样和异常数据监控，强化学习控制模块进行深度Q学习网络的训练和最优能耗的空调设定温度生成。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2） 引入经验回放和e-greedy贪婪策略提升模型训练效果。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智能体与环境交互得到的训练样本存储到经验池。训练时，每次从经验池中选取小批量的样本，通过梯度下降法更新网络参数，经验回放能够打破样本的相关性，使得训练模型更稳定。在动作选择上引入e-greedy贪婪策略，即有1-e的概率选择平均奖励最高的动作，剩下的e概率会随机选择一个动作作为探索机制，防止进入局部最优。在实验前期，由于缺乏经验数据，e概率会被设定为较大的值，增加探索性。随着迭代时间的增长，e会逐渐减小，从而增加效果的稳定性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ae/ae53298c2545cdfa380fc6b95ad800ff.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"保障安全之路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不管是机房还是基站，空调的系统安全和环境的安全都是影响业务生产关键因素，稍有偏差可能会导致不可预估的严重后果，因此无论采用哪种控制手段或者再先进的AI模型，首要的是保证系统安全问题和及时的应急策略。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于强化学习是通过真实环境交互进行训练，因此在算法前期会有随机探索动作的发生，为了保证动作下发的安全性，通过引入专家经验，将各类工况条件下的系统控制变量的合理区经验和安全约束条件结合到模型训练过程中进行动作空间的约束和警报阈值的应急处置，最大限度保证环境安全和稳定。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9d/9d910e5101f055d39553133227383739.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"浩鲸实战","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今年初在浩鲸科技数据中心实现基于强化学习的风冷空调控制模型的实践应用，通过空调实时动态控制下发、空调实时功率及机房温度变化监测，使得机房的能源利用效率有明显提升。上线两个月后，空调能耗和PUE（机房总能耗/IT设备能耗）都有所改善：单日空调能耗平均降低了15.2%, PUE为1.44平均降低了5.6%；单周空调能耗平均降低了13.6%, PUE为1.48，平均降低了4.5%。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bd/bda9f9939732b5d47fedd59e9f144ebc.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"空调实时控制下发参数","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ae/aef20bb98e2f861e92dfe1f340a86fe4.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"空调实时功率及机房温度变化趋势","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/97/97fbd8b4a2ef931555b09cc2c8e5f735.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"空调能耗趋势","attrs":{}}]}]}