Uber：模型性能调优

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 使用机器学习模型来支持关键的业务决策。一种机器学习模型要经过多次实验迭代才能进入生产阶段。在实验阶段，数据科学家或机器学习工程师将探索添加特性、调优参数以及运行离线分析或回溯测试。在保证生产中高模型质量的同时，我们通过强化平台来减少这一阶段的人力和时间。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"超参数优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/5f\/5f56ae2b7e38a1c88dacd4556df60526.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图 1：基于 TURBO 的超参数优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于维数灾难，在大型超参数搜索空间中的搜索成本很高，即使对于能够平衡探索与利用并通过后验结合信念更新的 贝叶斯优化 技术也是如此。超参数的重要性和选择对于帮助降低搜索空间的复杂性具有重要意义，因为超参数对模型性能有很大的影响。结果表明，对于像 XGBoost 这样的基于树的模型来说，最大树深、最大树数、学习率和最小子重等的调整通常会会产生更好的性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这样做的目的是主动显示已知类别问题的适当参数，从整体上节省时间。除了优化 二值分类问题 的 AUC或 回归分析 问题的 均方误差（MSE）之外，我们还公开了额外的目标函数，以便数据科学家和机器学习工程师可以为他们的问题选择合适的优化目标。为避免 过拟合，我们在优化函数中引入一个惩罚项，来捕捉训练和测试性能之间的差异。要加速对超大数据集的 超参数搜索，如果我们在新的试验中没有发现明显的改进，则允许超参数优化研究提前停止。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"学习曲线与提前停止"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在训练期间，通过绘制模型或估计器对评估数据集进行多次迭代，学习曲线 通常被用来评估和诊断机器学习模型。它分析了模型从额外的迭代训练中获得的增量性能收益，并就 偏差 - 方差权衡 问题提供了深入的见解。从图 2 中可以看到，对增量学习过程的知识使用户能够对模型的拟合作出明智的判断，并在每一次实验的基础上急切地应用 提前停止。超参数优化设置中，在一个实验中存在多个同时进行的实验，提前停止也可以跨实验联合应用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/9a\/9a27f14e26eafb64951dfb5dd6480825.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图 2：使用提前停止节省可能的计算时间"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根据每次实验结果，利用学习曲线可以估算出数百个相互作用达到稳定状态前的收敛性，在一般的超参数优化设置中，经过大量实验，每个实验的效率增益与实验的数量成比例。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/b1\/b1b0f03dcc4d55b1be37cb8e8c7f1d12.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图 3：属于同一超参数优化工作的所有试验的学习曲线。红色边界框突出显示曲线，根据早期表现与其他正在进行的试验相比，这些曲线不太可能成为最佳候选者"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"尽管我们可以将提前停止应用於单个实验，以修剪多余的迭代（如图 3 所示），但是通常可以通过在所有同时进行的实验中共享信息来获得额外的空间。在一个实验的所有实验中联合使用学习曲线信息，可以让我们估算该实验将优于其他实验的概率。诸如 ASHA（"},{"type":"text","marks":[{"type":"strong"}],"text":"A"},{"type":"text","text":"synchronous "},{"type":"text","marks":[{"type":"strong"}],"text":"S"},{"type":"text","text":"uccessive "},{"type":"text","marks":[{"type":"strong"}],"text":"H"},{"type":"text","text":"alving "},{"type":"text","marks":[{"type":"strong"}],"text":"A"},{"type":"text","text":"lgorithm，异步连续减半算法）等技术，利用由分布式实验执行的中央调度器促进的顺序贝叶斯优化，可以联合优化整个研究，使整个超参数调整过程更加迅速。在接下来的博文中，我们将详细介绍我们的自动超参数优化服务架构，以及在生产中采用黑盒优化服务的结果。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"特征变换"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 的机器学习模型利用了丰富的地理空间特征，通常需要用特征工程来提取有意义的信号。一种方法是将经度和纬度离散化，但是这会产生非常高的基数（cardinality）特征。将经度和纬度变换为不同的嵌入空间，然后从嵌入空间中提取出地理空间信号。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于树的模型执行分段线性函数近似（piecewise linear functional approximation），这种方法无法很好地捕捉复杂的非线性交互作用。在基于树的模型训练过程中，我们没有使用内核技巧，而是自动加入顶级复合特征，重要性低的复合特征将被删除。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"额外策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 的业务数据有很强的时间因素。行程信息被摄入并按照日期字符串划分到数据湖中。多数 Uber 的机器学习模型都是在日期范围内训练的。举例来说，使用更广泛的日期范围使模型能够以周或月为周期来捕捉。增加数据的日期范围是提高模型准确性的常见方法，这也是一个关键的最佳实践。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，我们也看到一些情况，日期范围的增加会导致模型的准确性降低。出现这种情况的原因之一是，更大的范围会放大异常值的影响。在这种情况下，我们检查数据并在 DSL 中进行补偿。另外一种可能是由于预测的行为会随着时间而改变。因此，对训练数据进行时间加权衰减可以起到一定的作用。通过使用 XGBoost 列加权特性和 DSL 转换来完成此任务。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"支持最佳实践的工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"应用模型调优的最佳实践需要进行大量的工作和跟踪。我们称这一过程为“金矿开采”。金矿开采始于克隆客户的 Michelangelo 项目。这样可以避免在客户的生产环境中运行实验性训练作业。每个实验性运行都涉及许多容易出错的步骤，包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"仔细设置（例如，延长日期范围意味着改变训练 \/ 测试拆分，如果它是基于日期的）。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"应用相关的启发式方法（例如，设置超参数搜索范围）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更新计算资源（例如，分区模型需要更多的工作器或并行性，以避免更高的延迟）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"记录实验结果并确定当前的最佳模型"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们已经将上述步骤、模型探索和模型分析自动化为一个工具，以加快金矿开采的过程。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"创建和维护高性能的模型是一个迭代的过程。Michelangelo 提供了大量的功能目录，如超参数优化、提前停止、特征变换等，可在模型开发和调优阶段使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Michelangelo 还提供了执行全面重新调优的工具，包括模型架构搜索、确定特征协同和冗余性等。最终的结果是，我们已经能够为 Uber 部署的机器学习模型提供一个迭代调整和一次性全面调优的框架。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介绍："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Joseph Wang，Uber 机器学习平台团队软件工程师。负责特征存储、实时模型预测服务、模型质量平台和模型性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Michael Mui，Uber 机器学习平台团队高级软件工程师。负责分布式训练基础设施、超参数优化、模型表示和评估。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Viman Deb，Uber 机器学习平台团队高级软件工程师。负责黑盒优化服务、Uber 定制贝叶斯优化算法以及 Michelangelo 的超参数优化工作流。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Anne Holler，Uber 机器学习平台团队机器学习框架 TLM 前雇员。负责机器学习模型的标识和管理，以及训练和离线服务可靠性、可伸缩性和调优。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文链接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/tuning-model-performance\/","title":"","type":null},"content":[{"type":"text","text":"https:\/\/eng.uber.com\/tuning-model-performance\/"}]}]}]}