一年40万次实验，字节跳动A/B测试平台是怎么炼出来的？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"采访嘉宾 | 王珂"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"编辑 | Tina"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"2012年，刚刚建立的字节跳动便开启了A\/B测试之旅，随着今日头条、抖音、西瓜视频等全线业务的使用，将A\/B测试应用在产品命名、交互设计、推荐算法、用户增长、广告优化和市场活动等各方面决策上。据今年4月字节跳动旗下火山引擎技术开放日上透露的数据显示，字节跳动每天同时进行的A\/B测试达到上万场，单日新增实验数量超过1500个，覆盖400多项业务。随着公司发展，这些数字还在不断扩大，仅最近一年就已经做了40万次A\/B测试。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"那么在字节跳动里，支撑这些测试的技术平台是怎么炼出来的？我们采访了字节跳动A\/B 实验平台技术负责人王珂，通过他的回复我们可以得到一个初步的了解。王珂将在2021年10月21-23日"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/shanghai\/","title":null,"type":null},"content":[{"type":"text","text":"QCon上海站"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"分享主题为《"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/shanghai\/presentation\/3861","title":null,"type":null},"content":[{"type":"text","text":"字节跳动 A\/B 测试平台演进历史及实践"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"》的演讲，更多内容可以通过观看演讲进行了解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"嘉宾简介：王珂"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"，字节跳动A\/B 实验平台技术负责人。目前就职于字节跳动数据平台，负责内部 A\/B 实验平台的研发，支撑内部各个业务线的 A\/B 实验需求，在 A\/B 实验领域有比较深入的理解。曽经任职于亚马逊中国库存健康优化团队，有多年大数据及服务架构经验。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有说法是目前这种形式的A\/B 测试最早出现于 1990 年代，您认为其核心原理这么多年是否有改变？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"核心原理基本没有发生什么改变，仍然是依赖随机采样获取两个样本集合，施加不同的策略，采集结果比较和分析。如果要说变化，更多的是应对实际的A\/B测试场景，在测试方式、指标设计和显著性分析方法等细节上有了更多的探索和演进。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"能否以一个简单的例子，说明A\/B 测试如何工作？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"简单来说，一个A\/B测试生命周期大致分三个部分：A\/B测试设计（包括测试内容、预期收益、测试时长、测试流量、观测指标等的确认）、A\/B测试执行（利用A\/B测试平台的能力完成分流、配置分发、数据回收等）、A\/B测试结果分析（产出指标数据，显著性分析，多维下探等，最后形成分析报告）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"A\/B 测试适合哪些场景？能运行 A\/B 测试需要哪些必备条件？字节跳动的 A\/B 测试平台主要适用于什么场景？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通常而言，可以将样本随机划分为互不相关的两组，同时施加不同策略，并可以提供量化指标衡量策略效果的，都可以进行A\/B测试。比较典型的如政策调整，无法随机将民众划分为两部分，一部分执行新政策一部分执行旧政策，这种就不适合进行传统意义的A\/B测试，而通常会尝试在一个城市试点新政策，通过 DID 或者 SCM 等分析方法检验效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"字节跳动的 A\/B 测试平台立足于服务和支撑字节跳动内各大业务线的 A\/B 测试需求，当前主要适用于算法迭代、产品演化、智慧运营等场景，未来也会随公司的脚步覆盖更多的场景。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"字节跳动的A\/B 测试平台有哪几个主要部分组成，整个平台大概经历了什么样的迭代过程？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"经过多年的迭代，字节跳动A\/B测试平台由最初服务于推荐算法迭代，到现在包含A\/B测试、配置发布、自动调参和探索实验室四大部分，覆盖了A\/B测试的整个生命周期。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"数据采集部分，一般平台使用的是埋点或日志数据，那么字节跳动的平台是通过什么方法实现的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我们的平台也是基本基于埋点和日志数据来生产测试数据的。在字节跳动，埋点和日志数据汇集都有系统化的解决方案，使得我们的 A\/B 测试平台可以比较容易的给出A\/B测试结果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"是否有一些测试比较复杂？字节跳动如何降低复杂性，让业务人员易于理解和使用？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"会遇到一些比较复杂的场景，平台也会尝试优化产品以降低使用门槛。一个比较典型的案例是算法侧的超参选择。在机器学习模型中经常会遇到一些超参，需要算法工程师凭借经验和A\/B测试结果来调整这些超参的取值。传统做法下，算法工程师需要花几个月的时间，通过不停的 A\/B 测试对比调整遴选合适的超参取值。为降低该场景的使用复杂性，字节跳动的A\/B测试平台通过一些统计学方法，自动化的循环执行A\/B测试，分析测试结果，预测最优解取值，协助算法工程师寻找到合适的超参，使得调参耗时由几个月缩减到几个礼拜。这也就是我们的自动调参系统。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"人们在做 A\/B 测试时会犯哪些常犯的错误\/陷阱？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"比较常见的是“连续观测”。举个夸张点的例子就是，一个A\/B测试启动起来，使用者每天都会过来看一下是不是指标有显著正向；直到突然有一天，指标正向了，使用者开心的关掉A\/B测试，撰写上线报告。这种连续观测，一旦显著立即决策的做法会令使用者拿到错误结论的风险大幅度上升，是不可取的。因此在字节跳动，A\/B测试使用方式的宣讲是我们需要例行去做的很重要的一个事情。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"A\/B 测试，可能因为不同的受众行为不同，对一家公司有效的东西不一定对另一家公司有效。那么字节跳动的A\/B 测试平台如何具备普适性？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一方面，字节跳动的A\/B测试始终以平台的方式将A\/B测试做合理抽象，向不同的业务场景提供测试能力，考虑到公司较为复杂的产品矩阵，A\/B测试平台从诞生至今的一路迭代中始终站在A\/B测试最基本的抽象上，以保证其普适性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"另一方面，像指标体系等与业务场景关联密切的资产，我们既要考虑它可能不具备普适性，而需要做到因业务而异；也要考虑到相似的业务线可能会重复建设相似的指标体系。因此，能够“将经验复制一份”也是我们平台需要频繁考虑的东西。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"A\/B 测试平均耗费时长是多少？如何减少“延时”，以比较快的速度得到结果，这方面您们有哪些可供大家参考的经验？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"不同场景，不同目的下，A\/B 测试需要进行的时间也会有比较大的差异。例如搜索算法的小流量测试是为了快速探索算法迭代的可行性，几天或几个小时便能给出有价值的结果；而产品界面的变更，为了规避所谓的新奇效应，避免有些用户出于好奇心而带来的短期指标上扬，测试可能会开启几个礼拜甚至几个月。比较典型的A\/B测试通常会持续1-3个礼拜。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通常而言一个A\/B测试需要耗费多久，和A\/B测试内容、测试设计有关。相比而言，减少一个A\/B测试消耗的时间不如提升A\/B测试的并发性，让系统同时容纳更多的A\/B测试，对产品的整体迭代效率提升更加有益。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"字节跳动A\/B 测试平台有哪些未来规划？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"王珂："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有三个比较重要的方向是我们的A\/B测试平台在当前比较关注，在未来会加大力度投入的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一个是对A\/B测试分析能力的更大力度支持。开A\/B测试在字节跳动相对是比较容易的，但是A\/B测试分析和更多有价值信息的挖掘却没有那么容易。曾经有一个测试，指标上看并没有显著的提升，然而在一个特殊的维度上我们发现了显著的效果，进一步分析推理之后我们对策略进行了调整，最后还是拿到了比较大的收益。由此可见分析能力对于测试平台而言的重要性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"另一个是场景化支撑的能力。字节跳动的产品矩阵复杂度越来越高，不同的业务领域对A\/B测试有着不同的诉求，相较于能力堆砌功能强大的巨无霸，一个清新简洁切合业务属性的系统对于业务迭代效率的提升更加有益，这也是我们架构演进的必经之路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"最后一个也是最重要的一个，是方法论的探索和储备，拓宽A\/B测试的边界，应对今天和明天我们在业务上会遇到的新问题，例如在社交领域如何更好的解决网络效应等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"活动推荐："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在今年10月21-23日举办的QCon全球软件开发大会上，王珂老师将在【大数据分析技术选型】专题分享主题为《字节跳动 A\/B 测试平台演进历史及实践》的演讲。在为期三天的大会上，来自不同技术团队的一线专家还将围绕云原生、AIOps落地实践、大数据分析技术选型、移动新生态、可观测性技术等25个专题方向分享100+技术实践案例，会议日程已上线："},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/shanghai\/schedule?utm_source=wechat&utm_medium=infoq&utm_campaign=9&utm_term=0927&utm_content=yueduyuanwen","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/qcon.infoq.cn\/2021\/shanghai\/schedule?utm_source=wechat&utm_medium=infoq&utm_campaign=9&utm_term=0927&utm_content=yueduyuanwen"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]}]}