全网播放超 5亿的秒级实时台网互动是如何实现的？

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一年一度的双十一又双叒叕来了，给技术人最好的礼物就是大促技术指南！ 而经过这些年的发展，大促早已不仅仅局限于电商行业，现在各行各业其实都会采用类似方式做运营活动，汽车界有 818，电商有 618 、11.11 等等，各种各样的大促场景，对包括数据库在内的基础软件提出了很多新挑战，同时也积累了诸多最佳实践。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在双十一到来前，PingCAP 与汽车之家、易车网、京东、中通等用户展开一系列深入探讨，希望为大家揭秘逐年飙升的销量背后隐藏着什么样的技术难题？用什么技术架构才能平稳地扛住流量洪峰？"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"818 全球汽车节"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中国互联网有三大购物节，11.11、618 还有 818。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"618 与 11.11 都是大家非常熟悉的，818 则比较特殊，它是专为购车用户打造的节日狂欢。汽车之家 “818 全球汽车夜” ，就是由汽车之家与湖南卫视联手打造的汽车行业顶级盛典，到今年已经成功举办三届。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相对于其它两个购物节，818 可以说是全世界唯一的，其他任何汽车最发达的国家也没有这类活动。对此，汽车之家资深工程师张帆解释道：“我个人感觉，现在做电商和线上交易的这一块，地球上应该没有哪个国家能超越中国。而为什么汽车之家是最早来做这个事情的呢？首先，汽车之家是全球访问量最大的汽车类型网站。正是有着这样巨大的凝聚力与用户基础，汽车之家才能做这个事情，才能在广大用户中带来这样的影响力。此外，这个活动的初衷就是希望为汽车用户和汽车爱好者，提供一个类似于 11.11、618 一样真正能在买车的时候得到优惠的机会，因此广受用户欢迎。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从 2019 年开始，汽车之家与湖南卫视合作的 “818 全球汽车夜” 已经持续了三年。与传统大促不同，818 全球汽车夜通过电视直播和与 APP 活动同步的方式将汽车购物节推到高峰，为 8 月的汽车行业带来一场购车盛宴。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"818 直播活动带来的挑战"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"张帆坦言，在汽车之家的 818 活动中，直播环节是最难的。与录播完全不同，直播的过程中，会有非常多的变数，也许会有节目时间的拉长，也许会有主持人的即兴发挥，也许后台还会有一些突发的数据处理。而作为整场晚会的亮点，一元秒杀车、红包抽奖以及超级大锦鲤等活动，是用户参与度最高，峰值流量出现的环节。这些活动开始与结束的时机，"},{"type":"text","marks":[{"type":"strong"}],"text":"必须以秒级的精度来让前台、后台配合"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直播当天，汽车之家通常会专门派一支团队到湖南卫视直播现场，通过手机、电话、5G 对讲机、在线视频连线等多路通讯与位于北京的“作战室”之间实时沟通。由于直播信号通常比现场信号晚一分钟，当前面主持人在说三二一秒杀开始后，后台其实只有一分钟的准备时间。一分钟后，就要让电视机前的上百万用户在手机上真的能看到三、二、一，秒杀的按纽点亮，可以去按下它参与活动。这个过程完全不能出错，必须实现一比一同步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个过程对于后方“作战室”中的张帆他们来说，感受非常直观。这个“作战室”内有数据大屏、监控大屏，以及现场直播的信号和直播看到的电视信号。每一次秒杀开始或红包开始时，监控大屏中的几条线就会随着参与人数和互动次数的增加呈现断崖式的波动。这些代表着业务指标的线被他们称作“心电图”，而在直播中某些高人气明星出场时，这个波动甚至会比其他时段高 2-4倍多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"与此同时，"},{"type":"text","marks":[{"type":"strong"}],"text":"现场的数据大屏也在以 1-2 秒的速度，实时展示大约 20 项数据指标"},{"type":"text","text":"，包括活动参与人数、用户互动次数、奖品发放情况，甚至细化到这一轮一元秒杀车活动参与的用户有哪些人，在什么地方，中了什么车。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些实时的数据不仅会被后台工作人员看到，同时数据也会实时展示到直播现场。这对现场活动的气氛起到了非常重要的烘托作用。举例来说，当用户在屏幕前看到这场晚会人气火爆，并真的有许多人参与到一元抢车互动中，这对他而言就相当于一个反向激励，继而也参与其中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在这个过程中，"},{"type":"text","marks":[{"type":"strong"}],"text":"实时数据大屏不仅要解决实时交易问题，还要将实时分析数据反馈给现场的主持人"},{"type":"text","text":"。当主持人几乎实时地将中奖信息公布出来时，晚会气氛也推到了高位，这对于吸引更多人参与其中起到了关键作用。而随着秒杀的车越来越贵，越靠后系统所承受的波峰也越高。相对于汽车之家平时的业务，晚会经历的流量翻了十倍都不止，对整个系统的压力不言而喻。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"汽车之家大促解决之道：分布式系统全家桶"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大促场景通常要求系统具备快速扩展与高可用的能力，而分布式系统天然就具有这种能力。汽车之家采用了全家桶式的分布式系统，包括数据库、队列、缓存等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，分布式数据库主要表现出三种能力，分别是"},{"type":"text","marks":[{"type":"strong"}],"text":"水平高扩展性、容灾能力、云端能力"},{"type":"text","text":"。基于分布式架构的 TiDB 从一开始就支持这些特性，并在汽车之家的场景中得到了很好的验证。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"汽车之家数据库负责人陶会祥表示，传统关系型数据库，如 MySQL 、SQL Server 等，在数据量特别大时，常常会碰到一些数据库单机承载能力上限的问题。 TiDB 从 TiDB Server，到 TiKV 、PD 都可以进行水平扩展，性能随着水平扩展可以得到线性提升，很好地满足了汽车之家对于性能和扩展性的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"818 对于汽车之家而言是一年中最重要的活动，系统必须保障绝对的可靠稳定。所以这次 818 活动，汽车之家在公有云上采用了同城三中心部署 TiDB 集群，避免万一某一个机房出了问题，影响整体活动的服务质量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同城三数据中心方案，即同城有三个机房部署 TiDB 集群，同城三数据中心间的数据同步通过集群自身内部( Raft 协议)完成。同城三数据中心可同时对外进行读写服务，任意一个数据中心故障时，集群能自动恢复服务，不需要人工介入，并能保证数据一致性。TiDB 同城三中心架构在 818 晚会期间顺利地支撑了业务，运行表现十分稳定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/e8\/d7\/e8c70698685d5f0a73dfaa2d65712bd7.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"汽车之家 818 TiDB 集群整体架构图"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本次 818 项目中，汽车之家使用了 TiDB 最新的版本 5.1.1，MySQL 的版本是 Percona 5.7.25；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TiFlash 是 TiDB HTAP 形态的关键组件，它是 TiKV 的列存扩展，主要用于 OLAP 业务。TiFlash 跨区部署提高容灾能力，汽车之家利用 TiFlash 解决统计分析类的 SQL，实时展示在大屏；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TiCDC 是一款通过拉取 TiKV 变更日志实现的 TiDB 增量数据同步工具，具有将数据还原到与上游任意 TSO 一致状态的能力，支持其他系统订阅数据变更。 TiCDC 跨区部署， 将 TiDB 集群数据实时同步至下游的 MySQL 数据库，作为故障应急的备份，实现业务容灾能力的提升；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL 跨区部署主从，作为 TiDB 集群的应急、降级之用，实现业务容灾能力的提升。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"数据库压测"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 818 活动前，数据库团队联合业务方一起做了一轮一轮严格的故障演练压测，确保后端的高可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"陶会祥透露，汽车之家的故障演练分为多种，光数据库就会演练主库故障和机房故障，一共做了三轮。每一轮测试中 TiDB 的表现都非常优秀，KV 故障基本在几十秒，只需 20 秒即可恢复，即使机房故障也能在一分钟之内进行自动切换。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了保障活动平稳支撑，PingCAP 社区技术专家连续三年为汽车之家提供了社区技术支持。在今年的压测环节中，社区技术专家与汽车之家 DBA 一起完成了调优，良好地解决了写入热点问题，将性能翻了好几倍。最终在 818 高峰时期，TiDB 顺利支撑了晚会期间 APP 用户 9048 万次互动，并抗住了最大每秒 40 万行的写入，SQL 99 稳定在 30ms 以下。TiCDC 性能表现也十分强劲，向下游 MySQL 同步速度高达 13 万行每秒 。跨中心的 TiFlash MPP 架构，为大屏近实时展示助力总次数、秒杀和摇奖的每轮参与用户等信息提供了强有力的支撑。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"陶会祥都对大促中 TiDB 的表现给予十分高的评价：TiDB 在这种十亿以上的数据量级场景下是非常适合的，一是 TiDB 的分析能力是实时的，二是 TiDB 的数据存储能力比传统数据库，如 SQL Server 之类强太多。 TiDB 结合了传统数仓和传统关系型数据库的优点，非常适合应用在大促这种量级的业务环境。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"未来规划"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"汽车之家的数据库团队在本次 818 大促中，也总结出了非常多的最佳实践："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如同城三中心五副本架构，机房之间延迟应当尽量小，最好控制在 2ms 以内；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OLTP 的业务，通常压测瓶颈在于 TiKV 的磁盘 IO 上，对于普通 SSD ，可以做成 RAID0 来提升 IOPS；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一旦某个可用区整体故障，正常不需要手动干预，但是为了避免性能下降严重，建议手动将五副本调整为三副本；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"合理设计表结构和索引，尽量避免热点问题，和业务一起做好充分压测，压测期间尽早发现问题并优化。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于本次活动中的良好表现，陶会祥表示，汽车之家接下来还会在更多业务中推进 TiDB 上线。比如以前汽车之家的很多数据会跑在 Hive 里，需要到第二天才能知道昨天发生了什么事。如果应用 TiDB ，可以针对运营需要的用户数据、业务指标的分析，去做秒级的准实时推送，预计能够将这一时间压缩到 5-10秒。业务方可以立即知道上一刻用户有什么变化，数据有什么更新。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]}]}