FreeWheel实时数据系统弹性伸缩实践

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.引言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着大数据技术的不断发展，数据实时性的需求变得越来越迫切，这对实时数据处理的基础架构提出了更高的要求。如何应对实时数据的流量变化，特别是突发流量，成为实时数据处理架构不得不面对的挑战性问题。对于FreeWheel这样一家服务全美90%的主流电视媒体和运营商的视频广告投放和管理平台，对于突发流量的应对能力尤为重要。数据基础架构团队作为FreeWheel数据处理的载体，提供了公司级的实时和离线数据处理及查询服务。在本文中，我们将和大家分享我们在实时数据处理弹性伸缩方面的实践。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.FreeWheel实时数据平台架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FreeWheel"},{"type":"text","text":"数据基础架构基于大数据开源软件和部分自研软件构建，所有服务均部署在"},{"type":"text","text":"AWS EC2"},{"type":"text","text":"上。目前，整个数据平台为"},{"type":"text","text":"Lambda"},{"type":"text","text":"架构，并逐渐向流批一体的融合平台发展。其中，实时数据平台整体上主要分为四个部分：基于"},{"type":"text","text":"Kafka"},{"type":"text","text":"构建的消息队列，自研的消息处理中间件"},{"type":"text","text":"Matcher"},{"type":"text","text":"，基于"},{"type":"text","text":"SQL"},{"type":"text","text":"开发方式的实时计算平台"},{"type":"text","text":"SQL on Streaming(SOS)"},{"type":"text","text":"以及以"},{"type":"text","text":"Druid"},{"type":"text","text":"为主体构建的实时数据查询服务。"},{"type":"text","text":"  "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/dc\/64\/dcfaec4bb9yy5e11b43867816d647664.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}