接口需求HiveMysqlRedis性能要求请求QPS>1s<1√√√<1s<10
| √√<500ms>100
|
| √<100ms>100
|
| √"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这就造成了机票大数据部门的redis集群内存需求暴涨，目前我们统计redis使用的数据：挂在机票大数据部门的redis集群数量有几十个，内存达到了十几个T。当然接口的性能也达到了前所未有的快速和高效，基本都是10ms左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.2 如何查询"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Redis的查询方式比较单一：通过唯一key去查询value。这种查询方式在简单的唯一值查询中比较有效，但是当遇到，同一个数据源多关键字查询的时候，就得维护多份数据源。举例：在价格趋势的接口中，我们提供了多种价格趋势组合：国内、国际、单程、往返、航线、航班。如果使用redis存储，需要维护同一份数据多种key的存储方式，极大地浪费了存储空间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Redis还有一个问题是时间范围的筛选，还是在上面的价格趋势接口中，需要按照查询时间返回历史同期在一定起飞时间范围的价格数据，所以我们需要存储多个时间日期的数据（当然也可以用set等结构，但是会面临如何删除过期数据的问题），同时在查询的时候需要循环取一定时间范围的价格。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.3 如何维护"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1）接口维护"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大数据基础平台团队一共维护了几百个接口，其中1\/3的接口是提供数据给调用方的，这当中又有一些接口只是提供简单的查询操作，但就是这些简单的查询，需要我们提供海量的数据存储、快速精准的查询。每个接口的上线需要经过项目资源申请（包括机器资源、人员资源）、数据同步、开发、测试流程，最后才能上线。一整套流程走下来，耗费2-3天\/人，而且基本上都是是重复性的工作。如何解放这些人力和机器资源，就变得很迫切了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2）数据同步"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提供给外部使用的数据大部分都是存储在hive中，在不使用presto api的方式访问时，我们需要将hive数据导入到redis或者mysql中，供接口访问。在zeus平台上，我们建立了各种导数据的流程，如何将这些简单、重复度高的流程自动化呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个接口的架构图如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/24\/24c5923a25368dcc09a63bae23ec92a8.jpeg","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1 redis\/mysql作为主要存储的架构图"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、机票大数据接口的大道之旅"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"认真研究了接口调用方本身的性能，我们发现调用方在调用第三方提供的接口时，基本都是异步进行的。如果把调用方调用的所有第三方接口当成一个木桶，机票大数据基础架构团队的接口就是其中的一块木板，只要不是最短的木板，就可以在保证性能的情况下降低整个接口的响应时间（当然这不是技术上的退步，而是选择合适的方案）。通过上面的存储选型对比之后，发现在100ms-500ms这个性能段里面没有一个合适的存储方案能够提供。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们调研了几种NOSQL数据库方案，综合存储、查询等指标发现CrateDB比较符合现实需求。将几种存储做了一个对比，如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"对比RedisMongoCrateDB查询速度<10ms100ms~500ms100ms~500msSQL不支持不支持支持数据结构化不支持支持支持存储机制hashSharding+partitionSharding+partition资源利用内存资源硬盘+内存硬盘+内存数据可重复使用不支持，单一固定key支持支持"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 CrateDB介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CrateDB是构建在NoSQL（ElasticSearch）基础之上的分布式SQL数据库，它结合了SQL的通晓程度和NoSQL的可扩展性和数据灵活性："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a、使用SQL处理结构化或非结构化的任何类型的数据"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b、以实时速度执行SQL查询，甚至JOIN和聚合"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"c、简单缩放"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 CrateDB与接口存储"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CrateDB很好地解决了100ms-500ms性能段的短板，并且使用磁盘+内存的方式存储数据，减少了内存的使用。目前在我们生产时间中，通过12台8核24G虚拟机30%的磁盘空间覆盖了10亿数据（如果是redis至少需要300G的内存，如果做slave，容量double）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3 CrateDB与接口查询"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CrateDB提供了如MYSQL的表、字段等概念（底层使用ES存储引擎），我们可以将同一份数据源进行多维度的操作，比如上述讲到的价格趋势里面基于航线和航班的价格趋势，这两个接口可以使用同一套数据源，因为航线的价格可以基于航班数据进行聚合操作，这样就大大减少了冗余的数据。同时类MYSQL表的特性使得时间范围的查询变的so easy了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4 CrateDB与接口维护"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1）与接口结合使用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因为CrateDB支持标准的SQL，我们开发了机票大数据基础平台的通用性api系统，通过将取数逻辑SQL化的方式，同时利用qconfig api将新增的数据需求进行模板化、配置化，统一了接口代码开发的流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置页面如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/66\/669c26a6518c3b07e0814bd670cc86e1.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图3 接口配置页面"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2）数据同步"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过zeus api将同步数据流程模板化，配置页面如下图。并且在zeus平台上，使用spark shell方式将hive数据导入到CrateDB中，抛弃了以前jar包的方式。这种方案可以在几分钟内导入千万级的数据（取决于CrateDB表的数据结构，减少索引、doc_values以及刷新间隔会都有利于导入的速度）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/66\/66186d3ad351eb6408915abb90f65a5e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图4 zeus流程配置页面"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3）容器化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何更加有效地管理、维护CrateDB集群？为此我们上了k8s，将CrateDB容器化。为了更好地管理这些k8s集群，引入了rancher，rancher是开源的企业级容器管理平台，通过rancher，我们再也不必自己去从头搭建容器服务平台。同时rancher提供了在生产环境中使用的管理docker和kubernetes的全栈化容器部署与管理平台。将网络、磁盘虚拟化之后，资源的利用率大大提高，减少了虚拟机的使用。自动水平扩展，以及pod的监控等特性，都极大地提高了维护CrateDB的能力，我们管理的CrateDB集群如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/79\/79e0c98d003358d52fe3aa87b36279f3.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图5 rancher管理CrateDB集群图"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.5 与接口结合的其他优势"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）存储机制多样化，底层的存储机制支持多样化的数据类型，同时支持partition、sharding；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）数据结构化，CrateDB提供结构化的展示，有利于数据的可视化以及降低非技术人员的理解难度，解决了redis可读性差的问题；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）存储可靠性，数据持久化存储在磁盘上，支持replica，相比于redis的内存存储更加可靠（当然redis也可以落盘，但这就会限制redis的速度）；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4）成熟的优化机制，针对es的优化我们有丰富经验的技术人员支持。举个例子：我们有9000万+的用户行程数据，因为数据比较详细，字段的内容比较庞大。通过去掉部分字段的索引，去掉doc_values等操作将数据存储大小从90G降到了30G，同时也提升了搜索速度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前在生产上我们部署了2个CrateDB集群，其中一个集群由12台8核24G内存虚拟机组成。在集群中建立了12个数据表，存储了20+亿条数据，经受了生产的实际考验，接口性能指标如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"数据量99line95lineavg查询特点描述10亿+200ms80ms10ms多关键字、时间范围查询整个集群请求量1500qps500w+150ms50ms10ms多关键字查询、排序单个表请求量400qps9000w+200ms100ms60ms多关键字查询| 单个表请求量10qps"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"性能满足了大部分调用方的使用需求，同时系统数据上线的流程由以前的申请资源、开发代码、测试、上线，到现在的系统配置、测试、上线，释放了部分的开发资源，并且保证了数据的质量。接口上线时间由以前平均2-3天，缩短为2-3小时。新的接口架构图如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1e\/1ebc38f2b23fd3bd3bbe3ce09dac9a3e.jpeg","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图6 CrateDB作为主要存储的架构图"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、安装部署"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CrateDB有官方版以及社区版，为了更好地进行自维护，我们选择了社区版（通过源码编译）。CrateDB的部署与ES的部署基本一致。需要注意的是，在分配内存的时候尽量多留一些内存给系统，这将有利于数据查询速度。部署后的webui如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/49\/49b1f4bcecfce06f1321e9ec52260f9d.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图7 CrateDB webUI"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、数仓中的实现"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前在数仓中的应用主要体现在各种指标dashboard、metrics的展示，比如fltinsight。与以往通过presto接口获取数据的方式相比，更加直接、高效。而且CrateDB支持各种字段的聚合、统计，是各种指标存储、展示的不二之选。当然后续数仓组也会在数据展示这一块全面推广CrateDB的使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"六、小结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"没有完美的存储方案，只有最适合的存储方案。"},{"type":"text","text":"通过上述机票大数据平台在数据存储这一块的实践经验，相信每个团队在面对选择存储方案的时候，结合自身需求去选择适合自己的存储技术方案，达到“大道”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"嘉宾介绍："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Loredp，携程数据分析经理，关注大数据存储、大数据处理以及linux等领域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自：携程技术中心（ID：ctriptech）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/8mLllQZq6E0eePutkr_L1A","title":"xxx","type":null},"content":[{"type":"text","text":"干货 | CrateDb在携程机票BI的实践"}]}]}]} | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | |
{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着整个互联网流量红利进入末期，各大厂在着力吸引新客的同时，在既有客户群体的运营上也是煞费苦心，各种提高客户体验、个性化服务的场景层出不穷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"携程机票大数据部门在实践过程中需要同步数据、选型引擎来存储处理数据，利用接口将模型结果开放给生产环境调用，因此我们的数据存储修炼之旅会涉及到"},{"type":"text","marks":[{"type":"strong"}],"text":"接口现状、接口大道之旅、安装部署、同步数据、生产应用以及未来的趋势-如何实现容器化"},{"type":"text","text":"。这当中，我们遇到了很多问题，也解决了很多问题，本文将分享机票大数据平台在数据存储这一块的实践经验。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、机票大数据接口现状"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"携程机票大数据平台接口组碰到的问题："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何存储"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何查询"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何维护"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.1 如何存储"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"机票大数据基础架构团队接口组在2018年之前，数据的存储方案基本是：hive、mysql、redis。以下是我们现有的存储选型："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"