云原生数据中台技术与趋势解读

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据中台发展至今，大体经历了4个重要阶段：数据库-数据仓库-大数据平台-数据中台。每次新的变革，都是为了解决上一阶段存在的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当前，走向云原生成为数据中台的必然和必须。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"云原生从何而来？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"云原生是用于指导如何在云上构建和运行应用的方法论。奇点云高级技术专家，奇点云数据智能平台DataSimba总负责人地雷谈到，“云原生”并不是一个新的概念。回顾云计算史，从个人端应用到企业级应用，都早已开始“上云”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"起初，这些上云的“非原住民”应用，延续了私有化部署的技术架构，把本地软件不加修改地通过ECS迁至云端。而ECS的弊端在于只能承载计算，无法实现存储。虽然上云后的应用实现了业务打通，但随着业务扩大，原有的架构“可用性”明显下降。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"国内云厂商为了解决数据存储问题，制作了云磁盘，将其挂靠在云主机上，实现数据备份，且无需更改程序。传统软件上云的“高可用”问题得以解决。然而，这种方式引发了另一弊端——成本高。客户把Hadoop不加修改直接部署到ECS节点上，数据通过HDFS存储在云磁盘上，需花费大量成本。因此必须修改HDFS底层，把数据存到对象存储上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着需求不断丰富，系统必须按照IaaS、PaaS的技术特点进行重构，以便跟上业务和数据的爆炸性增长。 在私有化部署以及上一代传统技术的软件架构运维方法论的基础上，带着“高可用”、“低成本”等属性，“云原生”升级而出。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"云原生数据中台具有哪些技术要素？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“云原生”概念发展至今，我们已并不陌生。而为什么要强调“云原生数据中台才是未来”？分级多域数据治理的刚需、云原生技术降本增效的天然特征、国内基础设施自主可控的要求……都将数据中台推向云原生。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，奇点云将云原生数据中台的技术要素归纳为6点：CI\/CD（持续集成持续交付）、容器化、对象体系、存储计算分离、跨云多域数据治理和元数据管理。这其中，对象体系、跨平台、自主可控是全新迸发出的几个要素——奇点云的云原生数据中台DataSimba，实现跨云的多workspace管理，以帮助客户的数据和应用跨云治理和迁移。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"CI\/CD（持续集成持续交付）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CI\/CD的本质是提高开发和部署效率。在业务量巨大的情况下，大数据和云的运维人力成本极高。因此需要使用大量的自动化工具和大数据预测算法进行自动化运维。通过版本管理系统和DevOps基础设施，实现自动化测试和持续集成。一个典型流程是，程序员提交代码到特定的tag，触发测试接口自动化测试脚本执行并发送报告。由此实现测试、发布和部署自动化。在此基础上构建特定的数据环境，对重要接口和链路进行自动化检测。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"容器化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容器化本质上是一种虚拟化技术，一台主机可虚拟出上千个容器。单个容器的启动时间更快，占用空间更小，而且可以根据实际应用的大小来弹性分配资源，无需额外采购服务器，加快研发速度。使用容器编排基础设施，对服务和作业进行治理，根除版本地狱，大幅度提高运维和集成效率。容器化编排与CI\/CD是相互结合的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在数据中台领域，往往几十台机器、上百个进程同时运行，且在这些进程中不仅要运行本身的程序，也要运行客户的程序。因此，底层微服务的进程繁多。基于安全合规要求，客户之间的程序需要保持分隔。因此，数据中台对于容器化的要求高于其他基于云原生的应用。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"对象体系"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根据现有业务抽象出核心对象，以标准Restful风格提供API服务，解耦核心对象与业务层服务，以应对不同环境、不同业务场景的需求。这一系列正交的核心对象就构成了平台对象体系，上层业务可在此基础上构建应用，高效演进。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对象体系的API应该是优雅且向前兼容的，一旦发布，很难改变。例如，在WIN32研发时，出现某个单词错误，几十年后都无法修改。因此，需要把对象体系设计得极为详尽和准确。奇点云云原生数据中台作为开放式平台，其上的API提供了一套对象，如：项目、作业、数据、源数据、账号等，具有集中数据接口。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"存储计算分离"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于云具有分布式特点，在云上无法天然将数据存储在ECS中。因此必须将关键数据、状态型数据存储在对象存储中。大量私有化组件都需要被改写。如果把Hadoop、Spark等常规开源大数据引擎直接应用于云主机，海量数据带来的存储成本和吞吐压力，很快会压垮客户。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，必须引入中间缓存实现计算存储分离，将数据存储到对象存储上，同时兼容HDFS协议，能够根据业务需求进行弹性扩容，就能大幅度降低成本，提高集群性能。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"跨云多域数据治理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"云原生数据中台的一大优势在于可以实现跨云多域。例如，客户在AWS上使用数据中台，一旦需要转移到其他平台，云原生数据中台可实现在不修改代码的基础上直接迁移。对于具有多重业务、庞大数据体量的大型企业来说，为避免数据资产被一个平台所绑定，供应商必须呈现多样化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，在客户与一家供应商合作的同时，也需要使用独立的第三方数据中台提供跨云多域的数据治理能力，从而提高基础设施的可控性和安全性。奇点云云原生数据中台DataSimba以「第三方」的角色，为企业解决数据多云并存的跨云多域治理问题。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"元数据管理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于数据量急剧增长，对数据的管理成为一大问题。云原生数据中台的元数据管理功能，对数据的结构、指标、标签、权限、上下游血缘、生产作业等元信息进行规范化管理，建立智能数据治理体系。同时支持数据盘点、安全审计、血缘分析、关键分级等应用，最终实现数据资产化。例如，某顶级品牌商具有73个业务系统，各自存储在不同的数据库和存储介质中，需要将73个系统全部集中在一个数据中台上进行标签打通。在此需求下，数据治理十分重要，核心就是元数据的管理。因此，云原生数据中台必须具备元数据管理功能。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"云原生数据中台能为用户解决什么问题？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具备以上6大技术能力的数据中台是走向云原生后的重要升级。基于这些能力，云原生数据中台究竟能为用户解决哪些问题，带来降本增效？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提高研发效率"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过微服务、CI\/CD、对象体系、DevOps等一系列技术，提升迭代速度，增强在云的复杂环境下的控制、自动化运维控制等。提高代码开发、测试、发布效率，降低迭代成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"降低运维成本"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过上述的技术也可以实现开发及运维高效协同，有效提升对故障的响应速度，实现持续集成和交付。使得快速部署应用成为业务流程和企业竞争力的重要组成部分，降低运维成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"降低存算成本"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大数据基础设施的存储计算成本惊人。存算分离和容器化能够更高效地使用IaaS资源，降低存储成本。存储和计算节点分离后，可以在不对存储进行扩容的情况下快速增加计算资源。另一方面，单个容器的启动时间更快，占用空间更小，而且可以根据实际应用的大小来弹性分配资源，无需额外采购服务器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提高治理效率"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"治理效率不局限于数据治理，也包括微服务治理、系统治理和API治理，需要自动化设计和框架。使用跨云治理、元数据管理等技术，会大幅度提高企业积累数据资产的效率，降低安全风险，提高供应商的多样化。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"作者介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"地雷，奇点云高级技术专家，奇点云数据智能平台DataSimba总负责人，阿里大数据底层核心引擎ODPS初代产品经理。曾支持蚂蚁金服、菜鸟等算法与应用建设。"}]}]}