Hadoop 的“遗产”

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"宣布“Hadoop 已死”已成为一种时尚。但，Hadoop 让企业失去了对大数据的恐惧。Hadoop 反过来又释放出一种创新的良性循环，为我们今天所知的云分析和人工智能服务带来了大量市场。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最近ZDNet 的 Big on Data 专栏撰稿人 Andrew Brust发表的一篇关于"},{"type":"link","attrs":{"href":"https:\/\/www.zdnet.com\/article\/apache-software-foundation-retires-slew-of-hadoop-related-projects\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Hadoop 项目“春季大扫除"}]},{"type":"text","text":"”的文章阅读数爆表，显然触动了人们的神经。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"迄今为止，"},{"type":"link","attrs":{"href":"https:\/\/hadoop.apache.org\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Apache Hadoop"}]},{"type":"text","text":"项目系列不再像十年前那样是大数据的中心，事实上，有关"},{"type":"link","attrs":{"href":"https:\/\/www.infoworld.com\/article\/2922720\/hadoop-demand-falls-as-other-big-data-tech-rises.html?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Hadoop 已死"}]},{"type":"text","text":"的论调已经流传很久，以至于听起来更像是“"},{"type":"link","attrs":{"href":"https:\/\/amp.en.google-info.org\/28819\/1\/generalissimo-francisco-franco-is-still-dead.html?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"弗朗西斯科・弗朗哥最后还是死了"}]},{"type":"text","text":"”这则老标语的最新版本。（"},{"type":"text","marks":[{"type":"strong"}],"text":"译注"},{"type":"text","text":"：弗朗西斯科・弗朗哥，Francisco Franco，西班牙国家元首、最高统帅、大独裁者，1975 年 9 月 27 日被宣告政治死亡，1975 年 11 月 20 日，死于帕金森症。）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你想进一步了解情况，看看招聘信息就知道了。最近，"},{"type":"link","attrs":{"href":"https:\/\/terenceshin.medium.com\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Terence Shin"}]},{"type":"text","text":"在上个月发布的"},{"type":"link","attrs":{"href":"https:\/\/towardsdatascience.com\/the-most-in-demand-skills-for-data-scientists-in-2021-4b2a808f4005?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"一项调查"}]},{"type":"text","text":"（如下图所示），通过网络搜索整理了超过 15000 个数据科学家的工作清单，清单显示，雇主对 Hadoop 技能的需求正在急剧下降，C++、Hive 和一些遗留的专用语言也在其中。顺便说一下，对了，Spark 和 Java 也同样在清单中。如果向数据工程师提出同样的问题，结果是否会有所不同？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解决方案“Hadoop”被认为是 2014 年的事情。对于大数据来说，这个世界也在不断发展。大数据之所以被贴上这样的标签，是因为在那个时候，很少有人会对多达 TB 或 PB 级的数据进行梳理，对非关系数据进行分析的能力也有限。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如今，多模型数据库已经变得越来越普遍，而大多数关系数据仓库（Data Warehouse）也增加了解析 JSON 数据和叠加图形数据视图的功能。在云存储中直接查询数据和 \/ 或从数据仓库进行联合查询的功能也已变得司空见惯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正如Andrew 所说，“春季大扫除”旨在“清除蜘蛛网”。和传统的观念相反，Hadoop 并没有死亡。在"},{"type":"link","attrs":{"href":"https:\/\/www.cloudera.com\/products\/cloudera-data-platform.html?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Cloudera Data Platform"}]},{"type":"text","text":"（CDP）中，Hadoop 生态系统的一些核心项目仍在继续，这一产品非常有活力。因为幸存下来的是 CDP 之前还没有出现的打包平台，所以我们不再称之为 Hadoop 了。现在，动物园里的动物都安全地关进了笼子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e6\/e67f01822ed28a4d78d6988ca685a9cb.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用几个甚至更多的独立开源项目来组建自己的集群的想法已经过时了。既然有其他可供选择的方案（我们不仅仅是在讨论 CDP），为什么还要浪费时间手工实现 Apache"},{"type":"link","attrs":{"href":"https:\/\/hadoop.apache.org\/docs\/current\/hadoop-mapreduce-client\/hadoop-mapreduce-client-core\/MapReduceTutorial.html?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"MapReduce"}]},{"type":"text","text":"、"},{"type":"link","attrs":{"href":"https:\/\/hive.apache.org\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Hive"}]},{"type":"text","text":"、"},{"type":"link","attrs":{"href":"https:\/\/ranger.apache.org\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Ranger"}]},{"type":"text","text":"或者"},{"type":"link","attrs":{"href":"https:\/\/atlas.apache.org\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Atlas"}]},{"type":"text","text":"呢？至少在过去 30年里，这一直是数据库领域的常态；当你购买 Oracle 时，你不必分别安装查询优化器和存储引擎。为什么我们用来调用Hadoop的数据平台会有所不同呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到 2020 年，对于新的项目，你的组织可能计划实施云服务，而非安装打包的软件。尽管推动云计算最初是为了转移成本，但现在更多是关于公共控制平面下的操作简化和敏捷性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现在，有多种方法可以分析过去称为“"},{"type":"link","attrs":{"href":"https:\/\/bigdataldn.com\/intelligence\/big-data-the-3-vs-explained\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"三个 V"}]},{"type":"text","text":"”（"},{"type":"text","marks":[{"type":"strong"}],"text":"V"},{"type":"text","text":"olum、"},{"type":"text","marks":[{"type":"strong"}],"text":"V"},{"type":"text","text":"elocity、"},{"type":"text","marks":[{"type":"strong"}],"text":"V"},{"type":"text","text":"ariety，即体积、速度、种类）的数据。如今，你可以随时访问位于云对象存储中的数据，也就是事实上的数据湖（Data Lake）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过使用"},{"type":"link","attrs":{"href":"https:\/\/aws.amazon.com\/athena\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Amazon Athena"}]},{"type":"text","text":"等服务进行特别查询，可以实现这一点；在大多数云数据仓库服务中利用可选的联合查询功能；使用"},{"type":"link","attrs":{"href":"https:\/\/databricks.com\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Databricks"}]},{"type":"text","text":"等专用服务或"},{"type":"link","attrs":{"href":"https:\/\/azure.microsoft.com\/en-us\/services\/synapse-analytics\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Azure Synapse Analytics"}]},{"type":"text","text":"等云数据仓库服务对数据运行"},{"type":"link","attrs":{"href":"https:\/\/spark.apache.org\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Spark"}]},{"type":"text","text":"。由于数据仓库和数据湖之间的界限越来越模糊，现在很多人采用了模糊术语"},{"type":"link","attrs":{"href":"https:\/\/databricks.com\/blog\/2020\/01\/30\/what-is-a-data-lakehouse.html?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"数据湖屋"}]},{"type":"text","text":"（Data Dakehouses），或者整合跨数据仓库和数据湖的访问，或者把数据湖变成 80% 的数据仓库。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且我们甚至还没有涉及到人工智能和机器学习。就像早期的 Hadoop 只属于数据科学家（在数据工程师的帮助下）一样，最初机器学习和更广泛的人工智能也是如此。如今，数据科学家拥有许多工具和框架来管理他们所创建模型的生命周期。对于公民数据科学家而言，"},{"type":"link","attrs":{"href":"https:\/\/cloud.google.com\/automl?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"AutoML 服务"}]},{"type":"text","text":"使构建机器学习模型变得触手可及，而云数据仓库正在增加它自己的"},{"type":"link","attrs":{"href":"https:\/\/cloud.google.com\/bigquery-ml\/docs\/introduction?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"预打包机器学习模型"}]},{"type":"text","text":"，可以"},{"type":"link","attrs":{"href":"https:\/\/aws.amazon.com\/redshift\/features\/redshift-ml\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"通过 SQL 命令"}]},{"type":"text","text":"来触发。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可能人们很容易忘记，仅仅在十年前，这一切似乎还是不可能发生的事情。谷歌的创新研究推动了这一领域的发展。借助"},{"type":"link","attrs":{"href":"https:\/\/research.google.com\/archive\/gfs-sosp2003.pdf?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"谷歌文件系统"}]},{"type":"text","text":"，这家互联网巨头设计出了一个仅限于附加的文件系统，利用廉价磁盘的优势，突破了传统存储网络的限制。通过"},{"type":"link","attrs":{"href":"https:\/\/research.google.com\/archive\/mapreduce-osdi04.pdf?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"MapReduce"}]},{"type":"text","text":"，谷歌破解了这一密码，它在商品硬件上实现了几乎线性的可扩展性。在当时广泛采用的扩展性 SMP 架构中，很难做到这一点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"谷歌发表了这些论文，这对"},{"type":"link","attrs":{"href":"https:\/\/www.linkedin.com\/in\/cutting\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Doug Cutting"}]},{"type":"text","text":"和"},{"type":"link","attrs":{"href":"https:\/\/web.eecs.umich.edu\/~michjc\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Mike Cafarella"}]},{"type":"text","text":"来说是件好事，他们当时正在开发一个能够索引至少 10 亿页的"},{"type":"link","attrs":{"href":"https:\/\/www.geeksforgeeks.org\/hadoop-history-or-evolution\/?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"搜索引擎"}]},{"type":"text","text":"项目，发现了一条开源之路，可以大幅降低实现这样一个系统的成本。后来，社区的其他成员接过了 Cutting 和 Cafarella 的工作，例如，Facebook 开发了"},{"type":"link","attrs":{"href":"https:\/\/en.wikipedia.org\/wiki\/Apache_Hive?fileGuid=f7uaCPa9UPkktDUb","title":"","type":null},"content":[{"type":"text","text":"Hive"}]},{"type":"text","text":"，它提供了一种类似于 SQL 的编程语言，用来在 PB 级别梳理 PB 级各种数据集。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如今，随着经典 Hadoop 项目的采用率的下降，人们很容易忘记，Hadoop 项目的发现带来了一个良性循环，创新吞噬了年轻一代。在 Hadoop 出现的时候，数据就变得如此庞大，以至于我们不得不对数据进行计算。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着云原生架构的出现，廉价、大量的带宽使之成为可能：再一次将更多的存储层、计算层和数据层分开。并不是说这两种方法都是正确或错误的，而是说它们适用于当时设计时就已存在的技术。这就是科技创新的周期性本质。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从 Hadoop 学到的经验突破了规模化处理的限制，从而促生了一个循环，很多旧的假设，比如 GPU 严格用于图形处理，都被抛在了一边。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop 的“遗产”不仅在于它所催生的创新良性循环，还在于它使企业能够克服对数据处理的恐惧，而且还是海量数据。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"作者介绍："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tony Baer，dbInsight LLC 的负责人，负责大数据和数据管理以及一些系统工程。领导 Ovum 的大数据研究领域，从事该行业的 25 年中，研究了数据集成、软件和数据架构、中间件和应用开发等问题。拥有多学科背景，涉及企业软件的不同层次。与他人合作出版了一些关于 Java 和 .NET 框架的早期书籍，并在多家杂志社发表过许多文章。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文链接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/www.zdnet.com\/article\/hadoops-legacy-no-more-fear-of-data\/#ftag=RSSbaffb68"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}