vesoft 于新林：开源，让 Nebula Graph 图数据库更好

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"万物互联让数据呈指数级增长，在面对一些海量高并发的场景时，传统的关系型数据库已不适用。而随着人们对数据处理需求的快速增长，加之图数据库的技术也逐渐成熟，人们发现用树和拓扑图更容易表达这充满关系的现实世界。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Nebula Graph 作为一款开源的分布式图数据库，定位为大规模的属性图。以处理大规模海量数据为设计目标，是世界上唯一能够容纳千亿个顶点和万亿条边，并提供毫秒级查询延时的图数据库解决方案。接下来，就让我们跟 vesoft 的 CTO 于新林一起来聊一聊 Nebula Graph。视频下方也有文字版。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"embedcomp","attrs":{"type":"video","data":{"id":"450455","name":"对话vesoft于新林","poster":"","url":"https:\/\/media001.geekbang.org\/8684473dfdf943848f555a9fae60e4be\/d388e25ff6f745e29817cd6abd6eb73b-86a3651cc50e0adc54f801c2de18f55f-sd.m3u8"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":12}}],"text":"vesoft CTO 于新林"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：请于老师从自己的工作经历方面做一下个人介绍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林："},{"type":"text","text":"我上个工作是在阿里，大约待了十四年，前十年在支付宝做支付宝的首席架构师，后四年在阿里云的IoT 负责 IoT 的基础 PaaS。2021年5、6月份加入欧若数网，现在在做图数据领域相关的工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：刚开始，公司为什么想做Nebula Graph 这款产品呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林"},{"type":"text","text":"：公司CEO在图数据库这一领域的沉淀和积累较深。他之前在Facebook，再到后来的蚂蚁一直是做图数据库的。因为CEO有比较深的图数据库技术背景，刚好也在这一领域涉猎多年，所以就出来做图数据库相关的技术，然后就做了Nebula Graph这款产品。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：据了解，目前Nebula Graph是一款开源的分布式图数据库，为什么刚开始就选择了开源？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林"},{"type":"text","text":"：第一，我们公司成立在2018年10月份左右，那个时候感觉业界整个开源的势头越来越强，这是一个大的趋势；第二，因为我们公司是做图数据库的，图数据库是底层的infrastructure 的产品，这种产品只要是商业模式找到了，其实开源是一个比较好的手段，它可以快速地让你接触到你的客户、接触到开发者，可以让客户和开发者给你提需求、提Bug，甚至贡献代码，这样更适合产品的发展，进行快速迭代，所以当时我们选择了开源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：作为公司的CTO，您当初在Nebula Graph的架构设计上有哪些考虑？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林"},{"type":"text","text":"：对于架构设计，首先对于一个产品我们有一些大的原则，这些原则包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一，我们选择开源；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二，因为数据量越来越大，我们要支持海量、大并发的业务，它肯定是一个分布式的（架构）；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三，我们希望有一个相对比较灵活的架构，所以采用了计算层和存储层分离的架构；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四，现在基于K8s的云原生越来越火，并且它也是一个必然的趋势，我们现在支持云原生，全面拥抱云原生；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第五，我们之前主要的精力是放在OLTP上，但是图的计算和分析，一定也是未来的趋势，所以说我们也会做HTAP的融合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上是我们基于这些大的设计原则来做的Nebula Graph这款产品。另外图数据库本身的发展，要依赖于整个的生态，还包括周边的一系列的配套的工具、产品，我们也希望和社区一起基于Nebula Graph打造一个图数据库相关领域的生态，类似于周边的一些SDK、工具、平台，让这个生态能够更加繁荣。这是一些基本的思考。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：Nebula Graph 研发过程中的核心技术有哪些？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林："},{"type":"text","text":"因为我们是一个分布式的、存储和计算分离的框架。第一，图数据库领域本身就有一定难度。一般像底层基础设施难的是操作系统，再上面是数据库，所以核心技术就包括怎么解决分布式的问题，一旦数据存在不同的物理机上，一定涉及到分布式事务的一致性问题；第二，怎么去处理这种高并发、实时海量的数据的问题。第三，因为数据库本身要稳定，又稳又快，而且要安全，所以怎么样去通过各种手段来保证数据库的稳定，给客户提供持续可靠的服务。刚说到数据库本身要速度快，所以要怎么优化？因此在查询层类似于优化器、调度器、解析器，都要去做比较深入细致的工作，才能真的让性能提升上来。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很多客户对安全性要求也高，包括用户的权限、网络通信、甚至底层数据加密的存储等一系列的问题。光从分布式的计算存储分离的数据库来看，技术的挑战点就很大、很多，以上是我认为一些比较难的点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：对比市面上其他的图数据库，Nebula Graph 有哪些优势？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林："},{"type":"text","text":"第一，因为Nebula Graph 是一个开源的、完全自研的图数据库，所以我们的可控性就高了很多，当然自研包括我们跟社区共建产品；第二，因为可控性很高（自研），我们可以做很多极致的优化。我们最大的场景可以到千亿点和万亿边，现在有的客户已经达到这种规模，在这种海量高并发场景下采用（Nebula Graph）很适合。另外在这种高并发下，客户对于响应时间的要求比较严格。因为Nebula Graph是一个OLTP（联机事务数据库），在性能上，在海量高并发下可以实现毫秒级返回，这些都是我们目前感觉做的相对比较好的点，当然也有一些不足我们要持续的优化和完善。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：在Nebula Graph 研发过程中，你们面临的比较大的挑战有哪些？针对这些挑战，你们又是如何解决的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林"},{"type":"text","text":"：研发当中肯定会面临很多的挑战。拿稳定性来讲，客户一般对于数据库产品的要求比较高。要保证数据库的稳定其实有很多方面，包括兼容怎么做、内存怎么处理，在高并发下服务会不会Crash 掉。对于一些网络和故障，分布式处理怎么样？为了应对稳定性这一问题，我们做了很多的工作，包括混沌测试、压测、收集客户的场景的特制案例等，在这些方面都做了很多的优化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"还有一些自身的挑战，比如因为（Nebula Graph）本身是一个分布式的数据库，分布式数据库一定会涉及到最终的ACID，这是一个分布式事务的问题，怎么样保证事务的最终一致性，其实挑战是比较大的。再一个就是，在这种集群下，它怎么样保证集群之间数据的同步、集群出故障如何快速地处理故障切换服务等等一系列的挑战。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其实再细化一些，例如在查询层优化器做的好不好，其实也是一个很大的挑战。一条语句，经过语法、语义解析之后产生的这种执行计划，还涉及到索引选择一系列的问题，优化的策略也包括基于规则的优化、基于成本的优化，甚至后续出现类似于智能的优化。另外调度器的挑战也很大。例如，执行一条语句，其实可以考虑在一台机器上执行，但会有一个问题，这台机器的计算资源是有限的，当其他机器空闲的时候，很难充分利用集群的优势，这时候可以考虑通过把这台机器的任务调度到其他机器上，能尽快的并发执行，这是关于调度器和执行器的优化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于存储层，一个是刚刚说过的分布式事物，另外一个是可以考虑大的内存，这样可以把整个图数据库的拓扑图搬到内存里去，一旦搬到内存里去，就涉及到内存和磁盘的一致性的问题，当内存空间不够的时候，又怎么和磁盘进行交换？另外，图一定会涉及到多跳查询，每次出现多跳查询就会涉及到与集群中其他进程的交互，怎么通过一些内存和存储的优化去降低这种频繁的多跳交互带来的网络开销或者性能降低？很多挑战是需要我们去解决的。也希望社区的小伙伴有时间多给我们贡献，或者加入我们一起解决问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：目前，Nebula Graph 适用的业务场景有哪些？于老师可以举一些比较典型的例子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林"},{"type":"text","text":"：典型的应用场景像金融欺诈的检测，类似于反洗钱；公安的一些伴随、跟随，还有犯罪团伙的发现；国安和公安这类业务场景，通过一些关系找到一些犯罪团伙；像知识图谱；像平时我们的开发同学遇到的数据血缘关系、系统的链路调度关系；还有比较陌生的一些，像芯片设计的EDA软件等都会用到图数据库。我觉得图领域在将来的场景会越来越丰富、越来越渗透到整个社会发展的各行各业里面去，其实它的空间还是蛮大的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：Nebula Graph 的下一步打算是什么？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林"},{"type":"text","text":"：第一步，先把Nebula Graph上云，提供DBaaS服务，类似于PaaS服务，客户不用去考虑怎么样去搭建、部署、运维一个数据库，只要使用数据库的服务就好了。我们现在已经在和一些云厂商合作，把我们图数据库搬到云上来。第二步，我们其实后续也在加大AP这个领域的投入，现在已经有AP相关的服务，但是跟我们理想中的还有差距，之后通过加大对AP领域的投入，真正把TP和AP融合起来，做到HTAP。第三步，我们后续也是希望能把图的计算和图的分析以SaaS化的服务形式搬上云，这样对一些客户来讲，就不需要去搭建一套图计算和图分析的平台，只需要用SaaS服务把数据上传，进行图计算和图分析便可得到结果，这样可实现按需付费，按照计算单元和时间付费，这是我们接下来的一些大的思考和规划。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：研发过程中，有没有令您印象深刻的事？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"于新林"},{"type":"text","text":"：令我印象最深的一件事是我们有一个版本发布以后，客户说他们性能急剧下降，但是我们自己都测过，根本没出现过这个问题。我们也很诧异为什么会出现这种情况，于是立刻联系客户，希望在第一时间去解决客户的问题，并与客户迅速建立电话会议。经过排查后发现，客户这种场景是我们以前案例里没有的。其实这个也是客户和我们一起共建社区的一个过程，通过这种案例，也可以提高我们产品的健壮性和稳定性。在这样紧急的情况下，经过大家一起共建，很快就发现了问题，并解决了问题。当天就打了一个Patch 包解决了客户的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"诸如此类的事情还有很多，但那次我印象特别深，因为很快地响应客户并快速地解决了客户的问题，同时感谢客户给我们贡献我们以前没有想到或者遇到的场景。这是令我印象比较深的一件事情。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]}]}