实时计算的业务劣势、思维误区和改进之道

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#1E4E79","name":"user"}}],"text":"技术优势如何变成业务劣势？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“实时”一词过于笼统，我们不妨通过“时效性”来进行量化："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"时效性为“天”级别以上的，从业务习惯来讲我们称之为“离线计算”；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"时效性为“小时”级别的，我们称之为“准实时计算”；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"时效性为“分”、“秒”级别的，我们称之为“实时计算”；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“时效性”常常和“时间精度”混淆。其实两者并没有直接联系："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"时效性为“天”的“离线计算”，同样可以提供时间精度为“秒”的计算，只不过上一天数据的计算结果今天才会输出；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"时效性为“秒”的“实时计算”，同样可以提供时间精度为“天”的计算，当天的计算结果当天就输出并按秒更新，只是在一天结束前，计算结果都不完整；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"显然，高时效性的计算技术在应用场景上有极大优势：时效性高的计算技术可以用于时效性要求低的场景，但是时效性低的计算技术无论如何也满足不了时效性高的场景的需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实时计算的特性为高时效性，这会在数据业务上会产生什么影响？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有点反直觉，会有一个偏负面的影响："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实时计算的高时效性特性，令其在数据业务创新和推广的生命周期中，处于下游、末端的地位。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其实这就是当前的实时计算业务现状。我们从逻辑上也不难推演："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"只有在需要关注和利用当前“分钟”和“秒”级的高时效性信息，你才需要进行实时计算。注意这里不要混淆“时效性”和“时间精度”：如果你需要 “分钟”精度的历史数据，你并不需要进行实时计算。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"需要使用高时效性数据的数据业务，只有投产阶段才需要实时计算。如果把数据业务的创新和推广流程分为探索、调研、实验、投产这四个阶段，那么在前三个阶段，可能需要高时间精度的数据，但基本不需要高时效性的数据：数据业务的探索、调研、实验等环节耗时都以天为单位计量，“分钟”和“秒”级时效的数据自然没有太大意义和帮助。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在数据业务中，探索、调研、实验这三个环节的业务附加值很高，但实时计算却无法参与，只有在最后的投产环节，才出现其身影。尽管实时计算在技术上不可或缺，然而处于业务流程的下游和末端，在创新和推广上奉献价值反而很小。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"结合当前数据业务中研发和工程的分工，实时计算的高时效性这一特性，更是为其相关工作的开展制造了一个困局。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顺便一提。回到一开始的量化，所谓“实时计算”和“离线计算”，从数据业务角度看，可能更多是时效性高低的区别。然而，我们使用“实时计算”“离线计算”这样的二元化定义，可能让自己潜意识地跳进认知陷阱，还导致了一些思维误区。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#1E4E79","name":"user"}}],"text":"我们常抱有什么样的思维误区？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"普遍的思维误区主要有两点："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"我们潜意识中认为，数据时效性越高，价值自然越高，从而忽视了业务规律，进而导致我们没有聚焦到与实时计算更契合更有价值的业务上。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我们潜意识中认为，实时计算的推进的关键和瓶颈在于技术上。只要技术进步，业务自然会出现，使得我们习惯被动等待需求，加上实时计算的特性，让我们更容易与业务脱钩。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更进一步分析，这些思维误区，和目前数据业务的分工模式也有着较大的关联。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当前，数据业务通常会分工为研发和工程两种。简单来说，就是数据科学家和数据工程师两种角色。笔者所在的公司和团队即是如此。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"早期，这两种角色通常是一个人兼职的。后来，因为技术门槛和开发耗时等原因，部分人员就分化出来专职从事数据工程师这一角色，专门负责数据开发、业务实现和进一步的平台化等工作，这样效率更高，更有规模效应。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实时计算自然更多是数据工程师的范畴。然而，在业务方面，这种分工模式会带来一些深层次的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，如前文所言，因为实时计算的高时效性，其在数据业务创新和开发流程中，参与程度会更加小。按照当前数据业务分工模式，数据工程师只专注于业务实现上的话，业务空间自然会压缩得更小。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，基于技术进步和数据工程师二次开发的框架，实时计算业务开发的成本和门槛已经大大降低。将来，数据科学家有更好的条件进行业务实现，这也会进一步压缩数据工程师的业务空间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，实时计算的高时效性特性，其业务的个性化特性会更加明显。在当前分工模式下，实时计算的落地和推广实际上依赖和受制于数据科学家的需求，这会带来一些负面影响："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要高时效数据的业务的个性化明显，数据科学家在探索、调研和实验阶段可能难以利用现有平台，而是手动实现数据流程，导致路径依赖，对实时计算方案感知度低；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据工程师主要把数据科学家作为业务推进的目标受众，因为数据科学家对实时计算的感知度低，不能给数据工程师提供有效的反馈，导致实时计算业务的推广和落地受阻；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些因素相互影响和交织，形成实时计算业务推广的瓶颈和阻碍，这是目前实时计算面临的困境。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在实时计算这一领域，专注业务实现、被动等待需求，给业务创新和推广带来的瓶颈和阻碍会越来越明显，不能长久。现在已经不是能不能实现的问题，我们要更多地考虑实现什么的问题。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#1E4E79","name":"user"}}],"text":"业务推进策略该如何调整？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实时计算面临的困境，是由数据业务的分工模式和实时计算的高时效性特性共振而来。所以，若要摆脱这个困境，我们首先可以尝试突破这种分工模式，直接面向产品，主动寻找可进入的业务："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接触业务，主动发现问题和推动问题解决。同时注意“需要更快的马”的典型伪需求；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在数据业务创新和业务的流程中，以合理分工的方式，参与更上游的环节；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以面向产品的角度，进行业务的落地和推广工作；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"举个例子，原本实时计算推进工作是主要面向数据科学家，按照其拆解的需求，完成实时计算的业务实现。现在变为，数据工程师和数据科学家一起，直接面向产品的完整需求，通过合理分工参与数据业务上游部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这样做的好处，一是两者都可以弥补在业务和技术知识上的缺失和差距，二是实时计算的推广工作可以转向以产品为目标，有更好的反馈和成效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后，我们应该为实时计算寻找有价值的业务点和进入方式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实时计算应该进入的头部业务，应该具有以下两个特性："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"高时效性的数据可以产生更大价值，或者必须使用高时效性数据；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"业务产出尽量可以客观量化，需要人主观评价的成分少；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如，个性化推荐就是一个适合实时计算进入的业务，进入角度可以是全业务过程的系统化、平台化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，个性化推荐直接满足上述两个特性："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"许多对照实验证明实时推荐效果更好，并且一些推荐业务必须基于实时数据；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"一般来说，命中率和使用率就足以衡量业务的价值。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"严格来说，个性化推荐并不是特别“新颖”的着力点。直观上，谈起实时计算，个性化推荐大部分人第一时间能想到的应用点。不过，这不意味着腾挪空间就少。"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"个性化推荐的形式非常多样，如商品推荐、好友推荐、助战推荐，等等。要把这些统合在一起，可做的工作很多；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"个性化推荐不仅仅只是推荐本身，还涉及概率控制、舆论控制、升降档、兜底等因素要考虑，在全系统上也有很多可做文章之处；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，数据预警（异常用户、异常交易等），数据接口（实时大屏等）等，也是适合实时计算的头部业务，分析过程类似，关键在于进入的角度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}