数据分析师成长体系漫谈-数仓模型设计

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"序"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 看到标题，可能很多小伙伴都会疑惑，为什么笔者把数仓模型设计也纳入了数据分析师的成长体系之中，因为可能大多数公司会有单独的数仓部门，分析师只需要通过数仓提供的库表进行统计分析即可。不过，你是否遇到过以下的问题："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 1、从原始的日志层提取数据，需要写复杂的逻辑，执行的时候也需要消耗很多的服务型性能去计算；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 2、数仓进行初步分分层构建，但是构建的中间层并不满足业务分析需求，需要提需求增加统计指标或者自己写sql从底层捞；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从这两个常见的问题中可以发现，虽然有专门的数据仓库部门，可以保证日志数据准确、完整的入库，也会制定规范，并且构建一些基础的中间层，但问题的核心是在于数仓与业务方的“距离”，并且对于业务的理解和需求，分析师理解的相对更加透彻。所以，在一些企业中，数仓建设上是存在一些分工的，比如数仓的同学保证离线、实时数据流的稳定和准确入库，分析师负责构建中间层和应用层来满足业务方需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 基于以上的描述，笔者从自身的工作经验中总结出一些作为分析师构建数仓模型所需要了解的知识点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"size","attrs":{"size":16}},{"type":"strong"}],"text":"starting~~"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"数据仓库"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 数据仓库（Data Warehouse ，DW），引用百度百科的解释，是为企业所有级别的策略制定过程，提供所有类型数据支持的战略集合。它是单个数据存储，出于分析性报告和决策支持目的而创建。 为需要业务智能的企业，提供指导业务流程改进、监视时间、成本、质量以及控制。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四个主要特点"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 面向主题"},{"type":"text","text":"：与传统的面向事务的数据库不同，数据仓库具有较高的抽象性，从用户的需求出发，将不同平台的数据按照特定的主题进行划分和整合。主题可以理解为研究的对象，通过面向主题的组织数据，可以完整、统一的刻画出研究对象的所有业务数据。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 数据集成"},{"type":"text","text":"：数据仓库的数据大多来自于传统数据库的，但是数据通常不是直接入库，而且需要先进行数据清洗。因为事务性数据库中的数据通常会存在脏数据，这些脏数据会对基于数仓进行的分析和挖掘造成影响。数据集成是数据仓库建设中最重要、复杂的步骤。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 不可更新"},{"type":"text","text":"：数据仓库的数据主要为决策者提供数据依据。决策依据的数据是不允许修改的。所以对于入库到数据仓库的数据，用户只能查询和分析，不可以修改。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 随时间不断变化"},{"type":"text","text":"：数据仓库数据会随时间变化而定期更新，不可更新是针对应用而言，即用户分析处理时不更新数据。简单理解是，传统数据库定期将数据写入到数据仓库中的对应时间分区内，而不修改历史分区的数据。数据仓库中的表通常会有生命周期，对于超过生命周期的分区数据，一般会删除或者改变数据存储结构单独存储。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"与传统数据库区别"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从数据仓库的主要特点可以看出其与传统的数据库差异；"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/04/04fa3477f64636094955c177d23226ce.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"了解数据仓库概念和与传统数据库的区别后，我们来看一下数据仓库内的一些主要概念和组成元素。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"元数据"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通俗来说元数据就是用来描述数据的数据，对数据以及信息资源的描述性信息。元数按照用途分为技术元数据和业务元数据。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/05/05622a545ee44601a9e24871483aa9b0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可能这么描述有些难以理解，举几个例子："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、数据仓库建模工具的元数据：数据定义、数据仓库模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、查询工具：查询定义、数据导出属性、映射"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、ETL：运行的顺序"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、数据源：元系统逻辑模型、物理模型，数据结构定义"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"事实表与维度表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比元数据，事实表和维度表大家应该可能就很熟悉了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事实表（Fact Table）指存储有事实记录的表，比如销售数据、用户浏览数据等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"维度表（Dimension Table）与事实表相对应，是存储维度的属性值，比如地域信息、商品信息等"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事实表与维度表的一些特种："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3d/3da4bd92b57dc9bdd6867c859d0669f5.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"事实表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.可加、半可加、不可加事实  ：可加，例如pv(点击量) ； 半可加，例如数值差额，uv(用户量)；不可加，例如比率；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.NULL值处理：可以存在空值度量，但是外键不能存在空值；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.事实一致性：不同事实表中的事实，应保证事实的定义是相同的，且具有相同的命名，如果不兼容，则须用不同命名方式，便于应用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4.周期事实：某天、某周等周期性，周期内未发生过程，也会有null或0等事实；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5.累计事实，开始与结束之间可预测步骤内的度量事件；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6.无事实的事实：比如：某天学生参加课程的事件；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7.聚集事实：聚合，提高查询性能；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"8.合并事实：同粒度表进行合并；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"维度表"},{"type":"text","text":"，可以跟事实表做关联，相当于是将事实表中经常重复的数据抽取、规范出来用一张表管理，常见的有日期、地区表等，所以维度表的变化通常不会太大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.维度下钻：例子：如果我知道上海市的数据，但是我想查看各区的数据，维度级别变细，称为下钻，相反称为上卷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.退化维度：维度除了主键外无其他内容，例如订单号，发票号"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.非规范化扁平维度：将多张范式表 合并成统一的扁平的非规范化的维度，能够实现维度建模的双重目标：简化与速度，比如将一张商品表，和一张商品分类信息表合并成一张表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4.维度层次：比如 年月日， 国家 省份 城市 等"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"数仓数据架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通常数据仓库的数据架构大概可以分为数据采集、数据整合、数据应用，此处以阿里DataWorks数仓架构为例。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7c/7c74564c1274cc18f0c2081afabb5cdd.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"数据采集层"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 数据采集层的任务就是把数据从各种数据源中采集和存储到数据库上，期间有可能会做一些ETL（抽取extra，转化transfer，装载load ）操作。数据源种类可以有多种："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 日志：所占份额最大，存储在备份服务器上 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 业务数据库：如Mysql、Oracle "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 来自HTTP/FTP的数据：合作伙伴提供的接口 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 其他数据源：如Excel等需要手工录入的数据"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"数据计算"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前面使用Hive、MR、Spark、SparkSQL分析和计算的结果，还是在HDFS上，但大多业务和应用不可能直接从HDFS上获取数据，那么就需要一个数据共享的地方，使得各业务和产品能方便的获取数据。 这里的数据共享，其实指的是前面数据分析与计算后的结果存放的地方，其实就是关系型数据库和NOSQL数据库。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​"},{"type":"text","marks":[{"type":"strong"}],"text":"数据应用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 报表：报表所使用的数据，一般也是已经统计汇总好的，存放于数据应用层。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 接口：接口的数据都是直接查询数据应用层即可得到。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 即席查询：即席查询通常是现有的报表和数据应用层的数据并不能满足需求，需要从数据存储层直接查询。一般都是通过直接操作SQL得到。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"数仓分层"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"敲黑板，数仓分层是本文的重点知识之一。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 数据仓库建设过程中一个重要的概念-数仓分层。分层是数据仓库解决方案中，数据架构设计的一种数据逻辑结构 ，通过分层理念建立的数据仓库，它的可扩展性非常好，这样设计出来的模型架构，可以任意地增减、替换数据仓库中的各个组成部分。通俗来说就是将不同整合粒度的数据划分到相应的层级中。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/82/8277cb7b2f49063d0cd6a45de950f1c6.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据仓库分层的好处显而易见："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"用空间换时间"},{"type":"text","text":"：通过数据预处理提高效率，通过大量的预处理可以提升应用系统的用户体验（效率），相应的数据仓库会存储大量冗余的数据."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"增强可扩展性"},{"type":"text","text":"：方便以后业务的变更。如果不分层的话，当源业务系统的业务规则发生变化整个数据仓库需要重建，这样将会影响整个数据清洗过程，工作量巨大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​ "},{"type":"text","marks":[{"type":"strong"}],"text":"简化清洗过程"},{"type":"text","text":"：通过分层管理来实现分步完成工作，简化数据清洗的过程，使每一层处理逻辑变得更简单。因为把原来一步的工作分到了多个步骤去完成，相当于把一个复杂的工作拆成了多个简单的工作，把一个大的黑盒变成了一个白盒，每一层的处理逻辑都相对简单和容易理解，这样我们比较容易保证每一个步骤的正确性，当数据发生错误的时候，往往我们只需要局部调整某个步骤即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同样以阿里数据仓库分层为例"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1a/1a424fdca834dffda0fb53ee83ed1be0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"建模方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"敲黑板，第二个重点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此处建模主要是指构建数据仓库模型并非指统计算法模型。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 数据仓库建模的方法主要分为范式建模和维度建模，两者思路是完全相反，根据其定义，在数据仓库中进行建模维度建模更适用。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/66/6673fab42564a8b728f283250b03f59c.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 说到维度建模，主流的两种方式是星型和雪花型。其中雪花型可以理解为是对星型模型的一种拓展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 星型模型：星型架构是一种非正规化的结构，多维数据集的每一个维度都直接与事实表相连接，不存在渐变维度，所以数据有一定的冗余。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4a/4ab98fe4c5ef265a3bed2aaadc7b993b.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 雪花型模型：当有一个或多个维表没有直接连接到事实表上，而是通过其他维表连接到事实表上时，其图解就像多个雪花连接在一起，故称雪花模型。雪花模型是对星型模型的扩展"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/78/7879117a429adc2f8d1a3230b4ad6fc9.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"两种模型思路的优劣也比较明显："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/02/02fe411984de3c574985cd9f61cdb86c.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"数据域"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 不敲了，第三个重点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 数据仓库存放了企业多个业务或者产品的数据，虽然可以根据抽象粒度进行分层，但是我们仍然需要标识业务类型和研究对象，就出现了业务域和主题域。业务域顾名思义就是描述产生此数据的业务，如：短信业务、二手车业务、金融业务、租房业务等，而主题域可以理解为研究的主体，比如：用户、商品、广告等。通过将两者组合来分别数据模型存储的信息内容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"指标与维度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 数据仓库自下而上的构建，就是为了获取各种维度下的指标统计量，来分析数据制定决策。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数仓中指标最基础指标为原子指标，也成为度量，用来表述最基础的信息，比如“元”，“用户数”等，在原子指标的基础上，会根据不同的场景生成派生/衍生指标，规则是原子指标+时间周期+修饰词的组合。比如近7日销售额，近7日北京销售额。当然通过原子指标加减乘除得到的指标也是派生/衍生指标。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 一个数仓模型涵盖内容如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8f/8f1e1f7b934859cc3d691159c7976e78.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"如何做数仓建模？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 以上用较大的篇幅介绍了数据仓库及需要理解的知识点，那么下一步就是如何去做数仓建模。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此处的前提是已经有数仓技术框架和产品，毕竟我们主要是构建数仓模型，而不是搭建完整的数据仓库。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"规范化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、定义数仓模型层级标准，如："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d1/d17d40e62eab8de316baf25a4fa6ce4d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、定义主题域与业务域"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、定义模型命名规范，如：{层级}_{业务域}_{主题域}_{产品}_{事实描述}_{更新周期}，dws_video_user_h5_active_stat_di，表示视频业务用户维度活跃统计轻度汇总层，天级更新；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、定义原子指标、修饰类型、修饰词，原子指标如：PV,UV，修饰类型如日期、行为，修饰词对应就是具体内容如3d表示3天，act表示活跃缩写，3d_act_uv表示近三天活跃用户数；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5、定义指标存储类型，如订单编号varchar，用户数bigint，时间戳varchar等"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"数仓建模流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在数仓建模设计中，大多DWD层模型是非业务需求的，是对业务日志数据的轻度整合，方便向上构建DWS、ADS，或者方便直接即席查询。通常数仓建模流程是："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、对接需求：了解业务方的维度和指标需求，看是否有现成的结果或者可以从已有的中间层加工"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、数据调研：对于数仓中暂时没有的数据，通常是新业务没有同步数据到数仓，或者说没有埋点采集，前者直接同步数据，后者需要新建埋点；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、需求分析评审：评估需求合理性及复杂度，评估需求是否常用，如果不是走临时需求开发流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、数据统计逻辑和物理模型设计"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5、ETL脚本开发"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6、脚本及数据校验"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7、ETL脚本配置调度上线"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一些建议"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 数据模型设计比较考验用户的下钻和上卷逻辑性，从整体去思考层级结构。虽然说数据仓库允许一定成的读冗余，但是多人协同建设的时候尽量不要出现同一个指标出现在多个人构建的模型内（除非是作为下游直接获取），不仅是口径发生变化时候更新成本高，同时如果出现信息不对等，那么就会出现同一个指标不同值的情况。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 建议大家在承接一个业务的时候，优先把这个业务的指标体系构建出来，这样哪怕业务方暂时没有提出相应的需求，你也可以预先进行构建中间层。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"写在最后"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 本文完全是笔者从分析师角度实践的一些经验，数据仓库是一个庞大的体系，文中可能会有一些不准确或者理解不同的描述，请大家指正，也欢迎大家一起交流心得。"}]}]}