數據分析師成長體系漫談-數倉模型設計

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"序"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 看到標題，可能很多小夥伴都會疑惑，爲什麼筆者把數倉模型設計也納入了數據分析師的成長體系之中，因爲可能大多數公司會有單獨的數倉部門，分析師只需要通過數倉提供的庫表進行統計分析即可。不過，你是否遇到過以下的問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 1、從原始的日誌層提取數據，需要寫複雜的邏輯，執行的時候也需要消耗很多的服務型性能去計算；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 2、數倉進行初步分分層構建，但是構建的中間層並不滿足業務分析需求，需要提需求增加統計指標或者自己寫sql從底層撈；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從這兩個常見的問題中可以發現，雖然有專門的數據倉庫部門，可以保證日誌數據準確、完整的入庫，也會制定規範，並且構建一些基礎的中間層，但問題的核心是在於數倉與業務方的“距離”，並且對於業務的理解和需求，分析師理解的相對更加透徹。所以，在一些企業中，數倉建設上是存在一些分工的，比如數倉的同學保證離線、實時數據流的穩定和準確入庫，分析師負責構建中間層和應用層來滿足業務方需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 基於以上的描述，筆者從自身的工作經驗中總結出一些作爲分析師構建數倉模型所需要了解的知識點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"size","attrs":{"size":16}},{"type":"strong"}],"text":"starting~~"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"數據倉庫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據倉庫（Data Warehouse ，DW），引用百度百科的解釋，是爲企業所有級別的策略制定過程，提供所有類型數據支持的戰略集合。它是單個數據存儲，出於分析性報告和決策支持目的而創建。 爲需要業務智能的企業，提供指導業務流程改進、監視時間、成本、質量以及控制。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四個主要特點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 面向主題"},{"type":"text","text":"：與傳統的面向事務的數據庫不同，數據倉庫具有較高的抽象性，從用戶的需求出發，將不同平臺的數據按照特定的主題進行劃分和整合。主題可以理解爲研究的對象，通過面向主題的組織數據，可以完整、統一的刻畫出研究對象的所有業務數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 數據集成"},{"type":"text","text":"：數據倉庫的數據大多來自於傳統數據庫的，但是數據通常不是直接入庫，而且需要先進行數據清洗。因爲事務性數據庫中的數據通常會存在髒數據，這些髒數據會對基於數倉進行的分析和挖掘造成影響。數據集成是數據倉庫建設中最重要、複雜的步驟。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 不可更新"},{"type":"text","text":"：數據倉庫的數據主要爲決策者提供數據依據。決策依據的數據是不允許修改的。所以對於入庫到數據倉庫的數據，用戶只能查詢和分析，不可以修改。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 隨時間不斷變化"},{"type":"text","text":"：數據倉庫數據會隨時間變化而定期更新，不可更新是針對應用而言，即用戶分析處理時不更新數據。簡單理解是，傳統數據庫定期將數據寫入到數據倉庫中的對應時間分區內，而不修改歷史分區的數據。數據倉庫中的表通常會有生命週期，對於超過生命週期的分區數據，一般會刪除或者改變數據存儲結構單獨存儲。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"與傳統數據庫區別"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從數據倉庫的主要特點可以看出其與傳統的數據庫差異；"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/04/04fa3477f64636094955c177d23226ce.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"瞭解數據倉庫概念和與傳統數據庫的區別後，我們來看一下數據倉庫內的一些主要概念和組成元素。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"元數據"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通俗來說元數據就是用來描述數據的數據，對數據以及信息資源的描述性信息。元數按照用途分爲技術元數據和業務元數據。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/05/05622a545ee44601a9e24871483aa9b0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可能這麼描述有些難以理解，舉幾個例子："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、數據倉庫建模工具的元數據：數據定義、數據倉庫模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、查詢工具：查詢定義、數據導出屬性、映射"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、ETL：運行的順序"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、數據源：元系統邏輯模型、物理模型，數據結構定義"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"事實表與維度表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比元數據，事實表和維度表大家應該可能就很熟悉了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事實表（Fact Table）指存儲有事實記錄的表，比如銷售數據、用戶瀏覽數據等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"維度表（Dimension Table）與事實表相對應，是存儲維度的屬性值，比如地域信息、商品信息等"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事實表與維度表的一些特種："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3d/3da4bd92b57dc9bdd6867c859d0669f5.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"事實表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.可加、半可加、不可加事實  ：可加，例如pv(點擊量) ； 半可加，例如數值差額，uv(用戶量)；不可加，例如比率；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.NULL值處理：可以存在空值度量，但是外鍵不能存在空值；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.事實一致性：不同事實表中的事實，應保證事實的定義是相同的，且具有相同的命名，如果不兼容，則須用不同命名方式，便於應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4.週期事實：某天、某周等週期性，週期內未發生過程，也會有null或0等事實；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5.累計事實，開始與結束之間可預測步驟內的度量事件；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6.無事實的事實：比如：某天學生參加課程的事件；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7.聚集事實：聚合，提高查詢性能；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"8.合併事實：同粒度表進行合併；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"維度表"},{"type":"text","text":"，可以跟事實表做關聯，相當於是將事實表中經常重複的數據抽取、規範出來用一張表管理，常見的有日期、地區表等，所以維度表的變化通常不會太大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.維度下鑽：例子：如果我知道上海市的數據，但是我想查看各區的數據，維度級別變細，稱爲下鑽，相反稱爲上卷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.退化維度：維度除了主鍵外無其他內容，例如訂單號，發票號"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.非規範化扁平維度：將多張範式表 合併成統一的扁平的非規範化的維度，能夠實現維度建模的雙重目標：簡化與速度，比如將一張商品表，和一張商品分類信息表合併成一張表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4.維度層次：比如 年月日， 國家 省份 城市 等"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數倉數據架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通常數據倉庫的數據架構大概可以分爲數據採集、數據整合、數據應用，此處以阿里DataWorks數倉架構爲例。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7c/7c74564c1274cc18f0c2081afabb5cdd.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"數據採集層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據採集層的任務就是把數據從各種數據源中採集和存儲到數據庫上，期間有可能會做一些ETL（抽取extra，轉化transfer，裝載load ）操作。數據源種類可以有多種："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 日誌：所佔份額最大，存儲在備份服務器上 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 業務數據庫：如Mysql、Oracle "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 來自HTTP/FTP的數據：合作伙伴提供的接口 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 其他數據源：如Excel等需要手工錄入的數據"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"數據計算"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前面使用Hive、MR、Spark、SparkSQL分析和計算的結果，還是在HDFS上，但大多業務和應用不可能直接從HDFS上獲取數據，那麼就需要一個數據共享的地方，使得各業務和產品能方便的獲取數據。 這裏的數據共享，其實指的是前面數據分析與計算後的結果存放的地方，其實就是關係型數據庫和NOSQL數據庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​"},{"type":"text","marks":[{"type":"strong"}],"text":"數據應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 報表：報表所使用的數據，一般也是已經統計彙總好的，存放於數據應用層。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 接口：接口的數據都是直接查詢數據應用層即可得到。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 即席查詢：即席查詢通常是現有的報表和數據應用層的數據並不能滿足需求，需要從數據存儲層直接查詢。一般都是通過直接操作SQL得到。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數倉分層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"敲黑板，數倉分層是本文的重點知識之一。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據倉庫建設過程中一個重要的概念-數倉分層。分層是數據倉庫解決方案中，數據架構設計的一種數據邏輯結構 ，通過分層理念建立的數據倉庫，它的可擴展性非常好，這樣設計出來的模型架構，可以任意地增減、替換數據倉庫中的各個組成部分。通俗來說就是將不同整合粒度的數據劃分到相應的層級中。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/82/8277cb7b2f49063d0cd6a45de950f1c6.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據倉庫分層的好處顯而易見："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"用空間換時間"},{"type":"text","text":"：通過數據預處理提高效率，通過大量的預處理可以提升應用系統的用戶體驗（效率），相應的數據倉庫會存儲大量冗餘的數據."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"增強可擴展性"},{"type":"text","text":"：方便以後業務的變更。如果不分層的話，當源業務系統的業務規則發生變化整個數據倉庫需要重建，這樣將會影響整個數據清洗過程，工作量巨大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​ "},{"type":"text","marks":[{"type":"strong"}],"text":"簡化清洗過程"},{"type":"text","text":"：通過分層管理來實現分步完成工作，簡化數據清洗的過程，使每一層處理邏輯變得更簡單。因爲把原來一步的工作分到了多個步驟去完成，相當於把一個複雜的工作拆成了多個簡單的工作，把一個大的黑盒變成了一個白盒，每一層的處理邏輯都相對簡單和容易理解，這樣我們比較容易保證每一個步驟的正確性，當數據發生錯誤的時候，往往我們只需要局部調整某個步驟即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同樣以阿里數據倉庫分層爲例"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1a/1a424fdca834dffda0fb53ee83ed1be0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"建模方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"strong"}],"text":"敲黑板，第二個重點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此處建模主要是指構建數據倉庫模型並非指統計算法模型。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據倉庫建模的方法主要分爲範式建模和維度建模，兩者思路是完全相反，根據其定義，在數據倉庫中進行建模維度建模更適用。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/66/6673fab42564a8b728f283250b03f59c.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 說到維度建模，主流的兩種方式是星型和雪花型。其中雪花型可以理解爲是對星型模型的一種拓展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 星型模型：星型架構是一種非正規化的結構，多維數據集的每一個維度都直接與事實表相連接，不存在漸變維度，所以數據有一定的冗餘。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4a/4ab98fe4c5ef265a3bed2aaadc7b993b.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 雪花型模型：當有一個或多個維表沒有直接連接到事實表上，而是通過其他維表連接到事實表上時，其圖解就像多個雪花連接在一起，故稱雪花模型。雪花模型是對星型模型的擴展"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/78/7879117a429adc2f8d1a3230b4ad6fc9.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"兩種模型思路的優劣也比較明顯："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/02/02fe411984de3c574985cd9f61cdb86c.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數據域"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 不敲了，第三個重點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據倉庫存放了企業多個業務或者產品的數據，雖然可以根據抽象粒度進行分層，但是我們仍然需要標識業務類型和研究對象，就出現了業務域和主題域。業務域顧名思義就是描述產生此數據的業務，如：短信業務、二手車業務、金融業務、租房業務等，而主題域可以理解爲研究的主體，比如：用戶、商品、廣告等。通過將兩者組合來分別數據模型存儲的信息內容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"指標與維度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據倉庫自下而上的構建，就是爲了獲取各種維度下的指標統計量，來分析數據制定決策。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數倉中指標最基礎指標爲原子指標，也成爲度量，用來表述最基礎的信息，比如“元”，“用戶數”等，在原子指標的基礎上，會根據不同的場景生成派生/衍生指標，規則是原子指標+時間週期+修飾詞的組合。比如近7日銷售額，近7日北京銷售額。當然通過原子指標加減乘除得到的指標也是派生/衍生指標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 一個數倉模型涵蓋內容如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8f/8f1e1f7b934859cc3d691159c7976e78.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"如何做數倉建模？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 以上用較大的篇幅介紹了數據倉庫及需要理解的知識點，那麼下一步就是如何去做數倉建模。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此處的前提是已經有數倉技術框架和產品，畢竟我們主要是構建數倉模型，而不是搭建完整的數據倉庫。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"規範化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、定義數倉模型層級標準，如："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d1/d17d40e62eab8de316baf25a4fa6ce4d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、定義主題域與業務域"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、定義模型命名規範，如：{層級}_{業務域}_{主題域}_{產品}_{事實描述}_{更新週期}，dws_video_user_h5_active_stat_di，表示視頻業務用戶維度活躍統計輕度彙總層，天級更新；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、定義原子指標、修飾類型、修飾詞，原子指標如：PV,UV，修飾類型如日期、行爲，修飾詞對應就是具體內容如3d表示3天，act表示活躍縮寫，3d_act_uv表示近三天活躍用戶數；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5、定義指標存儲類型，如訂單編號varchar，用戶數bigint，時間戳varchar等"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數倉建模流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在數倉建模設計中，大多DWD層模型是非業務需求的，是對業務日誌數據的輕度整合，方便向上構建DWS、ADS，或者方便直接即席查詢。通常數倉建模流程是："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、對接需求：瞭解業務方的維度和指標需求，看是否有現成的結果或者可以從已有的中間層加工"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、數據調研：對於數倉中暫時沒有的數據，通常是新業務沒有同步數據到數倉，或者說沒有埋點採集，前者直接同步數據，後者需要新建埋點；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、需求分析評審：評估需求合理性及複雜度，評估需求是否常用，如果不是走臨時需求開發流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、數據統計邏輯和物理模型設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5、ETL腳本開發"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6、腳本及數據校驗"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7、ETL腳本配置調度上線"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一些建議"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據模型設計比較考驗用戶的下鑽和上卷邏輯性，從整體去思考層級結構。雖然說數據倉庫允許一定成的讀冗餘，但是多人協同建設的時候儘量不要出現同一個指標出現在多個人構建的模型內（除非是作爲下游直接獲取），不僅是口徑發生變化時候更新成本高，同時如果出現信息不對等，那麼就會出現同一個指標不同值的情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 建議大家在承接一個業務的時候，優先把這個業務的指標體系構建出來，這樣哪怕業務方暫時沒有提出相應的需求，你也可以預先進行構建中間層。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"寫在最後"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 本文完全是筆者從分析師角度實踐的一些經驗，數據倉庫是一個龐大的體系，文中可能會有一些不準確或者理解不同的描述，請大家指正，也歡迎大家一起交流心得。"}]}]}