眼觀六路耳聽八方還不知疲倦？數倉智能運維服務體系是怎麼做到的？

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​​摘要：本文主要介紹GaussDB（DWS）數據庫智能監控運維服務體系的設計規劃和現狀。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自華爲雲社區","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/265548?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=ei&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"《眼觀六路耳聽八方還不知疲倦？數倉智能運維服務體系是怎麼做到的？》","attrs":{}}]},{"type":"text","text":"，原文作者：魯大師。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景介紹","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"早期，數據庫系統僅僅提供SQL命令來查詢其內部的運行狀態，導致數據庫運維操作門檻高，易用性差，DBA一度成爲高度專業化的關鍵崗位，享受高薪和大家羨慕的目光的同時，也爲企業的數據安全帶來了不確定性風險。並且，命令行運維不直觀，嚴重依賴運維人員經驗，不能做到快速的發現、定位、解決問題，導致數據庫運維問題，發現難，定位難，解決難。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了應對這個窘境，數據庫運行狀態可視化（數據庫監控系統）應運而生，通過可視化的手段以人類便於理解的圖表形式，將重點數據以圖形化的手段展示給運維人員，從而顯著的降低了數據庫運維的門檻，提高了數據庫運維的效率。這個階段有一些代表性的產品比如：OEM(Oracle), ViewPoint(Teradata)，等等。但是，這個時期用戶的數據的規模不是很大，數據庫也依然部署在用戶自己的數據中心，依然是幾個DBA運維幾套數據庫的階段。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着雲時代的到來，雲數據庫逐漸託管了客戶的數據存儲服務，雲化將一切繁重的IT運維工作都集中在雲後臺管理了起來，從而把客戶從專業，複雜，繁重的數據中心運維活動中解放了出來，使客戶能夠更加專注於其核心業務。同時，雲服務提供商作爲數據存儲服務的提供者，則需要在IT運維與數據庫運維上深耕細作，發揮其團隊穩定，專業化程度高，掌握海量數據庫運行數據的優勢；充分利用目前機器學習、人工智能領域的科研成果，使用技術手段逐步提高每名運維人員所能管理的數據庫數量，從而實現數據庫運維工作的“減員增效”。另一方面，數據庫服務上雲後，雲服務提供商所需要運維的數據庫數量與之前相比天差地別，以前的工具可能已經不適應雲時代的需求。如何做好雲上海量數據庫的運維工作將成爲雲服務提供商的一個巨大挑戰。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數據庫智能運維體系","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統意義上的數據庫監控服務僅僅是指（1）採集數據庫運行狀態；（2）上報/存儲數據庫運行數據；（3）圖形化展示數據庫運行狀態數據。但是，這僅僅是數據庫智能監控運維體系的一部分。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/61/6192e051550bbaa29bd7633982e05eb5.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果把整個數據庫智能監控運維體系比作一個人的話，傳統意義上的數據庫監控服務僅僅代表了，眼睛的角色。該服務只能做到發現問題，識別定位問題和解決問題都需要DBA的介入。因此DBA纔是傳統數據庫監控運維體系中的核心要素，這也是DBA人才爲何如此關鍵的原因之一。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而云時代的到來和大數據分析、人工智能等技術的成熟，給了數據庫監控運維更多的想象空間。我可以在傳統數據庫監控（眼睛）的基礎上，增加預測分析和根因判斷模塊，建立現象-根因-解決方案的映射關係（大腦），最後通過數據庫管理模塊執行解決方案（雙手），從而實現從發現問題，定位問題，到解決問題的運維閉環。並且機器不同於人類，只要算力允許，它可以做到眼觀六路，耳聽八方，不知疲倦，也不會覺得無聊，7x24的盯着成百上千數據庫系統的各種運行數據，不會放過任何一個微小的潛在問題。因此，數據庫運維工作的智能化中，使用規則或算法固化DBA判斷和決策經驗將是非常重要的一環。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"GaussDB(DWS)的數據庫智能運維體系","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考友商數據庫監控運維體系的建設經驗，結合GaussBD(DWS)數倉的自身特點，我們準備從眼，腦，手三個方面發力建立閉環的數據庫智能監控運維體系。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/80/8011bbb56c659927e663cdf925fd7f6f.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​GaussDB(DWS)使用DMS來承載數據庫的智能運維體系。DMS將會串起數據庫運維過程中的監控，分析，處理三個步驟，分別對應上文提到的數據庫智能運維體系中的眼，腦，手三部分，從概念設計上形成運維體系的閉環。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"監控部分","attrs":{}},{"type":"text","text":"：主要負責數據庫運行狀態數據的採集、存儲和可視化展示，這一部分基本等同於傳統的數據庫的監控業務。這一部分功能和指標的選取，我們參考了友商以及運維團隊的建議，將監控指標分爲底層IT系統運維指標和數據庫系統運維指標兩類，正在分別逐步補齊和完善中。監控模塊是DMS數據庫運智能監控運維體系首先發力，並要在短時間內形成競爭力的模塊。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分析部分","attrs":{}},{"type":"text","text":"：作爲整個DMS數據庫智能運維體系的大腦，該部分是承擔運維數據分析與決策的關鍵模塊。該部分因爲其複雜性，目前還處於設計構想階段。初步規劃有三個子模塊，時間序列的趨勢分析子模塊，該模塊主要用來做趨勢預測分析，用來預判潛在的問題；邏輯推斷子模塊，用戶分析問題現象與實際根因之間的關係，可以實現從問題現象到觸發原因的推斷，初步考慮使用搜索引擎技術實現；知識圖譜子模塊，主要用於現象、根因與解決方案之間的映射關係表示，方便從定位的根因中找到最合適的解決方案。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"處理部分","attrs":{}},{"type":"text","text":"：主要由DWS提供的數據庫管理功能承擔，目前可以提供數據庫參數配置（可配置參數少，需要進一步豐富），工作負載隊列配置，集羣安裝/卸載，集羣重啓，集羣擴容，集羣數據重分佈以及節點溫備等運維能力。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"GaussDB(DWS)數據庫智能運維典型用戶與需求","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了進一步理清數據庫智能運維產品的設計思路，我們計劃從用戶的角度分析其需求，然後從需求導出功能（工具）頁面設計，從功能（工具）頁面總結出所需監控數據庫指標。通過分析數據庫監控系統的各種使用場景，我們對數據庫監控系統的用戶做了用戶角色畫像，定義了數據庫運維過程中的三種角色，並認爲不同角色僅僅關注數據庫運維的一個側面。在實際的數據庫運維場景中，可能同一個用戶會身兼多種角色，但是這裏我們爲了方便分析僅僅從邏輯上定義這三種角色。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5b/5b62d6d4c9de39627d89c60e67b68613.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"應用開發","attrs":{}},{"type":"text","text":"：主要指客戶側的應用開發角色，他們負責設計具體的業務SQL。他們關心業務SQL執行的正確性和執行效率。應用開發工程師需要用到web SQL來調試其SQL語句的查詢效率；需要用到查詢監控頁面來查看業務SQL在實際執行場景中的表現和資源消耗；需要用到工作負載隊列監控來確認新開發的業務SQL是否在合適的工作負載隊列中，以及所配置的熔斷規則是否合理，等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"SRE","attrs":{}},{"type":"text","text":"：指的是華爲雲側的數據庫運維角色，他們通常一個人需要負責成百上千個集羣的穩定運行，他們需要能夠迅速識別出集羣運行狀態的異常，集羣資源瓶頸以及集羣潛在的擴容需求，並且他們還需要積極響應客戶的求助，幫助客戶定位，確認和解決問題。SRE需要節點資源監控來識別集羣中的資源傾斜；需要識別集羣資源消耗基線變化趨勢，從而識別到擴容需並提醒用戶；需要關注存儲變化以推算下一次常規保養的時間點並自動規劃；同時還需要響應用戶需求，使用DMS提供的問題定位工具，輔助用戶定位現網問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"DBA","attrs":{}},{"type":"text","text":"：指的是GaussDB(DWS)數據庫集羣專家，他們熟悉數據庫設計方法論，數據庫的調優，數據庫問題定位。他們需要分析定位數據庫的故障，從資源和業務角度運用多種工具綜合分析定位系統故障，系統穩定性和潛在瓶頸；也需要幫助用戶從業務、數據庫設計的角度去推薦數據庫的索引，分佈列配置，根據用戶業務水平推薦用戶購買合適的集羣規模等等；同時還需要輔助應用開發工程師調優引起性能劣化的SQL語句；在找到確切的故障根因後，推薦合適的解決方案修復故障。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一般來說在公有云場景中，用戶角色一般只有應用開發和SRE兩種，公有云場景中的SRE角色往往涵蓋了DBA的角色。我們在這裏將運維角色細分的目的，其實是要展示一個完整的運維場景沙盤，將客戶的運維訴求分門別類的羅列出來，爲後續進一步的功能（工具）頁面設計和運維場景設計提供基礎。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"GaussDB(DWS)數據庫智能運維指標","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫監控指標數量多，形式和邏輯複雜，根據指標類型可以分爲邏輯關係與物理關係兩種。其中邏輯關係指數庫內部邏輯關係，比如，最頂層是數據庫，數據庫中有多個schema，schema中有多個表，數據庫中有多個用戶，一個用戶可以有多個schema和表。而物理關係是指，gaussDB(DWS)集羣的拓撲關係，比如，一個數據庫集羣是由多個計算節點構成，每個計算節點上會部署多個計算實例。這兩種指標關係都會影響到數據庫指標的採集維度和聚合展示維度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/15/150b58955ce6a0b707f876d5c2eab1dd.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​因爲上面已經分析了指標的維度關係，所以我們下面將只討論具體的數據庫指標類型，而不會對指標的維度進行展開。數據庫是一個軟件服務，而它必須運行在一個宿主機和操作系統之上，因此監控指標大致可以分爲兩類：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"系統資源類指標","attrs":{}},{"type":"text","text":"：這一類指標主要描述系統上的各種資源消耗","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"數據庫相關指標","attrs":{}},{"type":"text","text":"：這一類指標主要描述數據性能相關的業務負載水平","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/28/28ac813083588f93333cccd7ba5b11e0.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​上圖總結了DMS採集的數據庫主要指標，具體指標項按照指標大類，原子指標和派生指標三個層次排列。不過，目前該指標地圖並不固定，未來隨着GaussDB(DWS)智能運維繫統的逐步成熟，該指標地圖會逐步完善並固定下來。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲MPP數據庫的特殊構型，數據庫實例是作爲進程試運行在節點上的。因此，我們的指標設計其實本身會自帶維度屬性，比如磁盤使用率指標，最小的維度應該是某個DN實例，上一級是節點級，再上一級就是整個集羣。所以，我們實際提供的監控指標應該是指標維度關係與集羣指標地圖的一個笛卡爾積。爲了描述這種情形，我們引入原子指標，派生指標和組合指標的概念。以上面的磁盤使用率爲例，我們將DN實例的磁盤使用率作爲原子指標，而其他維度的磁盤使用率作爲派生指標。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"原子指標","attrs":{}},{"type":"text","text":"：描述數據庫某個特性的最小維度指標，比如節點的CPU使用率，DN實例的磁盤使用率，等等。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"派生指標","attrs":{}},{"type":"text","text":"：（1）原子指標在不同維度上的聚合結果，比如集羣平均CPU使用率，集羣磁盤使用率，等等；（2）對原子指標做統計運算得到的新指標，比如CPU傾斜率，等等。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"組合指標","attrs":{}},{"type":"text","text":"：將多個原子指標或者派生指標組合在一起，從而得到的更加便於理解的新指標。比如集羣健康度，等等。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"        ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前DMS的指標建設更多停留在原子指標和派生指標階段，因爲我們認爲應該首先補齊數據庫的基礎指標形成基本的監控運維能力之後，才能結合用戶使用習慣，深度挖掘指標在各個維度下的運維含義以及多種指標組合後所代表的運維意義。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，總結一下，本文主要介紹了GaussDB（DWS）數據庫智能監控運維服務體系的設計規劃和現狀。本文作爲DMS系列文章的第一篇，主要起到一個概要介紹的作用，讓大家對GaussDB（DWS）數據庫智能監控運維服務體系有個概略的認識，更多幹貨細節歡迎期待後續的文章。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"想了解GuassDB(DWS)更多信息，歡迎微信搜索“GaussDB DWS”關注微信公衆號，和您分享最新最全的PB級數倉黑科技，後臺還可獲取衆多學習資料哦~","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=ei&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"點擊關注，第一時間瞭解華爲雲新鮮技術~","attrs":{}}]}]}]}