銀行業智能運維的探索與實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"前言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"國家的“十四五”規劃裏面多次提到數字化相關概念，其背後反映了一種趨勢——從數字化經濟、數字化生活到數字化國家。近年來產業數字化不斷加速，金融業的數字化更是走在前列，銀行作爲金融業的主要機構也在經歷數字化轉型的過程，在銀行業務高速增長的背後離不開IT系統的支撐與支持，因此對IT系統的依賴與日俱增。IT運維是IT系統自身的運營管理，隨着混合架構、互聯網應用與傳統應用的雙模態發展，雲計算、大數據、5G、移動互聯等技術的應用，一方面對銀行的發展帶來了極大的助力，同時也對傳統的IT運維帶來了很大的挑戰，智能運維的應用正在逐漸成爲銀行IT運維的新方向，已經有一些銀行開始了智能運維的實踐探索。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":12}},{"type":"strong","attrs":{}}],"text":" 銀行傳統運維的痛點","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（一）質量","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IT運維關係到系統運行的穩定和持續，是銀行經營的基礎，在數字化程度很高的今天，可以說IT運維質量是銀行經營的命脈之一。如果簡單的以系統不停機、可持續作爲衡量標準是一個粗放的目標，運維質量需要從運維工作的各個環節來提升質量，有多種衡量指標，有了過程的質量控制才能實現結果的可控。傳統運維在衡量體系上缺乏手段和工具，例如對運維環節的風險評估比較片面，一些人工複覈也缺少體系化，在準確度和覆蓋面都有可能遺漏，從而影響整個運維質量。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（二）成本","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"銀行的運維會涉及到IT基礎資源的規劃、購買和使用，這些都直接關係到運維成本的高低。銀行作爲商業機構，肯定有控制成本的壓力。傳統運維根據數據中心的歷史數據進行資源申請，有很大的侷限性，不能根據業務系統的類型進行區分，某個系統是平穩型系統，還是一個波動型的系統，傳統的評估是做不到的，因此在資源申請、規劃、使用中就會產生過多的冗餘或者不足，造成運維成本的上升。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（三）效率","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IT運維的效率可以從兩個角度來看發展趨勢。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一是在雲環境和混合異構環境之下，IT運維的複雜度在不斷上升。銀行的業務系統本來就比較龐雜，前中後臺的業務系統疊加了不同的技術路線、不同的產品，包括開源、半開源、商業化產品的混合使用，讓底層的IT基礎設施也各有差異。這就讓IT運維要在跨系統的環境中實現，導致運維複雜不斷攀升，一個有經驗的運維管理人員發現故障後，也要消耗大量時間檢查每一個系統，進行例如狀態數據分析、抓包分析才能定位故障。這種運維效率在系統複雜、工作量大的環境中就顯得捉襟見肘。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一個角度，銀行運維人員每天被各種運維事件拖住，處於“四處救火”的狀態，其實非常被動。而且要求在人員不增加的情況下實現高效運維，這就需要從運維思路上改進——把運維處置從事中、事後向事前處置的方向來發展，通過更多的事前處置來提高整個運維的效率。這是一種防患於事前的思路，把事中向事前做，這樣才能大幅度提升運維效率。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":12}},{"type":"strong","attrs":{}}],"text":"智能運維帶來的四點變化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智能運維的興起將爲解決上述提到的質量、效率、成本的三大痛點帶來大幅度的提升，智能運維通過數據、AI算法與人類專家庫經驗的結合，深入運維場景爲IT運維帶來跨越式的提升。從運維工作的環節上來看，可以帶來以下四點明顯的變化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（一）實現智能化監控","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"企業使用大量的監控工具如APM、NPM、日誌、基礎設施監控等，來實現各個技術棧的監控。然而大量無效、無用數據會增加後端數據處理的壓力，而數據的漏採可能導致問題、故障的漏報；此外，監控工具需要大量的人工調試配置、嚴重依賴運維人員的經驗，人工成本巨大。在智能運維中，通過基於機器學習算法的智能數據採集器來實現智能的數據過濾、關鍵數據識別、採集密度與頻率調整以及採集服務器的性能均衡，從而提升數據採集的準確度、最小化人爲干預程度、降低人力成本以及提高運維管理效能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（二）智能化問題發現","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"企業IT系統規模的擴大、運維環境的複雜化，使得運維人員從海量的數據中發現問題的難度也越來越大。智能運維可以通過智能異常檢測、故障關聯分析、故障根因分析和智能異常預測等能力，幫助運維人員快速定位問題、追溯故障根源，並實現故障的預測預警。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以智能異常檢測爲例，通過基於密度算法的異常檢測（LOF）方法、基於Ensemble的快速異常檢測方法、基於歷史數據模型的異常檢測等方法等AI技術，能夠自動、實時、準確地從監控數據中發現異常，爲後續故障的分析與處理提供基礎。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對故障進行根源分析是在衆多可能引起故障的因素中，追溯到導致故障發生的癥結所在，並找出根本性的解決方案。利用機器學習或者深度學習的方法可以找出不同因素的之間的強相關關係，並利用這些關係，推斷出哪些因素是根本性的因素，幫助用戶快速診斷問題、提高故障的定位速度以及修復效率。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（三）智能化告警","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的告警管理一般使用固定閾值並且需要運維人員手動設置，這種方式不僅工作量巨大且十分依賴運維人員的經驗，閾值設置不當可能導致告警風暴或者告警漏報等後果。當監控環境發生變化時，原先的固定閾值無法滿足告警管理的要求。智能運維採用動態基線告警方式，智能分析數據的動態基線（即相對於歷史時刻，當前狀態的數據範圍），彌補了以往人爲設置固定閾值的缺陷，智能地分析數據的發展趨勢以及分析數據動態基線，從而對告警做出智能的判斷。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"各種監控工具會產生海量的告警信息，這些告警信息中可能存在大量的冗餘告警甚至形成告警風暴，對運維人員產生極大干擾，降低了運維工作的效率。智能運維針對短時、大量的、甚至是持續的冗餘告警，可以通過相似度、相關性判斷對這些冗餘告警進行合併，從而爲運維人員提供有效的告警信息，能夠大幅降低運維工作的難度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（四）智能化處置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統運維管理中對故障的處理非常依賴運維人員的經驗，但人的經驗無法覆蓋所有故障範圍，運維人員經驗不足可能會使得運維效率低下或者產生錯誤決策。智能運維將API接入的實時監測結果或者預測結果引入決策知識庫（智慧大腦）智能生成決策建議，並根據實際結果及趨勢判斷採用的處理策略，可以是人工處理或者自動處理，有效減少問題排查的時間、大幅提升問題解決的效率，提升企業運維的標準化程度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":12}},{"type":"strong","attrs":{}}],"text":"某互聯網銀行智能運維實踐","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"某互聯網銀行在運維管理中，以提升整體運維服務質量爲導向，爲行內提供全方位的運維服務，進而制定了智能運維的目標：整合現有的運維數據資源，完成運維數據智能分析平臺的原始數據準備；通過對接現有現有工具平臺，實現現有運維數據的智能分析，事件根因定位，容量預測，動態基線監控，變更輔助決策等能力；通過建設運維數據智能分析平臺，完成運維工具智能化轉型，積累智能運維經驗，提高智能運維比例。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（一）做好智能運維的準備工作","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實踐智能運維需要在運維數據的標準化和規範化做好充分準備。該銀行把例如日誌、交易、事件、CMDB、工單流程，這些源數據進行統一的歸檔管理，納入歸檔平臺，做好了運維數據的標準化和規範化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"策略知識庫是提升智能運維的基座，日常處置的知識積累尤爲重要，應該轉化爲知識庫和自動化系統，從策略到智能，做的策略越多，後面的智能應用水平也越高。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，系統之間要充分開放API接口，才能打通各個系統。該行在系統之間接口開放上進行充分了準備，讓智能運維的應用能夠實現跨系統處置。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（二）研發智能分析平臺，探索智能運維場景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該銀行啓動了智能監控、智能分析項目。通過全方位的監控和數據採集，把系統監控、交易監控、業務監控的源數據進行整合，提供給智能分析平臺，再對輸入的數據進行算法輸出，結合CMDB、事件平臺做了一些智能運維的場景應用，包括異常檢測、動態閾值、告警壓縮、自愈、容量預測。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以異常檢測爲例。針對業務四大黃金指標而設計的智能曲線異常檢測系統。四大黃金指標包括交易量 (業務實時產生的交易量)、業務成功率 (業務成功量 / 交易量)、系統成功率 (系統成功量 / 交易量, 業務成功量和系統成功量的區別在於是否明確捕捉到系統異常)、平均時延 (交易的平均耗時)。這四大黃金指標都是分鐘級數據，因爲過去只要影響到業務層面的故障，最終都會在這四個指標上有所體現。因此只要能準確捕捉到這四個指標的異常波動，就可以檢測到所有影響業務的異常。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/97/971a95501c0a7274de8c01de3a593bd6.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"size","attrs":{"size":9}}],"text":"異常檢測場景設計","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以容量預測爲例。傳統方式是使用同環比監控，需要運維人員根據經驗設置閥值。智能運維能夠對異常事件實現自動發現和預警，徹底擺脫根據人工經驗定義異常閾值，使用機器學習算法實現無閾值 KPI 曲線異常識別，針對關鍵容量指標、進行容量預測及告警、關聯自動化平臺實現自動擴縮容和資源回收。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":12}},{"type":"strong","attrs":{}}],"text":"小結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智能運維作爲近幾年新生的運維方向，其本質是用人工智能算法在多個方面替代人的判斷和行爲，其所面向的應用場景仍然與現有的運維場景相似，都可以根據事件的不同分爲三大類，分別爲針對歷史事件、針對當前事件以及針對未來事件的智能運維。根據這三大分類也就形成了三大典型場景：一是分析類場景、二是檢測類場景、三是預測類場景（故障自動化處置建立在上述三個場景之上）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，銀行業對智能運維的實踐還處於探索階段，許多實踐領域需要數據、系統、算法與人類知識的結合，這不是一個機器算法完全代替人的實踐，而是一個綜合的工程化實踐，需要對應用場景的深刻理解與各類工具的細緻導入，人的經驗不可或缺。因此，無論是銀行業用戶，還是IT供應商都應該從場景落地的角度出發去實踐智能運維，以達到最佳的應用效果。","attrs":{}}]}]}