新一代設計編排運維套件分享

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"導讀：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今年是電信O3設計編排、移動O4二級編排(以下統稱爲業務編排中心)的大規模交付年；業務編排中心繫統採用了雲化的微服務架構，服務之間的調用鏈長，依賴關係錯綜複雜，讓運維複雜度陡增。當系統出現異常時，幾乎都是依賴人工運維，運維壓力大增，人力成本很高，給各地辦事處帶來了沉重的運維負擔。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這與公司今年推行的降本增效總體方針背道而馳。急迫需要一種工具能夠把人力從運維中解放出來，降低運維成本，運維子系統這種自動化運維產品套件因孕而生。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"客戶現場的運維壓力","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“爲什麼我的手機到現在還沒有開通？”客戶因等待業務開通時間過長產生不滿而詢問營業員時，營業員面對這種情況除了一邊勸客戶耐心等待，一邊向後端支撐人員求援以外也是毫無它法。爲了儘量少的影響客戶感知，現場運維人員不得不全力以赴解決此異常，給運維人員帶來的壓力可想而知。除了前端給運維人員帶來的壓力，異常的定位、修復效率也讓運維人員異常頭疼。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"雲化的微服務架構，讓異常定位難","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務編排中心爲了高擴展性，採用分佈式雲化微服務架構，既需要高性能又需要進行微服務集羣化部署。這樣做產生的副作用就是讓服務間的依賴關係越來越複雜，系統間的交互節點越來越多，調用鏈越來越長。目前業務編排中心對業務端到端全流程沒有做到全方位監控，當異常發生時需要通過查看各種日誌來還原異常發生的原因，讓異常定位比以前的單服務架構困難了很多倍。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"異常單點處理，無法形成有效可共享的經驗庫","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維人員在平常處理異常的過程中，都能夠總結出一定的經驗和方法，下次再遇到同類問題時，處理效率能夠提高很多。但因爲沒有相關工具來支撐，這些經驗和方法無法及時共享。換一個人或省份再遇到同類異常時，又得從頭再來，這無形中讓異常修復效率降低了很多。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"缺少運維工具，讓異常處理時效性難保障","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於同一類異常，就算有豐富的處理經驗，但因爲缺少運維工具的支撐，異常的修復完全依賴人工操作。少量出現時還能應付，但當發生大面積異常時，就會讓運維人員應接不暇、焦頭爛額了，異常的處理時效性也難以得到有效保障。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何能夠幫助前端運維人員減輕壓力，同時響應公司降本增效的號召，將人力從繁瑣的日常運維中解脫出來，增加端到端人效，是我們必須要思考並解決的一個問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"運維套件因此孕育而生。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"降本增效神器--運維套件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維套件就是把運維人員日常解決問題的經驗和方法通過腳本轉換成可動態運行的一個個組件，再通過運維編排將這些組件按照一定的順序自動執行，最終實現異常的自動化處理。要想知道運維套件是如何實現異常自動化運維的，需要對運維套件的工作原理、功能以及運行方式有一個深入的瞭解。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"運維套件自動化運維原理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"運維套件的功能體系和運行模式與業務編排中心是一致的","attrs":{}},{"type":"text","text":"，運維套件能夠實現自動化運維，離不開業務設計編排的賦能，業務設計編排將異常查詢能力、業務數據的查詢能力、業務數據的修復能力、流程引擎的調度能力等開放給運維套件使用；運維套件利用業務編排中心賦予的能力，通過設置一定的監控策略，能夠主動監控業務編排中心發生的異常。當異常發生時，運維套件根據策略自動匹配異常處理組件，啓動異常修復程序。對於修復成功的，運維套件通知業務編排中心讓流程繼續流轉下去。對於修復失敗的，則通過消息推送功能，將處理失敗的結果告知運維人員，轉爲人工處理。具體的處理過程如下圖所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1d/1d0dc6bd957f9f4bc6bb9235aef85699.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從運維套件異常修復流程來看，運維套件要能夠良好運行、實現自動化運維，必須做到主動發現、自動修復、修復失敗後能夠轉人工處理並通知相關運維人員。基於這個功能需求，就可以設計出運維套件的功能架構。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"運維套件功能架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從運維套件功能需求來看，運維套件包含運維監控、運維編排、異常修復、消息管理四大核心模塊。爲了管理運維套件，還需增加基礎管理模塊。具體功能架構如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎管理","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實現運維套件的基礎管理功能，包括運維人員管理、組織權限管理、日誌管理以及修復指令腳本管理。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維監控","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主動監控業務編排中心整個運行過程以及業務編排中心依賴的PAAS平臺。包括訂單監控、流程監控、API監控、PAAS平臺監控。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維編排","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對監控發現的異常，啓動流程實現修復流程調度。包括組件管理、運維流程編排、運維任務管理、運維調度管理。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異常修復","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實現異常的自動修復或生成人工修復工單。包括自動修復管理、人工修復管理、異常修復統計。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息管理","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於自動修復失敗的異常通過郵件或短信等方式通知運維人員，包括消息模板管理、消息推送管理、消息降噪規則管理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從功能架構可以看出運維套件與業務設計編排是相輔相成的。業務設計編排爲運維套件賦能，運維套件協助業務設計編排完成異常修復，運維套件可以與業務設計編排無縫銜接。運維套件除了實現異常的監控和修復兩大核心能力外，還附帶了其它兩大好處。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一省定製、多省共享","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"試點省份定製化運維組件，上線穩定運行後，可共享給其它各省份。其它省份出現類似異常時，可以直接加載使用，無需再次投入人力重複開發組件。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖表監控、一目瞭然","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維套件自帶圖表統計分析功能，包括異常組件數量統計、運維任務統計、異常修復率統計等，讓運維套件的運行情況一目瞭然。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維套件一項核心的能力就是","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"實現異常的自動修復","attrs":{}},{"type":"text","text":"，而自動修復是通過異常修復組件實現的。這裏的組件是基於Groovy腳本將平常的運維經驗和方法進行固化，可以根據需要實時動態定義。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面通過實例來看看是如何將異常處理方法和經驗封裝成組件的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"組件封裝實例","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"某省O3設計編排上線初期，異常單相對比較多，花費了運維人員極大的精力。通過對異常進行分析、總結、歸類發現，其中異常率最高的一類異常爲編排中心調用流程引擎創建工單時，因爲網絡不穩定，調用超時造成工單創建失敗。根據以往的處理經驗，我們很容易分析出處理這類異常需要分兩步走。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一步：獲取工單創建失敗的消息ID。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二步：根據消息ID判斷工單是否真的創建失敗，如果實際是創建成功的，則將該消息修改爲創建成功，如果創建失敗，則重發該消息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據以上兩步，我們可以分析出業務編排中心需要開放異常消息查詢能力、工單實例查詢能力、消息重置成功能力以及消息重發能力。根據業務編排中心提供的能力，基於Groovy腳本，我們可以定義兩個組件來修復該異常。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一個組件爲異常消息查詢組件，通過調用業務編排中心開放的異常消息查詢能力查詢工單創建失敗的消息ID，根據運維套件的組件規範要求，所有的處理組件都必須實現IGroovyExecutorExt接口，邏輯代碼如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3b/3ba6f2f6f1e18380229d67e0dc2e0613.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二個異常修復組件，首先調用業務編排中心開放的工單實例查詢能力判斷工單是否真的創建失敗，再根據判斷結果分別調用消息重置成功能力或消息重發能力，實現異常修復。其邏輯代碼如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/30/30174dc1649153f1b5b2026a79431979.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏之所以拆分成兩個原子組件是基於組件能力複用來考慮的。組件編寫完成後，可以即時加載到運維套件系統中，通過運維編排，將查詢組件的輸出結果作爲異常修復組件的輸入條件。前期爲了驗證組件的執行效果，可以通過手工執行的方式進行驗證。當驗證成功後，可以將該組件設定爲自動調度運行。運維套件下次再監控到該類異常時，就可以自動修復了，該組件穩定後可以共享給其它省份使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"運維套件價值成效","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在運維套件的建設之初，我們就爲它設定一『低』三『快』的目標。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一低即能夠低成本部署，應用部署時間約爲1人天左右，不需要高配置，並且套件本身消耗較小，可以與設計編排中心共享機器。如果需要額外申請機器，申請最小配置即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3b/3b53bb30e4f334d5fc7a1ff9119c8174.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三快即快速自動發現異常、快速自動定位異常、快速自動修復異常。三快給我們帶了效能的極大提高，主要體現爲：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"釋放人力，降低運維成本","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異常處理全自動化，只有在極少數自動化處理失敗的情況下，才需要人工介入，極大地釋放了運維人力，降低了運維成本。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"降低運維門檻，減少對高級人員的依賴","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"產品線將各種異常組件封裝後供各省共享使用，極大的減少了各省的運維門檻，減少了各省對高級運維人員的依賴，可以投入更多的資源服務於客戶。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"節約時間成本","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在未上線運維套件以前，遇到異常需要手工一個一個的修復，有了運維件套件以後，系統能夠一次性批量處理一類異常，大大節約了異常修復時間。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"保障業務系統穩定高效運轉","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"許多常見的異常都通過運維套件快速自動處理了，保障了業務系統穩定高效運行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以某省電信O3系統爲例，運維套件上線前，異常率佔比約爲0.08%。運維套件上線後，初期根據業務需要定製了5類處理組件，經過2周左右的時間運行，異常率減少到了約0.03%。在無需人工參與的情況下，異常率整整","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"減少了約62%","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過數據可以分析出，通過分類處理後，真正需要人工介入的異常相對比較少，大部分異常都可以通過運維套件自動化處理。因此不難看出自動化的運維套件是一個低成本高收益的降本增效神器！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動運維套件自建設以來已經加載了上百個運維組件，運維套件的建設對外能夠自動監控異常的發生，先於造成客戶影響前發現問題；自動化異常處理技術，實現異常自動化修復，先於客戶投訴前解決問題；處理失敗的異常告警能力，能夠將自動修復失敗的異常推送給相應運維人員，防止異常丟失，提高運維人員的認同感和滿意度；對內積極響應公司的降本增效號召，降低公司的運維成本，一舉四得。自從有了運維套件，運營業務再也不必擔心會遇到客戶的不滿投訴了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維套件目前只對接了業務設計編排中心，像資源、採控、調度等系統平常也會遇到同樣的運維問題，後續根據需要也可以對接資源、採控、調度等其它系統，運維套件必將發揮它應有的光芒。","attrs":{}}]}]}