Prometheus 與 nodata 告警

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着雲原生和高動態服務端的發展,在運維領域,以 Prometheus 爲代表的現代時間序列存儲正在加速替代以 Zabbix 爲代表的傳統監控系統。運維領域在享受時間序列技術發展紅利的同時,也面臨時間序列管理思路上的轉變和監控系統實際應用的上一些難點 —— nodata 告警便是其中之一。nodata 告警是傳統監控系統的必備功能,但卻缺席了幾乎所有現代時間序列存儲實踐,這給運維監控帶了諸多缺陷。本文嘗試分析其中原因,並給出一些可能的解決方法。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"nodata 告警觸發器的特殊性與必要性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"nodata 告警觸發器(","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Trigger","attrs":{}}],"attrs":{}},{"type":"text","text":")與普通告警觸發器相比具有原生的特殊性。普通告警觸發器的作用是對一組監控指標(","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Metric","attrs":{}}],"attrs":{}},{"type":"text","text":")的過濾,通常是基於數值大小的過濾。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/70/70827231b1c8e2c0738c07c5bbed4e21.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維監控場景下,發生 nodata 告警最大的可能性是監控系統本身的失效,比如採集點失效或採集對象失效,在我們的實踐中,服務器意外下線、磁盤故障、服務崩潰等都會導致 nodata 告警;另外還有一類監控指標,這類指標以 nodata 爲『正常狀態』,如 5xx code 產生的速率,在沒有 5xx code 產生時,雖然我們希望指標的數值爲 0 (而不是 nodata) ,但在實踐中往往很難保證,對於這類指標有效性的保證,我們會在其他文章中詳細說明。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"nodata 告警觸發器的難點之一在於全集 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"U","attrs":{}},{"type":"text","text":" 的獲取。在高動態的服務端環境中,往往很難得到『全部服務器集合』、『全部 IP 地址集合』、『全部 Pod 集合』、『某服務全部運行實例集合』這樣的全集。所以,在數值型的監控採集之外,必須建設更加結構化的信息組織方式,並配以自動、半自動與人工相結合的信息維護方法。假如在結構化信息中很難方便準確地獲取『全部某某集合』這樣的信息,就無法制作真正有效地 nodata 告警觸發器。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"nodata 告警觸發器的另一個難點是計算的開銷大。普通告警觸發器對指標的數值過濾,可以通過『帶條件的查詢』做到,這本質上是將告警計算的開銷一次性卸載到時間序列存儲系統中,而現代的時間序列存儲系統一般都支持這樣做。由上文對 nodata 告警觸發器的定義可以得到,nodata 的計算必須在數值過濾之前,也就是說 nodata 告警計算的計算對象是全量的監控指標,對全量監控指標求補集本身是一個開銷巨大的計算。另外全集 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"U","attrs":{}},{"type":"text","text":" 並不存在於時間序列存儲中(否則 nodata 告警就失去了客觀性),把全集 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"U","attrs":{}},{"type":"text","text":" 帶入 nodata 告警計算可會給時間序列存儲帶來額外的傳輸與計算壓力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然有諸多困難,但 nodata 告警的重要性不言而喻。如果沒有 nodata 告警,監控指標的失效是靜默的,監控系統本身的有效性無法得到保證。對於雲原生的服務端環境,監控對象的動態化程度更高,雖然可以製作更加宏觀的監控指標(如某類 Pod 的總實例數),但 nodata 告警可以幫助我們獲悉更加微觀的服務端運行工況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 OpsMind 的實踐中,我們使用 CMDB 和經典的 CMDB 方法來獲得全集  ,並改造 Prometheus ,將 nodata 計算卸載到存儲層。下文結合我們的實踐,並儘可能剝離我們特殊的業務場景,以 Prometheus 爲例,介紹幾個相對通用的 nodata 告警觸發器的實現思路。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"單一維度的 nodata","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單一維度的 nodata 是最常見的 nodata 告警觸發器,Zabbix 等傳統監控系統提供的也是這類 nodata 功能。以服務器 Load 監控爲例","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/4847257702fbc3fb95590466d76955b0.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在我們的實踐中,爲了將 nodata 的補集運算卸載到 Prometheus,我們將 CMDB 作爲一個監控點,由 Prometheus 向 CMDB 拉取全集 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"H","attrs":{}},{"type":"text","text":" ,具體的指標類似於","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"nodata_hosts{host=\"h1\", nodata=\"True\"} 1\nnodata_hosts{host=\"h2\", nodata=\"True\"} 1\nnodata_hosts{host=\"h3\", nodata=\"True\"} 1\n...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時,假設 Load 監控指標 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"L","attrs":{}},{"type":"text","text":" 類似於","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"host_cpu_load5{host=\"h1\"} 42\nhost_cpu_load5{host=\"h2\"} 43\nhost_cpu_load5{host=\"h3\"} 44\n...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們針對 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"L","attrs":{}},{"type":"text","text":" 生成如下告警觸發器","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"host_cpu_load5{host=~\"h.*\"} > 42","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"則卸載 nodata 之後的運算可表示爲","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"host_cpu_load5{host=~\"h.*\"} or on(host) nodata_hosts{host=~\"h.*\"} * 1/0 > 42","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此告警觸發器可以生成如下的告警信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"host_cpu_load5{host=\"h2\"} 43\nhost_cpu_load5{host=\"h3\"} 44\nhost_cpu_load5{host=\"hx\", nodata=\"True\"} +inf","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏有如下幾個關鍵點","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"將 CMDB 中的結構化信息轉儲到 Prometheus","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"or","attrs":{}}],"attrs":{}},{"type":"text","text":" 運算符做補集運算","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"on()","attrs":{}}],"attrs":{}},{"type":"text","text":" 的 label 爲 nodata 的單一維度","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"普通指標與 nodata 指標在 nodata 維度上的查詢條件一致","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"or","attrs":{}}],"attrs":{}},{"type":"text","text":" 之後的表達式通過 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"* 1/0","attrs":{}}],"attrs":{}},{"type":"text","text":" 轉爲 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"+inf","attrs":{}}],"attrs":{}},{"type":"text","text":" 以保證數值條件成立","attrs":{}}]}],"attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們通過將全集  轉爲監控指標,並通過 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"or","attrs":{}}],"attrs":{}},{"type":"text","text":" 運算符做補集運算實現了單一維度的 nodata 告警觸發器。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"多維度正交的 nodata","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多維度正交 nodata 也是運維監控場景中的常見需求。假設有 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"n","attrs":{}},{"type":"text","text":" 臺服務,每臺服務器上都運行相同的一組 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"m","attrs":{}},{"type":"text","text":" 個服務實例,那麼對於服務的監控指標,就需要在服務器和服務兩個維度上做 nodata 計算。問題描述如下","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c3/c3f896faf79e106a14763cf9dd0f8473.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與單維度類似,我們轉儲服務的全集指標","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"nodata_services{service=\"s1\", nodata=\"True\"} 1\nnodata_services{service=\"s2\", nodata=\"True\"} 1\nnodata_services{service=\"s3\", nodata=\"True\"} 1\n...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假設監控指標 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"Q","attrs":{}},{"type":"text","text":" 類似於","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"service_qps{host=\"h1\", service=\"s1\"} 42\nservice_qps{host=\"h1\", service=\"s2\"} 43\nservice_qps{host=\"h2\", service=\"s1\"} 44\n...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"則對於監控指標 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"Q","attrs":{}},{"type":"text","text":" 的一個告警觸發器類似於","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"service_qps{host=~\"h.\", service=~\"s.\"} > 42","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"卸載 nodata 計算之後的告警觸發器","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"service_qps{host=~\"h.*\", service=~\"s.*\"} or on(host, service) absent(nodata_hosts{host=~\"h.*\"}, nodata_services{service=~\"s.*\"}) * 1/0 > 42","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此告警觸發器可以生成如下告警信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"service_qps{host=\"h1\", service=\"s2\"} 43\nservice_qps{host=\"h2\", service=\"s1\"} 44\nservice_qps{host=\"hx\", service=\"s1\", nodata=\"True\"} +inf\nservice_qps{host=\"h1\", service=\"sx\", nodata=\"True\"} +inf\n...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了與單一維度 nodata 類似的關鍵點之外,這裏還有如下幾個關鍵點:","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"重寫 Prometheus 的 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"absent","attrs":{}}],"attrs":{}},{"type":"text","text":" 函數支持多 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"vector","attrs":{}}],"attrs":{}},{"type":"text","text":" 的正交計算","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"on()","attrs":{}}],"attrs":{}},{"type":"text","text":" 的 label 爲多個 nodata 的計算維度","attrs":{}}]}],"attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們通過與單一維度 nodata 類似的手法實現了支持多維度正交的 no","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"更一般的多維度 nodata","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多維度 nodata 更一般的表述是多維度之間無法形成正交關係的情況。這些情況較難處理,需要 CMDB 與時間序列存儲建立較密切的聯繫(而非簡單的數據轉儲),但如果可以處理得當,可以大大增強監控系統的能力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/06/066ecdcd076d239af0a0b363f77f2ec1.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這個監控指標的一個告警觸發器","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"service_qps{host=-\"h1,h2,h3\"service=-\"s1,s2\"} > 42","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意這裏我們擴展了 PromQL 的語法,支持『列表匹配』","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"=-","attrs":{}}],"attrs":{}},{"type":"text","text":",關於這個語法帶來的功能和性能的優化本文暫不贅述。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了對 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"Q","attrs":{}},{"type":"text","text":" 具有這類複雜多維度關係的指標,我們需要在 CMDB 中建立服務器與服務實例的關係表","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/99/99d93a4a0810367c2d7f80674cb6a267.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過 CMDB 中的關係表,將 nodata 卸載後的告警觸發器爲","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"service_qps{host=-\"h1[0,1],h2[2],h3[3]\"service=-\"s1[0,2,3],s2[1,3]\"} > 42","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏我們擴展了列表匹配的語法,支持 \"nodata key\" ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"[0,1]","attrs":{}}],"attrs":{}},{"type":"text","text":",表示某個列表項在全集中的 ID。Promethues 查詢時會針對每個 label 計錄缺失列表項的 nodata key,並將多個 label 記錄下來的 nodata key 求並集。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/55/5501224c7c43915620a03a2df373f9e0.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"結語","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分析了現代時間序列管理方案中 nodata 的特殊性與必要性,並以 Prometheus 爲例嘗試給出幾個解決方案。可以看到,現代時間序列管理方案中的 nodata 處理是十分複雜的,我們認爲這和雲原生環境下其他的監控難題一樣具有原生複雜性,這只是雲原生給運維帶來的諸多根本性挑戰的外在表現形式,這些挑戰需要系統性地分析和解決。OpsMind 爲應對雲原生的挑戰做了大量的技術研判和產品包裝,我們將在其他文章中與大家陸續分享。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章