kube-state-metrics代碼分析

前言

整體來看，kube-state-metrics是爲prometheus採集k8s資源數據的exporter。prometheus 的exporter任務是將和業務相關的數據指標轉換成prometheus的數據模型，當然prometheus 爲這種轉換提供了方法。

kube-state-metrics採集的數據

kube-state-metrics能夠採集絕大多數k8s內置資源的相關數據，例如pod、deploy、service等等。同時它也提供自己的數據，主要是資源採集個數和採集發生的異常次數統計。

prometheus指標類別

• Counter (累加指標)

  一個累加指標數據，這個值隨着時間只會逐漸的增加，比如程序完成的總任務數量，運行錯誤發生的總次數。常見的還有交換機中snmp採集的數據流量也屬於該類型，代表了持續增加的數據包或者傳輸字節累加值。

• Gauge (測量指標)

  Gauge代表了採集的一個單一數據，這個數據可以增加也可以減少，比如CPU使用情況，內存使用量，硬盤當前的空間容量等等。

• Summary (概略圖)

• Histogram (直方圖)

prometheus爲exporter提供的方法

定義指標

定義一個指標描述，第一個參數是指標的名字，第二個是指標的幫助信息，第三個是指標含有的label數組。

descDeploymentLabelsDefaultLabels = []string{"namespace", "deployment"}
descDeploymentCreated = prometheus.NewDesc(
		"kube_deployment_created",
		"Unix creation timestamp",
		descDeploymentLabelsDefaultLabels,
		nil,
	)

上面的代碼是kube-state-metrics定義了一個名爲kube_deployment_created指標描述。在prometheus中，這個指標的形式類似kube_deployment_created{“namesapces”=xxx, “deployment”=xxx, …}。

採集器

顧名思義，採集器就是採集數據，它是一個結構體，註冊到prometheus註冊器後，會被自動調用，從而實現採集功能。採集器要實現Describe和Collect方法。

type Collector interface {
    // 用於傳遞所有可能的指標的定義描述符
    // 可以在程序運行期間添加新的描述，收集新的指標信息
    // 重複的描述符將被忽略。兩個不同的Collector不要設置相同的描述符
    Describe(chan<- *Desc)

    // Prometheus的註冊器調用Collect執行實際的抓取參數的工作，
    // 並將收集的數據傳遞到Channel中返回
    // 返回的指標應該來自describe定義的指標。
    Collect(chan<- Metric)
}

註冊器

Registry註冊prometheus的collectors，並將所有collectors的數據聚合（gather）在一起。

那麼一個exporter的主要實現流程就是，定義指標->實現collector->將collector註冊到Registry；然後對外提供接口，調用Registry的gather功能，最終獲取到數據。

kube-state-metrics對外提供數據的方法是，創建http服務，將url /metrics的處理方法指向Registry的gather。

mux := http.NewServeMux()
mux.Handle(metricsPath, promhttp.HandlerFor(registry, promhttp.HandlerOpts{ErrorLog: promLogger{}}))

舉例：kube-state-metrics採集k8s deployment

定義collector

指標的describe定義省略。

type deploymentStore interface {
	List() (deployments []v1beta1.Deployment, err error)
}

type deploymentCollector struct {
	store deploymentStore
	opts  *options.Options
}

// Describe implements the prometheus.Collector interface.
func (dc *deploymentCollector) Describe(ch chan<- *prometheus.Desc) {
	ch <- descDeploymentCreated
    .........
	ch <- descDeploymentLabels
}

// Collect implements the prometheus.Collector interface.
func (dc *deploymentCollector) Collect(ch chan<- prometheus.Metric) {
	ds, err := dc.store.List()
    ............
	for _, d := range ds {
	    //採集每個deploy對象的具體數據
		dc.collectDeployment(ch, d)
	}

	glog.V(4).Infof("collected %d deployments", len(ds))
}

func (dc *deploymentCollector) collectDeployment(ch chan<- prometheus.Metric, d v1beta1.Deployment) {
	addGauge := func(desc *prometheus.Desc, v float64, lv ...string) {
		lv = append([]string{d.Namespace, d.Name}, lv...)
		//製作一個常量類型的metrics，傳遞給通道
		ch <- prometheus.MustNewConstMetric(desc, prometheus.GaugeValue, v, lv...)
	}
	labelKeys, labelValues := kubeLabelsToPrometheusLabels(d.Labels)
	addGauge(deploymentLabelsDesc(labelKeys), 1, labelValues...)
	if !d.CreationTimestamp.IsZero() {
		addGauge(descDeploymentCreated, float64(d.CreationTimestamp.Unix()))
	}
	addGauge(descDeploymentStatusReplicas, float64(d.Status.Replicas))
	...........
}

可以看到，deploymentCollector實現了Describe和Collect方法。

註冊collector

func RegisterDeploymentCollector(registry prometheus.Registerer, informerFactories []informers.SharedInformerFactory, opts *options.Options) {

	infs := SharedInformerList{}
	for _, f := range informerFactories {
		infs = append(infs, f.Extensions().V1beta1().Deployments().Informer().(cache.SharedInformer))
	}

	dplLister := DeploymentLister(func() (deployments []v1beta1.Deployment, err error) {
		for _, dinf := range infs {
			for _, c := range dinf.GetStore().List() {
				deployments = append(deployments, *(c.(*v1beta1.Deployment)))
			}
		}
		return deployments, nil
	})
	
    //使用client-go獲取namespace中的deploy數據，並將數據保存在dplLister
	registry.MustRegister(&deploymentCollector{store: dplLister, opts: opts})
	
	infs.Run(context.Background().Done())
}

該collector使用client-go的informer方式獲取k8s的deployment數據。看到這裏，kube-state-metrics的事情基本結束，剩下的就由prometheus相關方法處理。

處理註冊的collector

之前說過collector聲明瞭一些metric的描述，並把這些描述發送到一個Desc類型的通道，現在就要處理這些描述了。

func (r *Registry) Register(c Collector) error {
	var (
		descChan           = make(chan *Desc, capDescChan)
		newDescIDs         = map[uint64]struct{}{}
		newDimHashesByName = map[string]uint64{}
		collectorID        uint64 // Just a sum of all desc IDs.
		duplicateDescErr   error
	)
	//起個協程，讓collector把它的指標描述挨個發給通道
	go func() {
		c.Describe(descChan)
		close(descChan)
	}()
	r.mtx.Lock()
	defer r.mtx.Unlock()
	// Conduct various tests...
	//從通道里挨個讀指標，這裏要對指標的描述做一系列檢查，我猜測是和別的collector有衝突
	for desc := range descChan {

		// 檢查1：Is the descriptor valid at all?
		..........

		// 檢查2：Is the descID unique?
	    ..........

		// 檢查3：Are all the label names and the help string consistent with
		// previous descriptors of the same name?
		// First check existing descriptors...
	}
	// 檢查4：Did anything happen at all?

	// Only after all tests have passed, actually register.
	//通過所有檢查，將colletor註冊
	r.collectorsByID[collectorID] = c
	for hash := range newDescIDs {
		r.descIDs[hash] = struct{}{}
	}
	for name, dimHash := range newDimHashesByName {
		r.dimHashesByName[name] = dimHash
	}
	return nil
}

Register函數主要對聲明的describe做檢查，如果collector聲明的describe沒問題，則把collector存放在collectorsByID字典中。

從collector採集數據

func (r *Registry) Gather() ([]*dto.MetricFamily, error) {
	var (
		metricChan        = make(chan Metric, capMetricChan)
		metricHashes      = map[uint64]struct{}{}
		wg                sync.WaitGroup
		errs              MultiError          // The collected errors to return in the end.
		registeredDescIDs map[uint64]struct{} // Only used for pedantic checks
	)

	r.mtx.RLock()
	goroutineBudget := len(r.collectorsByID)
	metricFamiliesByName := make(map[string]*dto.MetricFamily, len(r.dimHashesByName))
	collectors := make(chan Collector, len(r.collectorsByID))
	for _, collector := range r.collectorsByID {
		collectors <- collector
	}
	// In case pedantic checks are enabled, we have to copy the map before
	// giving up the RLock.
	if r.pedanticChecksEnabled {
		registeredDescIDs = make(map[uint64]struct{}, len(r.descIDs))
		for id := range r.descIDs {
			registeredDescIDs[id] = struct{}{}
		}
	}
	r.mtx.RUnlock()

	wg.Add(goroutineBudget)

	collectWorker := func() {
		for {
			select {
			case collector := <-collectors:
				collector.Collect(metricChan)
				wg.Done()
			default:
				return
			}
		}
	}

	// Start the first worker now to make sure at least one is running.
	go collectWorker()
	goroutineBudget--

	// Close the metricChan once all collectors are collected.
	go func() {
		wg.Wait()
		close(metricChan)
	}()

	// Drain metricChan in case of premature return.
	defer func() {
		for range metricChan {
		}
	}()

collectLoop:
	for {
		select {
		case metric, ok := <-metricChan:
			if !ok {
				// metricChan is closed, we are done.
				break collectLoop
			}
			errs.Append(processMetric(
				metric, metricFamiliesByName,
				metricHashes,
				registeredDescIDs,
			))
		default:
			if goroutineBudget <= 0 || len(collectors) == 0 {
				// All collectors are already being worked on or
				// we have already as many goroutines started as
				// there are collectors. Just process metrics
				// from now on.
				for metric := range metricChan {
					errs.Append(processMetric(
						metric, metricFamiliesByName,
						metricHashes,
						registeredDescIDs,
					))
				}
				break collectLoop
			}
			// Start more workers.
			go collectWorker()
			goroutineBudget--
			runtime.Gosched()
		}
	}
	return normalizeMetricFamilies(metricFamiliesByName), errs.MaybeUnwrap()
}