根據Nginx Ingress指標對指定後端進行HPA

本文分享自華爲雲社區《根據Nginx Ingress指標對指定後端進行HPA》，作者： 可以交個朋友。

背景

生產場景下，Nginx Ingress的流量會通過域名和path路徑最終轉發至不同的應用，而有時候cpu和內存並不是nginx的性能瓶頸，此時可以基於nginx_ingress_controller_requests指標，爲其對應的應用配置HPA，以實現基於不同域名和path的請求量彈性指定後端工作負載

簡介

環境準備

• nginx-ingress已部署
• 雲原生監控插件kube-prometheus-stack已安裝（server模式)，插件默認監控nginx-ingress，開源環境請自行配置監控。
• 已配置kubectl命令或使用cloudshell

注意：由於HPA規則中scaleTargetRef和describedObject兩個字段都無法指定命名空間，所以指標來源、HPA和彈性目標需在同一命名空間，而nginx-ingress和業務工作負載一般處在不同命名空間；本次方案採用external類型的HPA，可以忽略指標來源的命名空間

操作步驟

創建演示需要的彈性目標工作負載，service以及ingress

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: test-hpa 
  labels: 
    app: test-app 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: test-app 
  template: 
    metadata: 
      labels: 
        app: test-app 
    spec: 
      containers: 
      - image: skto/sample-app:v2 
        name: metrics-provider 
        ports: 
        - name: http 
          containerPort: 8080 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: test-app 
  namespace: default 
  labels: 
    app: test-app 
spec: 
  ports: 
    - port: 8080 
      name: http 
      protocol: TCP 
      targetPort: 8080 
  selector: 
    app: test-app 
  type: ClusterIP 

--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: sample-app 
  labels: 
    app: sample-app 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: sample-app 
  template: 
    metadata: 
      labels: 
        app: sample-app 
    spec: 
      containers: 
      - image: skto/sample-app:v2 
        name: metrics-provider 
        ports: 
        - name: http 
          containerPort: 8080 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: sample-app 
  namespace: default 
  labels: 
    app: sample-app 
spec: 
  ports: 
    - port: 80 
      name: http 
      protocol: TCP 
      targetPort: 8080 
  selector: 
    app: sample-app 
  type: ClusterIP 
--- 
apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
  name: test-ingress 
  namespace: default 
spec: 
  ingressClassName: nginx 
  rules: 
    - host: test.example.com 
      http: 
        paths: 
          - backend: 
              service: 
                name: sample-app 
                port: 
                  number: 80 
            path: / 
            pathType: ImplementationSpecific 
          - backend: 
              service: 
                name: test-app 
                port: 
                  number: 8080 
            path: /home 
            pathType: ImplementationSpecific

分別查詢test.example.com/和test.example.com/home的nginx_ingress_controller_requests指標，指標正常

創建external類型的apiservices資源；創建後apiservices的狀態爲false是正常現象，添加externalRules後狀態變爲true

apiVersion: apiregistration.k8s.io/v1 
kind: APIService 
metadata: 
  name: v1beta1.external.metrics.k8s.io 
spec: 
  group: external.metrics.k8s.io 
  groupPriorityMinimum: 100 
  insecureSkipTLSVerify: true 
  service:         #指定prometheus-adapter對應的service，華爲CCE插件中adapter名稱爲custom-metrics-apiserver 
    name: custom-metrics-apiserver 
    namespace: monitoring 
    port: 443 
  version: v1beta1 
  versionPriority: 100

將externalRules規則添加到adapter的configmap中，修改後需要重啓prometheus-adapter服務

kubectl -n monitoring edit configmap user-adapter-config

externalRules: 
- metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>) 
  name: 
    as: ${1}_per_second 
    matches: ^(.*) 
  resources: 
    namespaced: false  #忽略指標來源的命名空間，該配置不適用rules規則 
  seriesQuery: nginx_ingress_controller_requests

seriesQuery：原始指標；可以直接寫指標名稱，也可以使用{labelKey=labelValue}的方式篩選出原始指標

metricsQuery：用PromQL對指標進行篩選匯聚；.Series表示原始指標，.LabelMatchers表示對指標進行標籤篩選，hpa中可以配置具體的篩選規則

name：將指標重命名

resources：hpa查詢指標時通過api的方式調用，調用路徑爲：

而resources的作用就是將指標中命名空間標籤的值替換路徑中的${namespace}，而我們本次方案需要忽略指標來源命名空間。

custom-metrics-apiserver服務重啓後需要等待1分鐘左右，執行命令查看指標是否正常

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/*/nginx_ingress_controller_requests_per_second

創建HPA規則

apiVersion: autoscaling/v2 
kind: HorizontalPodAutoscaler 
metadata: 
  name: sample-hpa 
spec: 
  scaleTargetRef: 
    apiVersion: apps/v1 
    kind: Deployment 
    name: sample-app 
  minReplicas: 1 
  maxReplicas: 10 
  metrics: 
    - type: External 
      external: 
        metric: 
          name: nginx_ingress_controller_requests_per_second 
          selector: 
            matchLabels:    #可以通過該字段對指標進行過濾，這裏標籤篩選條件會加入到externalRules的<<.LabelMatchers>>中。 
              exported_service: sample-app  #篩選後端服務爲sample-app的請求 
              host: test.example.com        #篩選訪問域名爲test.example.com的請求 
        target: 
          type: AverageValue   #External指標類型下只支持Value和AverageValue類型的目標值。 
          averageValue: 30 
--- 
apiVersion: autoscaling/v2 
kind: HorizontalPodAutoscaler 
metadata: 
  name: test-hpa 
spec: 
  scaleTargetRef: 
    apiVersion: apps/v1 
    kind: Deployment 
    name: test-app 
  minReplicas: 1 
  maxReplicas: 10 
  metrics: 
    - type: External 
      external: 
        metric: 
          name: nginx_ingress_controller_requests_per_second 
          selector: 
            matchLabels: 
              exported_service: test-app 
              host: test.example.com 
        target: 
          type: AverageValue 
          averageValue: 30

彈性演示

使用命令壓測sample-app對應的訪問域名和路徑，正常觸發彈性；請自行配置域名與ELB地址的映射

ab -c 50 -n 5000 http://test.example.com/

用同樣的方式壓測test-app，正常觸發彈性

ab -c 50 -n 5000 http://test.example.com/home

 

點擊關注，第一時間瞭解華爲雲新鮮技術~