應用級擴縮容是相對於運維級而言的。像監控CPU/內存的利用率就屬於應用無關的純運維指標，針對這種指標進行擴縮容的HPA配置就是運維級擴縮容。而像請求數量、請求延遲、P99分佈等指標就屬於應用相關的，或者叫業務感知的監控指標。
本篇將介紹3種應用級監控指標在HPA中的配置，以實現應用級自動擴縮容。
Setup HPA
1 部署metrics-adapter
執行如下命令部署kube-metrics-adapter(完整腳本參見：demo_hpa.sh)。：
helm --kubeconfig "$USER_CONFIG" -n kube-system install asm-custom-metrics \
$KUBE_METRICS_ADAPTER_SRC/deploy/charts/kube-metrics-adapter \
--set prometheus.url=http://prometheus.istio-system.svc:9090
執行如下命令驗證部署情況：
#驗證POD
kubectl --kubeconfig "$USER_CONFIG" get po -n kube-system | grep metrics-adapter
asm-custom-metrics-kube-metrics-adapter-6fb4949988-ht8pv 1/1 Running 0 30s
#驗證CRD
kubectl --kubeconfig "$USER_CONFIG" api-versions | grep "autoscaling/v2beta"
autoscaling/v2beta1
autoscaling/v2beta2
#驗證CRD
kubectl --kubeconfig "$USER_CONFIG" get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": []
}
2 部署loadtester
執行如下命令部署flagger loadtester：
kubectl --kubeconfig "$USER_CONFIG" apply -f $FLAAGER_SRC/kustomize/tester/deployment.yaml -n test
kubectl --kubeconfig "$USER_CONFIG" apply -f $FLAAGER_SRC/kustomize/tester/service.yaml -n test
3 部署HPA
3.1 根據應用請求數量擴縮容
首先我們創建一個感知應用請求數量(istio_requests_total)的HorizontalPodAutoscaler配置：
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo-total
namespace: test
annotations:
metric-config.external.prometheus-query.prometheus/processed-requests-per-second: |
sum(rate(istio_requests_total{destination_workload_namespace="test",reporter="destination"}[1m]))
spec:
maxReplicas: 5
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
metrics:
- type: External
external:
metric:
name: prometheus-query
selector:
matchLabels:
query-name: processed-requests-per-second
target:
type: AverageValue
averageValue: "10"
執行如下命令部署這個HPA配置：
kubectl --kubeconfig "$USER_CONFIG" apply -f resources_hpa/requests_total_hpa.yaml
執行如下命令校驗：
kubectl --kubeconfig "$USER_CONFIG" get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
結果如下：
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "prometheus-query",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [
"get"
]
}
]
}
類似地，我們可以使用其他維度的應用級監控指標配置HPA。舉例如下，不再冗述。
3.2 根據平均延遲擴縮容
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo-latency-avg
namespace: test
annotations:
metric-config.external.prometheus-query.prometheus/latency-average: |
sum(rate(istio_request_duration_milliseconds_sum{destination_workload_namespace="test",reporter="destination"}[1m]))
/sum(rate(istio_request_duration_milliseconds_count{destination_workload_namespace="test",reporter="destination"}[1m]))
spec:
maxReplicas: 5
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
metrics:
- type: External
external:
metric:
name: prometheus-query
selector:
matchLabels:
query-name: latency-average
target:
type: AverageValue
averageValue: "0.005"
3.3 根據P95分佈擴縮容
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo-p95
namespace: test
annotations:
metric-config.external.prometheus-query.prometheus/p95-latency: |
histogram_quantile(0.95,sum(irate(istio_request_duration_milliseconds_bucket{destination_workload_namespace="test",destination_canonical_service="podinfo"}[5m]))by (le))
spec:
maxReplicas: 5
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
metrics:
- type: External
external:
metric:
name: prometheus-query
selector:
matchLabels:
query-name: p95-latency
target:
type: AverageValue
averageValue: "4"
驗證HPA
1 生成負載
執行如下命令產生實驗流量，以驗證HPA配置自動擴容生效。
alias k="kubectl --kubeconfig $USER_CONFIG"
loadtester=$(k -n test get pod -l "app=flagger-loadtester" -o jsonpath='{.items..metadata.name}')
k -n test exec -it ${loadtester} -c loadtester -- hey -z 5m -c 2 -q 10 http://podinfo:9898
這裏運行了一個持續5分鐘、QPS=10、併發數爲2的請求。
hey命令詳細參考如下：
Usage: hey [options...] <url>
Options:
-n Number of requests to run. Default is 200.
-c Number of workers to run concurrently. Total number of requests cannot
be smaller than the concurrency level. Default is 50.
-q Rate limit, in queries per second (QPS) per worker. Default is no rate limit.
-z Duration of application to send requests. When duration is reached,
application stops and exits. If duration is specified, n is ignored.
Examples: -z 10s -z 3m.
-o Output type. If none provided, a summary is printed.
"csv" is the only supported alternative. Dumps the response
metrics in comma-separated values format.
-m HTTP method, one of GET, POST, PUT, DELETE, HEAD, OPTIONS.
-H Custom HTTP header. You can specify as many as needed by repeating the flag.
For example, -H "Accept: text/html" -H "Content-Type: application/xml" .
-t Timeout for each request in seconds. Default is 20, use 0 for infinite.
-A HTTP Accept header.
-d HTTP request body.
-D HTTP request body from file. For example, /home/user/file.txt or ./file.txt.
-T Content-type, defaults to "text/html".
-a Basic authentication, username:password.
-x HTTP Proxy address as host:port.
-h2 Enable HTTP/2.
-host HTTP Host header.
-disable-compression Disable compression.
-disable-keepalive Disable keep-alive, prevents re-use of TCP
connections between different HTTP requests.
-disable-redirects Disable following of HTTP redirects
-cpus Number of used cpu cores.
(default for current machine is 4 cores)
2 自動擴容
執行如下命令觀察擴容情況：
watch kubectl --kubeconfig $USER_CONFIG -n test get hpa/podinfo-total
結果如下：
Every 2.0s: kubectl --kubeconfig /Users/han/shop_config/ack_zjk -n test get hpa/podinfo East6C16G: Tue Jan 26 18:01:30 2021
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
podinfo Deployment/podinfo 10056m/10 (avg) 1 5 2 4m45s
另外兩個HPA類似，命令如下：
kubectl --kubeconfig $USER_CONFIG -n test get hpa
watch kubectl --kubeconfig $USER_CONFIG -n test get hpa/podinfo-latency-avg
watch kubectl --kubeconfig $USER_CONFIG -n test get hpa/podinfo-p95
3 監控指標
同時，我們可以實時在Prometheus中查看相關的應用級監控指標的實時數據。示意如下：
本文爲阿里雲原創內容，未經允許不得轉載。