knative revision最終創建的deploy，會在pod中啓用一個sidecar容器，即queue-proxy。
下圖展示了queue-proxy的用途，業務流量首先進入 Istio Gateway，然後會轉發到 Queue-Proxy 的 8012 端口，Queue-Proxy 8012 再把請求轉發到業務容器的監聽端口。
queue-proxy用途
- 應用請求數等指標統計。
- Pod健康檢查（k8s探針）
- 代理轉發流量
- 判斷Ingress是否ready需要通過訪問queue-proxy實現
queue-proxy預留的端口
knative的組件預留了一些端口作爲服務端口，例如8012端口默認是給queue使用的，因此應用的端口應該避開這些預留的端口。
- 8012，queue-proxy代理的http端口，一般訪問應用流量的入口。
- 8013，http2端口，用於grpc流量的轉發。
- 8022，queue-proxy管理端口。
- 9090，queue-proxy容器的監控端口，圖中的端口已過時。數據由Autoscaler採集，用於kpa。
- 9091，queue-proxy採集到的應用監控指標（req請求數，響應市場等），圖中的端口已過時。由prometheus採集，用於計費。
serving/pkg/apis/networking/ports.go
// The ports we setup on our services.
const (
// ServiceHTTPPort is the port that we setup our Serving and Activator K8s services for
// HTTP/1 endpoints.
ServiceHTTPPort = 80
// ServiceHTTP2Port is the port that we setup our Serving and Activator K8s services for
// HTTP/2 endpoints.
ServiceHTTP2Port = 81
// BackendHTTPPort is the backend, i.e. `targetPort` that we setup for HTTP services.
BackendHTTPPort = 8012
// BackendHTTP2Port is the backend, i.e. `targetPort` that we setup for HTTP services.
BackendHTTP2Port = 8013
// QueueAdminPort specifies the port number for
// health check and lifecycle hooks for queue-proxy.
QueueAdminPort = 8022
// AutoscalingQueueMetricsPort specifies the port number for metrics emitted
// by queue-proxy for autoscaler.
AutoscalingQueueMetricsPort = 9090
// UserQueueMetricsPort specifies the port number for metrics emitted
// by queue-proxy for end user.
UserQueueMetricsPort = 9091
// ServicePortNameHTTP1 is the name of the external port of the service for HTTP/1.1
ServicePortNameHTTP1 = "http"
// ServicePortNameH2C is the name of the external port of the service for HTTP/2
ServicePortNameH2C = "http2"
)
默認情況通過queue-proxy的流量會轉發到應用的8080端口，實際上queue-proxy是從USER_PORT讀取的應用端口值。
func getUserPort(rev *v1alpha1.Revision) int32 {
ports := rev.Spec.GetContainer().Ports
if len(ports) > 0 && ports[0].ContainerPort != 0 {
return ports[0].ContainerPort
}
return v1alpha1.DefaultUserPort
}
這裏可以看出userport等於第一個端口值，因此knative只支持一個端口，使用kantive serving時建議顯式指出服務端口。
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
name: wordpress-serving
namespace: default
spec:
template:
metadata:
labels:
app: wordpress
annotations:
autoscaling.knative.dev/target: "100"
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/knative-sample/wordpress:5.2-20190524100810
ports:
- name: http1
containerPort: 80
env:
- name: WORDPRESS_DB_HOST
value: rm-2xx.mysql.rds.aliyuncs.com:3306
- name: WORDPRESS_DB_USER
value: wordpress
- name: WORDPRESS_DB_PASSWORD
value: xxx
imagePullPolicy: Always
這裏我們主要注意的是port的name和端口值。
name 字段表示使用的協議，有效值只有http1 和 h2c 兩個，其中：http1 表示使用 http1 協議，比如 web 服務和 websock 都可以使用 http1。grpc 需要設置成 h2c。
containerPort 字段就是容器提供服務的唯一端口。
應用請求指標統計
knative kpa根據應用的請求次數進行擴縮容，因此queue-proxy的一個主要功能就是爲kpa提供相應指標。
serving/pkg/queue/request_metric.go
func (h *requestMetricHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rr := pkghttp.NewResponseRecorder(w, http.StatusOK)
startTime := time.Now()
if h.breaker != nil {
h.statsReporter.ReportQueueDepth(h.breaker.InFlight())
}
defer func() {
// Filter probe requests for revision metrics.
if network.IsProbe(r) {
return
}
// If ServeHTTP panics, recover, record the failure and panic again.
err := recover()
latency := time.Since(startTime)
if err != nil {
h.sendRequestMetrics(http.StatusInternalServerError, latency)
panic(err)
}
h.sendRequestMetrics(rr.ResponseCode, latency)
}()
h.handler.ServeHTTP(rr, r)
}
指標統計對應用的訪問，對k8s probe和knative probe不做統計。
knative probe
queue-proxy爲knative服務提供探針能力。
- ingress的狀態依賴於底層pod的狀態，所以會使用probe探測pod是否ready。
請求中header包含K-Network-Probe=probe
serving/pkg/network/probe_handler.go
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if ph := r.Header.Get(ProbeHeaderName); ph != ProbeHeaderValue {
r.Header.Del(HashHeaderName)
h.next.ServeHTTP(w, r)
return
}
hh := r.Header.Get(HashHeaderName)
if hh == "" {
http.Error(w, fmt.Sprintf("a probe request must contain a non-empty %q header", HashHeaderName), http.StatusBadRequest)
return
}
w.Header().Set(HashHeaderName, hh)
w.WriteHeader(200)
}
- ativitor要等到底層pod爲ready纔會轉發流量，所以也利用了queue-proxy的probe功能。請求中header包含
K-Network-Probe=queue
serving/cmd/queue/main.go
func handleKnativeProbe(w http.ResponseWriter, r *http.Request, ph string, healthState *health.State, prober func() bool, isAggressive bool) {
_, probeSpan := trace.StartSpan(r.Context(), "probe")
defer probeSpan.End()
if ph != queue.Name {
http.Error(w, fmt.Sprintf(badProbeTemplate, ph), http.StatusBadRequest)
probeSpan.Annotate([]trace.Attribute{
trace.StringAttribute("queueproxy.probe.error", fmt.Sprintf(badProbeTemplate, ph))}, "error")
return
}
if prober == nil {
http.Error(w, "no probe", http.StatusInternalServerError)
probeSpan.Annotate([]trace.Attribute{
trace.StringAttribute("queueproxy.probe.error", "no probe")}, "error")
return
}
healthState.HandleHealthProbe(func() bool {
if !prober() {
probeSpan.Annotate([]trace.Attribute{
trace.StringAttribute("queueproxy.probe.error", "container not ready")}, "error")
return false
}
return true
}, isAggressive, w)
}
k8s probe
k8s的probe功能，從應用容器上移到queue-proxy，爲pod配置的probe實際配置給了queue-proxy，然後由queue-proxy轉發給應用容器。
serving/cmd/queue/main.go
func handler(reqChan chan queue.ReqEvent, breaker *queue.Breaker, handler http.Handler,
healthState *health.State, prober func() bool, isAggressive bool) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
if network.IsKubeletProbe(r) {
handler.ServeHTTP(w, r)
return
}
if ph := network.KnativeProbeHeader(r); ph != "" {
handleKnativeProbe(w, r, ph, healthState, prober, isAggressive)
return
}
proxyCtx, proxySpan := trace.StartSpan(r.Context(), "proxy")
defer proxySpan.End()
// Metrics for autoscaling.
in, out := queue.ReqIn, queue.ReqOut
if activator.Name == network.KnativeProxyHeader(r) {
in, out = queue.ProxiedIn, queue.ProxiedOut
}
reqChan <- queue.ReqEvent{Time: time.Now(), EventType: in}
defer func() {
reqChan <- queue.ReqEvent{Time: time.Now(), EventType: out}
}()
network.RewriteHostOut(r)
// Enforce queuing and concurrency limits.
if breaker != nil {
if err := breaker.Maybe(r.Context(), func() {
handler.ServeHTTP(w, r.WithContext(proxyCtx))
}); err != nil {
switch err {
case context.DeadlineExceeded, queue.ErrRequestQueueFull:
http.Error(w, err.Error(), http.StatusServiceUnavailable)
default:
w.WriteHeader(http.StatusInternalServerError)
}
}
} else {
handler.ServeHTTP(w, r.WithContext(proxyCtx))
}
}
}
參考：https://yq.aliyun.com/articles/722412?spm=a2c4e.11155435.0.0.539af84bEVQCgz