knative revision最终创建的deploy，会在pod中启用一个sidecar容器，即queue-proxy。
下图展示了queue-proxy的用途，业务流量首先进入 Istio Gateway，然后会转发到 Queue-Proxy 的 8012 端口，Queue-Proxy 8012 再把请求转发到业务容器的监听端口。
queue-proxy用途
- 应用请求数等指标统计。
- Pod健康检查（k8s探针）
- 代理转发流量
- 判断Ingress是否ready需要通过访问queue-proxy实现
queue-proxy预留的端口
knative的组件预留了一些端口作为服务端口，例如8012端口默认是给queue使用的，因此应用的端口应该避开这些预留的端口。
- 8012，queue-proxy代理的http端口，一般访问应用流量的入口。
- 8013，http2端口，用于grpc流量的转发。
- 8022，queue-proxy管理端口。
- 9090，queue-proxy容器的监控端口，图中的端口已过时。数据由Autoscaler采集，用于kpa。
- 9091，queue-proxy采集到的应用监控指标（req请求数，响应市场等），图中的端口已过时。由prometheus采集，用于计费。
serving/pkg/apis/networking/ports.go
// The ports we setup on our services.
const (
// ServiceHTTPPort is the port that we setup our Serving and Activator K8s services for
// HTTP/1 endpoints.
ServiceHTTPPort = 80
// ServiceHTTP2Port is the port that we setup our Serving and Activator K8s services for
// HTTP/2 endpoints.
ServiceHTTP2Port = 81
// BackendHTTPPort is the backend, i.e. `targetPort` that we setup for HTTP services.
BackendHTTPPort = 8012
// BackendHTTP2Port is the backend, i.e. `targetPort` that we setup for HTTP services.
BackendHTTP2Port = 8013
// QueueAdminPort specifies the port number for
// health check and lifecycle hooks for queue-proxy.
QueueAdminPort = 8022
// AutoscalingQueueMetricsPort specifies the port number for metrics emitted
// by queue-proxy for autoscaler.
AutoscalingQueueMetricsPort = 9090
// UserQueueMetricsPort specifies the port number for metrics emitted
// by queue-proxy for end user.
UserQueueMetricsPort = 9091
// ServicePortNameHTTP1 is the name of the external port of the service for HTTP/1.1
ServicePortNameHTTP1 = "http"
// ServicePortNameH2C is the name of the external port of the service for HTTP/2
ServicePortNameH2C = "http2"
)
默认情况通过queue-proxy的流量会转发到应用的8080端口，实际上queue-proxy是从USER_PORT读取的应用端口值。
func getUserPort(rev *v1alpha1.Revision) int32 {
ports := rev.Spec.GetContainer().Ports
if len(ports) > 0 && ports[0].ContainerPort != 0 {
return ports[0].ContainerPort
}
return v1alpha1.DefaultUserPort
}
这里可以看出userport等于第一个端口值，因此knative只支持一个端口，使用kantive serving时建议显式指出服务端口。
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
name: wordpress-serving
namespace: default
spec:
template:
metadata:
labels:
app: wordpress
annotations:
autoscaling.knative.dev/target: "100"
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/knative-sample/wordpress:5.2-20190524100810
ports:
- name: http1
containerPort: 80
env:
- name: WORDPRESS_DB_HOST
value: rm-2xx.mysql.rds.aliyuncs.com:3306
- name: WORDPRESS_DB_USER
value: wordpress
- name: WORDPRESS_DB_PASSWORD
value: xxx
imagePullPolicy: Always
这里我们主要注意的是port的name和端口值。
name 字段表示使用的协议，有效值只有http1 和 h2c 两个，其中：http1 表示使用 http1 协议，比如 web 服务和 websock 都可以使用 http1。grpc 需要设置成 h2c。
containerPort 字段就是容器提供服务的唯一端口。
应用请求指标统计
knative kpa根据应用的请求次数进行扩缩容，因此queue-proxy的一个主要功能就是为kpa提供相应指标。
serving/pkg/queue/request_metric.go
func (h *requestMetricHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rr := pkghttp.NewResponseRecorder(w, http.StatusOK)
startTime := time.Now()
if h.breaker != nil {
h.statsReporter.ReportQueueDepth(h.breaker.InFlight())
}
defer func() {
// Filter probe requests for revision metrics.
if network.IsProbe(r) {
return
}
// If ServeHTTP panics, recover, record the failure and panic again.
err := recover()
latency := time.Since(startTime)
if err != nil {
h.sendRequestMetrics(http.StatusInternalServerError, latency)
panic(err)
}
h.sendRequestMetrics(rr.ResponseCode, latency)
}()
h.handler.ServeHTTP(rr, r)
}
指标统计对应用的访问，对k8s probe和knative probe不做统计。
knative probe
queue-proxy为knative服务提供探针能力。
- ingress的状态依赖于底层pod的状态，所以会使用probe探测pod是否ready。
请求中header包含K-Network-Probe=probe
serving/pkg/network/probe_handler.go
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if ph := r.Header.Get(ProbeHeaderName); ph != ProbeHeaderValue {
r.Header.Del(HashHeaderName)
h.next.ServeHTTP(w, r)
return
}
hh := r.Header.Get(HashHeaderName)
if hh == "" {
http.Error(w, fmt.Sprintf("a probe request must contain a non-empty %q header", HashHeaderName), http.StatusBadRequest)
return
}
w.Header().Set(HashHeaderName, hh)
w.WriteHeader(200)
}
- ativitor要等到底层pod为ready才会转发流量，所以也利用了queue-proxy的probe功能。请求中header包含
K-Network-Probe=queue
serving/cmd/queue/main.go
func handleKnativeProbe(w http.ResponseWriter, r *http.Request, ph string, healthState *health.State, prober func() bool, isAggressive bool) {
_, probeSpan := trace.StartSpan(r.Context(), "probe")
defer probeSpan.End()
if ph != queue.Name {
http.Error(w, fmt.Sprintf(badProbeTemplate, ph), http.StatusBadRequest)
probeSpan.Annotate([]trace.Attribute{
trace.StringAttribute("queueproxy.probe.error", fmt.Sprintf(badProbeTemplate, ph))}, "error")
return
}
if prober == nil {
http.Error(w, "no probe", http.StatusInternalServerError)
probeSpan.Annotate([]trace.Attribute{
trace.StringAttribute("queueproxy.probe.error", "no probe")}, "error")
return
}
healthState.HandleHealthProbe(func() bool {
if !prober() {
probeSpan.Annotate([]trace.Attribute{
trace.StringAttribute("queueproxy.probe.error", "container not ready")}, "error")
return false
}
return true
}, isAggressive, w)
}
k8s probe
k8s的probe功能，从应用容器上移到queue-proxy，为pod配置的probe实际配置给了queue-proxy，然后由queue-proxy转发给应用容器。
serving/cmd/queue/main.go
func handler(reqChan chan queue.ReqEvent, breaker *queue.Breaker, handler http.Handler,
healthState *health.State, prober func() bool, isAggressive bool) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
if network.IsKubeletProbe(r) {
handler.ServeHTTP(w, r)
return
}
if ph := network.KnativeProbeHeader(r); ph != "" {
handleKnativeProbe(w, r, ph, healthState, prober, isAggressive)
return
}
proxyCtx, proxySpan := trace.StartSpan(r.Context(), "proxy")
defer proxySpan.End()
// Metrics for autoscaling.
in, out := queue.ReqIn, queue.ReqOut
if activator.Name == network.KnativeProxyHeader(r) {
in, out = queue.ProxiedIn, queue.ProxiedOut
}
reqChan <- queue.ReqEvent{Time: time.Now(), EventType: in}
defer func() {
reqChan <- queue.ReqEvent{Time: time.Now(), EventType: out}
}()
network.RewriteHostOut(r)
// Enforce queuing and concurrency limits.
if breaker != nil {
if err := breaker.Maybe(r.Context(), func() {
handler.ServeHTTP(w, r.WithContext(proxyCtx))
}); err != nil {
switch err {
case context.DeadlineExceeded, queue.ErrRequestQueueFull:
http.Error(w, err.Error(), http.StatusServiceUnavailable)
default:
w.WriteHeader(http.StatusInternalServerError)
}
}
} else {
handler.ServeHTTP(w, r.WithContext(proxyCtx))
}
}
}
参考：https://yq.aliyun.com/articles/722412?spm=a2c4e.11155435.0.0.539af84bEVQCgz