凌晨 12 點突發 Istio 生產事故！一頓操作猛如虎解決了

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"事故起因","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務上新集羣，本來以爲\"灑灑水\"，11 點切，12 點就能在家睡覺了。流量切過來後，在驗證過程中，發現網頁能夠正常打開，在登錄時返回了 502，當場懵逼。在相關的容器日誌發現一個高頻的報錯條目“7000 端口無法連接”，向業務組瞭解到這是 redis 集羣中的一個端口，前後端是通過 redis 交互的，該集羣同時還有 7001-7003 其它三個端口。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用 nc 命令對 redis 集羣進行連接測試：向服務端發送 keys * 命令時，7000 端口返回的是 HTTP/1.1 400 Bad Request，其他三個端口是 redis 返回的 -NOAUTH Authentication required。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" $ nc 10.0.0.6 7000\n keys *\n HTTP/1.1 400 Bad Request\n content-length: 0\n connection: close\n\n $ nc 10.0.0.6 7003\n keys *\n -NOAUTH Authentication required\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"判斷 7000 端口連接到了其他應用上，至少不是 redis。在宿主機上抓包發現沒有抓到訪問 7000 端口的流量，然後查看容器的 nf_conntrackb 表，發現 7000 端口的數據只有到本地的會話信息；7003 的有兩條會話信息，一條到本機的，一條到目標服務器的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" $ grep 7000 /proc/net/nf_conntrack\n ipv4 2 tcp 6 110 TIME_WAIT src=10.64.192.14 dst=10.0.0.6 sport=50498 dport=7000 src=127.0.0.1 dst=10.64.192.14 sport=15001 dport=50498 [ASSURED] mark=0 zone=0 use=2\n\n $ grep 7003 /proc/net/nf_conntrack\n ipv4 2 tcp 6 104 TIME_WAIT src=10.64.192.14 dst=10.0.0.6 sport=38952 dport=7003 src=127.0.0.1 dst=10.64.192.14 sport=15001 dport=38952 [ASSURED] mark=0 zone=0 use=2\n ipv4 2 tcp 6 104 TIME_WAIT src=10.64.192.14 dst=10.0.0.6 sport=38954 dport=7003 src=10.0.0.6 dst=10.64.192.14 sport=7003 dport=38954 [ASSURED] mark=0 zone=0 use=2\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由此判斷出 istio 沒有代理轉發出 7000 的流量，這突然就觸及到了我的知識盲區，一大堆人看着，辦公室 26 度的空調，一直在冒汗。沒辦法了，在與業務商量後，只能先關閉 istio 注入，優先恢復了業務。回去後惡補 istio 的相關資料。終於將問題解決。記錄下相關信息，以供日後參考。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"背景知識補充","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"istio 的 Sidecar 有兩種模式：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ALLOW_ANY：istio 代理允許調用未知的服務，黑名單模式。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"REGISTRY_ONLY：istio 代理會阻止任何沒有在網格中定義的 HTTP 服務或 service entry 的主機，白名單模式。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"istio-proxy（Envoy）的配置結構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"istio-proxy（Envoy）的代理信息大體由以下幾個部分組成：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cluster：在 Envoy 中，Cluster 是一個服務集羣，Cluster 中包含一個到多個 endpoint，每個 endpoint 都可以提供服務，Envoy 根據負載均衡算法將請求發送到這些 endpoint 中。Cluster 分爲 inbound 和 outbound 兩種，前者對應 Envoy 所在節點上的服務；後者佔了絕大多數，對應 Envoy 所在節點的外部服務。可以使用如下方式分別查看 inbound 和 outbound 的 Cluster。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Listeners：Envoy 採用 listener 來接收並處理 downstream 發過來的請求，可以直接與 Cluster 關聯，也可以通過 rds 配置路由規則(Routes)，然後在路由規則中再根據不同的請求目的地對請求進行精細化的處理。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Routes：配置 Envoy 的路由規則。istio 下發的缺省路由規則中對每個端口(服務)設置了一個路由規則，根據 host 來對請求進行路由分發，routes 的目的爲其他服務的 Cluster。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Endpoint：Cludter 對應的後端服務，可以通過 istio pc endpoint 查看 inbound 和 outbound 對應的 endpoint 信息。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"服務發現類型","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cluster 的服務發現類型主要有：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ORIGINAL_DST：ORIGINAL_DST 類型的 Cluster，Envoy 在轉發請求時會直接採用 downstream 請求中的原始目的地 IP 地址。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EDS：EDS 獲取到該 Cluster 中所有可用的 Endpoint，並根據負載均衡算法（缺省爲 Round Robin）將 Downstream 發來的請求發送到不同的 Endpoint。istio 會自動爲集羣中的 service 創建代理信息，listener 的信息從 service 獲取，對應的 Cluster 被標記爲 EDS 類型。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"STATIC：缺省值，在集羣中列出所有可代理的主機 Endpoints。當沒有內容爲空時，不進行轉發。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"LOGICAL_DNS：Envoy 使用 DNS 添加主機，但如果 DNS 不再返回時，也不會丟棄。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"STRICT_DNS：Envoy 將監控 DNS，而每個匹配的 A 記錄都將被認爲是有效的。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"兩個特殊集羣","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BlackHoleCluster：黑洞集羣，匹配此集羣的流量將被不會被轉發。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" {\n \"name\": \"BlackHoleCluster\",\n \"type\": \"STATIC\",\n \"connectTimeout\": \"10s\"\n }\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"類型爲 static，但是沒有指定可代理的 Endpoint，所以流量不會被轉發。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PassthroughCluster：透傳集羣，匹配此集羣的流量數據包的目的 IP 不會改變。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" {\n \"name\": \"PassthroughCluster\",\n \"type\": \"ORIGINAL_DST\",\n \"connectTimeout\": \"10s\",\n \"lbPolicy\": \"CLUSTER_PROVIDED\",\n \"circuitBreakers\": {\n \"thresholds\": [\n {\n \"maxConnections\": 4294967295,\n \"maxPendingRequests\": 4294967295,\n \"maxRequests\": 4294967295,\n \"maxRetries\": 4294967295\n }\n ]\n }\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"類型爲 original_dst，流量將原樣轉發。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一個特殊的 Listener","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"istio 中有一個特殊的 Listener 叫 virtualOutbound，定義如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"virtualOutbound：每個 Sidecar 都有一個綁定到 0.0.0.0:15001 的 listener，該 listener 下關聯了許多 virtual listener。iptables 會先將所有出站流量導入該 listener，該 listener 有一個字段 useOriginalDst 設置爲 true，表示會使用最佳匹配原始目的地的方式將請求分發到 virtual listener，如果沒有找到任何 virtual listener，將會直接發送到數據包原目的地的 PassthroughCluster。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"useOriginalDst 字段的具體意義是，如果使用 iptables 重定向連接，則代理接收流量的目標地址可能與原始目標地址不同。當此標誌設置爲 true 時，偵聽器會將重定向流量轉交給與原始目標地址關聯的偵聽器。如果沒有與原始目標地址關聯的偵聽器，則流量由接收它的偵聽器處理。默認爲 false。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"virtualOutbound 的流量處理流程如圖所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7b/7b5c47be87a74349b9d09f016d185f81.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是 virtualOutbound 的部分配置：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" {\n \"name\": \"envoy.tcp_proxy\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy\",\n \"statPrefix\": \"PassthroughCluster\",\n \"cluster\": \"PassthroughCluster\"\n }\n }\n ……………\n \"useOriginalDst\": true\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"istio 的 outbond 流量處理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開啓流量治理後，pod 訪問外部資源的流量轉發路徑如圖所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fd/fd458585ac8051a2e5a24502d80da984.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"istio 注入後 istio-proxy 有一個監聽在 15001 的端口，所有非 istio-proxy 用戶進程產生的 outbond 流量，通過 iptables 規則被重定向到 15001。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" # Sidecar 注入的 pod 監聽的端口\n $ ss -tulnp\n State Recv-Q Send-Q Local Address:Port Peer Address:Port\n LISTEN 0 128 *:80 *:*\n LISTEN 0 128 *:15090 *:*\n LISTEN 0 128 127.0.0.1:15000 *:*\n LISTEN 0 128 *:15001 *:*\n LISTEN 0 128 *:15006 *:*\n LISTEN 0 128 [::]:15020 [::]:*\n\n # Pod 內部的 iptables 表項\n $ iptables-save\n # Generated by iptables-save v1.4.21 on Fri Sep 17 13:47:09 2021\n *nat\n :PREROUTING ACCEPT [129886:7793160]\n :INPUT ACCEPT [181806:10908360]\n :OUTPUT ACCEPT [53409:3257359]\n :POSTROUTING ACCEPT [53472:3261139]\n :istio_INBOUND - [0:0]\n :istio_IN_REDIRECT - [0:0]\n :istio_OUTPUT - [0:0]\n :istio_REDIRECT - [0:0]\n -A PREROUTING -p tcp -j istio_INBOUND\n -A OUTPUT -p tcp -j istio_OUTPUT\n -A istio_INBOUND -p tcp -m tcp --dport 22 -j RETURN\n -A istio_INBOUND -p tcp -m tcp --dport 15020 -j RETURN\n -A istio_INBOUND -p tcp -j istio_IN_REDIRECT\n -A istio_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006\n -A istio_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN\n -A istio_OUTPUT ! -d 127.0.0.1/32 -o lo -j istio_IN_REDIRECT\n -A istio_OUTPUT -m owner --uid-owner 1337 -j RETURN\n -A istio_OUTPUT -m owner --gid-owner 1337 -j RETURN\n -A istio_OUTPUT -d 127.0.0.1/32 -j RETURN\n -A istio_OUTPUT -j istio_REDIRECT\n -A istio_REDIRECT -p tcp -j REDIRECT --to-ports 15001\n COMMIT\n # Completed on Fri Sep 17 13:47:09 2021\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"istio-proxy 收到流量後，大致的處理步驟如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3b/3b9f74824eaa1beb587e9ff428d80b3e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Proxy 在 ALLOW_ANY 模式下沒有匹配上 listener 將被直接轉發。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"listener 關聯了 type 爲 ORIGINAL_DST 的 Cluster 將使用原始請求種的 IP 地址。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"匹配上了 BlackHoleCluster，將不會被轉發。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"被代理流量的匹配步驟大致如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/73/73adb9516df06d04dec1862884d492ca.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"疑問：isito 爲 svc 創建的 listener 地址是全零的，集羣內部的端口是會存在複用的，那 istio 到底是怎麼區分流量的呢？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關鍵就在於 route，route 由 virtual_host 條目組成，這些 virtual_host 條目就是根據 svc 的信息生成的，訪問集羣內部的 svc 時，在 route 裏可以根據域名或者 svc 對應的 virtual_ip 進行精確匹配，所以完全不需要擔心啦。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" $ kubectl get svc -A | grep 8001\n NodePort 10.233.34.158 8001:30333/TCP 8d\n NodePort 10.233.9.105 8001:31717/TCP 8d\n NodePort 10.233.60.59 8001:31135/TCP 2d16h\n NodePort 10.233.18.212 8001:32407/TCP 8d\n NodePort 10.233.15.5 8001:30079/TCP 8d\n NodePort 10.233.59.21 8001:31103/TCP 8d\n NodePort 10.233.17.123 8001:31786/TCP 8d\n NodePort 10.233.9.196 8001:32662/TCP 8d\n NodePort 10.233.62.85 8001:32104/TCP 8d\n ClusterIP 10.233.49.245 8000/TCP,8001/TCP,8443/TCP,8444/TCP\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是 route 下的 virtual_host 條目：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" {\n \"name\": \"8001\",\n \"virtualHosts\": [\n {\n \"name\": \"merchant-center.open.svc.cluster.local:8001\",\n \"domains\": [\n \"merchant-center.open.svc.cluster.local\",\n \"merchant-center.open.svc.cluster.local:8001\",\n \"merchant-center.open\",\n \"merchant-center.open:8001\",\n \"merchant-center.open.svc.cluster\",\n \"merchant-center.open.svc.cluster:8001\",\n \"merchant-center.open.svc\",\n \"merchant-center.open.svc:8001\",\n \"10.233.60.59\",\n \"10.233.60.59:8001\"\n ],\n \"routes\": [\n {\n \"name\": \"default\",\n \"match\": {\n \"prefix\": \"/\"\n },\n \"route\": {\n \"cluster\": \"outbound|8001||merchant-center.open.svc.cluster.local\",\n \"timeout\": \"0s\",\n \"retryPolicy\": {\n \"retryOn\": \"connect-failure,refused-stream,unavailable,cancelled,resource-exhausted,retriable-status-codes\",\n \"numRetries\": 2,\n \"retryHostPredicate\": [\n {\n \"name\": \"envoy.retry_host_predicates.previous_hosts\"\n }\n ],\n \"hostSelectionRetryMaxAttempts\": \"5\",\n \"retriableStatusCodes\": [\n 503\n ]\n },\n \"maxGrpcTimeout\": \"0s\"\n },\n …………………\n {\n \"name\": \"cashier-busi-svc.pay.svc.cluster.local:8001\",\n \"domains\": [\n \"cashier-busi-svc.pay.svc.cluster.local\",\n \"cashier-busi-svc.pay.svc.cluster.local:8001\",\n \"cashier-busi-svc.pay\",\n \"cashier-busi-svc.pay:8001\",\n \"cashier-busi-svc.pay.svc.cluster\",\n \"cashier-busi-svc.pay.svc.cluster:8001\",\n \"cashier-busi-svc.pay.svc\",\n \"cashier-busi-svc.pay.svc:8001\",\n \"10.233.17.123\",\n \"10.233.17.123:8001\"\n ],\n …………………\n {\n \"name\": \"center-job.manager.svc.cluster.local:8001\",\n \"domains\": [\n \"center-job.manager.svc.cluster.local\",\n \"center-job.manager.svc.cluster.local:8001\",\n \"center-job.manager\",\n \"center-job.manager:8001\",\n \"center-job.manager.svc.cluster\",\n \"center-job.manager.svc.cluster:8001\",\n \"center-job.manager.svc\",\n \"center-job.manager.svc:8001\",\n \"10.233.34.158\",\n \"10.233.34.158:8001\"\n ],\n ……………\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"問題分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上信息，對集羣內的 svc 進行端口過濾，終於發現了集羣中存在使用了 7000 端口的 service：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e1/e1c9bb1a895898a67d71c316fd7cd620.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用7000端口的svc","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"istio 會爲 10.233.0.115:7000 自動生成一個 0.0.0.0:7000 的 listener:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" ADDRESS PORT TYPE\n 0.0.0.0 7000 TCP\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看詳細配置信息，在該 listener 中對於 tcp 流量是不轉發（BlackHoleCluster），所以目標地址爲 10.0.x.x:7000 的流量被 listener_0.0.0.0:7000 匹配到時，因爲是 tcp 的流量(nc 命令默認 tcp 協議)，所以代理沒有對該流量進行轉發。這與開頭提到的 pod 沒有流量發出來現象一致。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" {\n \"name\": \"0.0.0.0_7000\",\n \"address\": {\n \"socketAddress\": {\n \"address\": \"0.0.0.0\",\n \"portValue\": 7000\n }\n },\n \"filterChains\": [\n {\n \"filterChainMatch\": {\n \"prefixRanges\": [\n {\n \"addressPrefix\": \"10.64.x.x\",\n \"prefixLen\": 32\n }\n ]\n },\n \"filters\": [\n {\n \"name\": \"envoy.tcp_proxy\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy\",\n \"statPrefix\": \"BlackHoleCluster\",\n \"cluster\": \"BlackHoleCluster\"\n }\n }\n ]\n }\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至於 7001-7003 爲什麼能通，是因爲 istio-proxy 默認使用的是 ALLOW_ANY 模式，對於沒有匹配上 listener 的流量是直接放行。可以通過 istio_configmap 配置信息來驗證一下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" $ kubectl get cm istio -n istio-system -o yaml | grep -i -w -a3 \"mode\"\n # REGISTRY_ONLY - restrict outbound traffic to services defined in the service registry as well\n # as those defined through ServiceEntries\n outboundTrafficPolicy:\n mode: ALLOW_ANY\n localityLbSetting:\n enabled: true\n # The namespace to treat as the administrative root namespace for istio\n --\n drainDuration: 45s\n parentShutdownDuration: 1m0s\n #\n # The mode used to redirect inbound connections to Envoy. This setting\n # has no effect on outbound traffic: iptables REDIRECT is always used for\n # outbound connections.\n # If \"REDIRECT\", use iptables REDIRECT to NAT and redirect to Envoy.\n # The \"REDIRECT\" mode loses source addresses during redirection.\n # If \"TPROXY\", use iptables TPROXY to redirect to Envoy.\n # The \"TPROXY\" mode preserves both the source and destination IP\n # addresses and ports, so that they can be used for advanced filtering\n # and manipulation.\n # The \"TPROXY\" mode also configures the Sidecar to run with the\n # CAP_NET_ADMIN capability, which is required to use TPROXY.\n #interceptionMode: REDIRECT\n #\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"解決方案","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後我們來解決開頭提到的問題，總共有三種解決方案。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"方法 1：Service Entry","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務條目（Service Entry）是 istio 重要的資源對象之一，作用是將外部的資源註冊到 istio 內部的網格服務中來，以提供網格內對外部資源的更加精細化的控制。我們可以簡單理解爲白名單，istios 根據 Service Entry 的內容生成 listeners。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們在命名空間 dev-self-pc-ct 中添加如下配置：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" $ kubectl apply -f - <\n Port: tcp-7001 7001/TCP\n TargetPort: 7001/TCP\n Endpoints: \n Port: tcp-7002 7002/TCP\n TargetPort: 7002/TCP\n Endpoints: \n Port: tcp-7003 7003/TCP\n TargetPort: 7003/TCP\n Endpoints: \n Session Affinity: None\n Events: \n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Listener 部分信息如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" {\n \"name\": \"10.233.59.159_7000\",\n \"address\": {\n \"socketAddress\": {\n \"address\": \"10.233.59.159\",\n \"portValue\": 7000\n }\n },\n \"filterChains\": [\n {\n \"filters\": [\n {\n \"name\": \"mixer\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig\",\n \"transport\": {\n \"networkFailPolicy\": {\n \"policy\": \"FAIL_CLOSE\",\n \"baseRetryWait\": \"0.080s\",\n \"maxRetryWait\": \"1s\"\n },\n \"checkCluster\": \"outbound|9091||istio-policy.istio-system.svc.cluster.local\",\n \"reportCluster\": \"outbound|9091||istio-telemetry.istio-system.svc.cluster.local\",\n \"reportBatchMaxEntries\": 100,\n \"reportBatchMaxTime\": \"1s\"\n },\n \"mixerAttributes\": {\n \"attributes\": {\n \"context.proxy_version\": {\n \"stringValue\": \"1.4.8\"\n },\n ......\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該 listener 指向了一個 Cluster：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" {\n \"name\": \"envoy.tcp_proxy\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy\",\n \"statPrefix\": \"outbound|7000||redis\",\n \"cluster\": \"outbound|7000||redis\"\n }\n }\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對應的 service 信息如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/59/5913d14aac0a6b342058142f8e896c3e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到 endpoint 就是剛纔我們指定的外部服務器地址：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2f/2f21a36edf3e84722c189f2a9418a1a2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"進行訪問測試：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ce/ceb5b5b950acaa5e74c2346ed41955d6.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"已經可以正常訪問了。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後我們來比較一下這三種方法。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法 1：通過添加 ServiceEntry，以允許訪問外部服務。可以讓你使用 istio 服務網格所有的功能去調用集羣內或集羣外的服務，這是官方推薦的方法。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法 2：直接繞過了 istio Sidecar 代理，使你的服務可以直接訪問任意的外部服務。但是，以這種方式配置代理需要了解集羣提供商相關知識和配置。將失去對外部服務訪問的監控，並且無法將 istio 功能應用於外部服務的流量。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法 3：這個方法相對於其他兩種，配置有點複雜，同時還要通過 service 的方式來訪問外部服務，這意味着對於已經存在的應用需要進行改造。具體能否實施看實際情況。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法 1 的做法類似於“白名單”，不但能達到訪問外部服務的目的，並且可以像集羣內部服務一樣對待（可使用 istio 的流量控制功能）。另外，即使服務受到入侵，由於“白名單”的設置入侵者也無法（或較難）將流量回傳到入侵機器，進一步保證了服務的安全性；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法 2 直接繞過了 istio Sidecar 代理，使你的服務可以直接訪問任意的外部服務。但是，以這種方式配置代理需要了解集羣提供商相關知識和配置。你將失去對外部服務訪問的監控，並且無法將 istio 功能應用於外部服務的流量；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法 3 雖然也可以使用 istio 的流量控制功能來管理外部流量，但是在實際操作中會存在配置複雜、改造應用等問題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，強烈推薦大家使用方法一。最後，特別提醒一下大家。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"將 includeOutboundIPRanges 設置爲空是有問題的","attrs":{}},{"type":"text","text":"，這","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"相當於將所有的服務都配置代理繞行","attrs":{}},{"type":"text","text":"，那 Sidecar 就沒起作用了，沒了 Sidecar 的 istio 就沒有靈魂了。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"作者","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"龍小蝦 K8s 愛好者 負責維護 KubeSphere 容器平臺之上的容器化業務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文由博客一文多發平臺 ","attrs":{}},{"type":"link","attrs":{"href":"https://openwrite.cn?from=article_bottom","title":"","type":null},"content":[{"type":"text","text":"OpenWrite","attrs":{}}]},{"type":"text","text":" 發佈！","attrs":{}}]}],"attrs":{}}]}