前言
最近在研究metallb，發現metallb轉發啓動後，容器沒法ping通，排查過程中發現CLUSTER-IP和EXTERNAL-IP也無法ping通，就確定和metallb無關，是kubernetes的問題。
排查過程
查看svc狀態正常
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 1d
nginx LoadBalancer 10.68.137.102 192.168.21.211 80:30145/TCP 1h
external ip通過metallb生成，但是無法ping通，curl正常
# ping 192.168.21.211
PING 192.168.21.211 (192.168.21.211) 56(84) bytes of data.
From 192.168.21.41 icmp_seq=1 Destination Host Unreachable
From 192.168.21.41 icmp_seq=2 Destination Host Unreachable
From 192.168.21.41 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.21.211 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3066ms
ping external ip，tcpdump能看到request但是沒有回覆
# tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:48:49.219904 IP 192.168.22.131 > 192.168.21.211: ICMP echo request, id 6874, seq 1, length 64
16:48:50.234166 IP 192.168.22.131 > 192.168.21.211: ICMP echo request, id 6874, seq 2, length 64
應該和metallb無關，因爲cluster ip仍然無法ping通，但curl正常
# ping 10.68.137.102
PING 10.68.137.102 (10.68.137.102) 56(84) bytes of data.
^C
--- 10.68.137.102 ping statistics ---
12 packets transmitted, 0 received, 100% packet loss, time 11264ms
定位過程
curl抓包,明顯收到了但是沒有回覆，有可能是被防火牆攔截了：
$ tcpdump -i eth0 host 192.168.22.131 and ! port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:17:00.248486 IP 192.168.22.131.43928 > 192.168.21.211.http: Flags [S], seq 3672218994, win 29200, options [mss 1460,sackOK,TS val 3480451309 ecr 0,nop,wscale 7], length 0
17:17:00.249000 IP 192.168.21.211.http > 192.168.22.131.43928: Flags [S.], seq 3948421548, ack 3672218995, win 28960, options [mss 1460,sackOK,TS val 24509624 ecr 3480451309,nop,wscale 7], length 0
17:17:00.249206 IP 192.168.22.131.43928 > 192.168.21.211.http: Flags [.], ack 1, win 229, options [nop,nop,TS val 3480451310 ecr 24509624], length 0
17:17:00.249273 IP 192.168.22.131.43928 > 192.168.21.211.http: Flags [P.], seq 1:79, ack 1, win 229, options [nop,nop,TS val 3480451310 ecr 24509624], length 78: HTTP: GET / HTTP/1.1
17:17:00.249290 IP 192.168.21.211.http > 192.168.22.131.43928: Flags [.], ack 79, win 227, options [nop,nop,TS val 24509624 ecr 3480451310], length 0
17:17:00.249387 IP 192.168.21.211.http > 192.168.22.131.43928: Flags [P.], seq 1:239, ack 79, win 227, options [nop,nop,TS val 24509624 ecr 3480451310], length 238: HTTP: HTTP/1.1 200 OK
17:17:00.249432 IP 192.168.21.211.http > 192.168.22.131.43928: Flags [P.], seq 239:851, ack 79, win 227, options [nop,nop,TS val 24509624 ecr 3480451310], length 612: HTTP
17:17:00.249568 IP 192.168.22.131.43928 > 192.168.21.211.http: Flags [.], ack 239, win 237, options [nop,nop,TS val 3480451310 ecr 24509624], length 0
17:17:00.249604 IP 192.168.22.131.43928 > 192.168.21.211.http: Flags [.], ack 851, win 247, options [nop,nop,TS val 3480451310 ecr 24509624], length 0
17:17:00.249936 IP 192.168.22.131.43928 > 192.168.21.211.http: Flags [F.], seq 79, ack 851, win 247, options [nop,nop,TS val 3480451310 ecr 24509624], length 0
17:17:00.249972 IP 192.168.21.211.http > 192.168.22.131.43928: Flags [F.], seq 851, ack 80, win 227, options [nop,nop,TS val 24509624 ecr 3480451310], length 0
17:17:00.250116 IP 192.168.22.131.43928 > 192.168.21.211.http: Flags [.], ack 852, win 247, options [nop,nop,TS val 3480451311 ecr 24509624], length 0
排查發現iptables中有如下兩條規則，規則針對tcp，並沒找到針對icmp的協議：
-A KUBE-SERVICES -d 10.68.137.102/32 -p tcp -m comment --comment "default/nginx:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-P4Q3KNUAWJVP4ILH
-A KUBE-SERVICES -d 192.168.21.211/32 -p tcp -m comment --comment "default/nginx:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-P4Q3KNUAWJVP4ILH
因此添加如下規則，接受icmp包：
$ iptables -A KUBE-SERVICES -d 192.168.21.211/32 -p icmp -m comment --comment "default/nginx:http loadbalancer IP" -m icmp --icmp-type Echo-Reply -j KUBE-FW-P4Q3KNUAWJVP4ILH
之後就能ping通：
$ ping 192.168.21.211
PING 192.168.21.211 (192.168.21.211) 56(84) bytes of data.
From 192.168.21.41: icmp_seq=2 Redirect Host(New nexthop: 192.168.21.211)
From 192.168.21.41: icmp_seq=3 Redirect Host(New nexthop: 192.168.21.211)
From 192.168.21.41: icmp_seq=4 Redirect Host(New nexthop: 192.168.21.211)
From 192.168.21.41: icmp_seq=5 Redirect Host(New nexthop: 192.168.21.211)
^C
--- 192.168.21.211 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4077ms
研究iptables相關，有如下規則：
[1]
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-SEP-7LRIL522ZHPCSXHI -s 172.20.113.8/32 -m comment --comment "default/nginx:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-7LRIL522ZHPCSXHI -p tcp -m comment --comment "default/nginx:http" -m tcp -j DNAT --to-destination 172.20.113.8:80
-A KUBE-SVC-P4Q3KNUAWJVP4ILH -m comment --comment "default/nginx:http" -j KUBE-SEP-7LRIL522ZHPCSXHI
[2]
-A KUBE-FW-P4Q3KNUAWJVP4ILH -m comment --comment "default/nginx:http loadbalancer IP" -j KUBE-MARK-MASQ
規則[2]表明了KUBE-FW-P4Q3KNUAWJVP4ILH規則直接應用KUBE-MARK-MASQ，而KUBE-SVC-P4Q3KNUAWJVP4ILH規則最終也會應用到KUBE-MARK-MASQ，兩者殊途同歸。
而KUBE-MARK-MASQ做的事情只是給packet打上標記
之後會被如下兩條規則處理，是允許接收，其match到mark 0x4000/0x4000就接收該數據包，說明之前icmp包未通過KUBE-MARK-MASQ應用到set-mark的規則，導致ICMP包直接被drop掉了
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE