Kubernetes POD IP会随POD的创建销毁动态变化，所以提出Service的方式访问POD网络，Kubernetes的Service可以使用Iptables实现也可以使用IPVS实现，本文简要分析Iptables实现方式。
部署POD，进入POD观察域名解析服务器地址：
Develop>kubectl exec -it -n default ncss-i-mysql-ha-0 /bin/bash
Defaulting container name to mysql-ha.
Use 'kubectl describe pod/ncss-i-mysql-ha-0 -n default' to see all of the containers in this pod.
[root@ncss-i-mysql-ha-0 /]# cat /etc/resolv.conf
nameserver 10.244.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[root@ncss-i-mysql-ha-0 /]#
/etc/resolv.conf文件配置说明：
nameserver：指明DNS服务器地址
search：当原始域名不能被DNS解析时，resolver会将该域名加上search指定的参数，重新请求DNS，直到被正确解析或试完search指定的列表为止
options：dns配置
ndots:5：所有DNS查询中，如果“.”的个数少于5个，则会根据search中配置的列表依次在对应域中先进行搜索，如果没有返回，则最后再直接查询域名本身
综上配置：
配置DNS服务器IP地址为“10.244.0.10”，首先检查域名是否带有5个“.”，如果没有就加上search所指定的域名开始查询
先猜想POD中的DNS服务器应该Kubernetes环境中的kube-proxy或者coredns，所以退出POD，在Kubernetes环境中找是否有“10.244.0.10”这个IP地址，但在pod和endpoint中都没出现对应这个IP地址。
Develop>kubectl get pod -o wide -A
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default ncss-i-mysql-ha-0 2/2 Running 6 2d2h 192.128.0.52 ncss-i <none> <none>
default ncss-i-mysql-ha-1 2/2 Running 6 2d2h 192.128.0.47 ncss-i <none> <none>
default ncss-i-nginx-gx8bm 1/1 Running 5 2d2h 192.128.0.55 ncss-i <none> <none>
default ncss-i-task-ngst8 1/1 Running 1 33h 192.128.0.44 ncss-i <none> <none>
default ncss-i-uwsgi-hr8sk 1/1 Running 0 5h13m 192.128.0.54 ncss-i <none> <none>
default ncss-i-vpp-dlx28 1/1 Running 3 2d2h 10.66.250.138 ncss-i <none> <none>
default ncss-i-vppcrd-hm2kv 1/1 Running 3 2d2h 192.128.0.42 ncss-i <none> <none>
kube-system coredns-6967fb4995-fnx28 1/1 Running 3 2d2h 192.128.0.48 ncss-i <none> <none>
kube-system coredns-6967fb4995-phs2j 1/1 Running 3 2d2h 192.128.0.51 ncss-i <none> <none>
kube-system elasticsearch-logging-0 1/1 Running 2 2d1h 192.128.0.43 ncss-i <none> <none>
kube-system elasticsearch-logging-1 1/1 Running 2 2d1h 192.128.0.49 ncss-i <none> <none>
kube-system fluentd-es-v2.5.2-p5tmn 1/1 Running 2 2d2h 192.128.0.50 ncss-i <none> <none>
kube-system kibana-logging-889fccc88-gwtvc 1/1 Running 2 2d2h 192.128.0.46 ncss-i <none> <none>
kube-system kube-apiserver-ncss-i 1/1 Running 3 2d2h 10.66.250.138 ncss-i <none> <none>
kube-system kube-controller-manager-ncss-i 1/1 Running 31 2d2h 10.66.250.138 ncss-i <none> <none>
kube-system kube-flannel-ds-amd64-z6k2g 1/1 Running 3 2d2h 10.66.250.138 ncss-i <none> <none>
kube-system kube-proxy-cpg7m 1/1 Running 3 2d2h 10.66.250.138 ncss-i <none> <none>
kube-system kube-scheduler-ncss-i 1/1 Running 31 2d2h 10.66.250.138 ncss-i <none> <none>
Develop>
Develop>
Develop>
Develop>kubectl get ep -A
NAMESPACE NAME ENDPOINTS AGE
default kubernetes 10.244.0.200:6443 2d2h
default mysql 192.128.0.47:3306,192.128.0.52:3306 2d2h
default mysql-read 192.128.0.47:3306,192.128.0.52:3306 2d2h
default nginx 192.128.0.55:10100 2d2h
default nginx-server 192.128.0.55:443,192.128.0.55:20103,192.128.0.55:20101 + 4 more... 2d2h
default uwsgi 192.128.0.54:8000,192.128.0.54:8001 5h13m
kube-system elasticsearch-logging 192.128.0.43:9200,192.128.0.49:9200 2d2h
kube-system kibana-logging 192.128.0.46:5601 2d2h
kube-system kube-controller-manager <none> 2d2h
kube-system kube-dns 192.128.0.48:53,192.128.0.51:53,192.128.0.48:53 + 3 more... 2d2h
kube-system kube-scheduler <none> 2d2h
Develop>
继续分析，可能在Iptables里面，使用“grep”开始搜索
Develop>iptables -t nat -S | grep 10.244.0.10
-A KUBE-SERVICES ! -s 192.128.0.0/16 -d 10.244.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.244.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 192.128.0.0/16 -d 10.244.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.244.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES ! -s 192.128.0.0/16 -d 10.244.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.244.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
Develop>
Develop>
Develop>iptables -t nat -S | grep KUBE-SVC-TCOU7JCQXEZGVUNU
-N KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.244.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -jKUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-SAYSLAX2SI43IZRU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-NZ4ZVJ6J3XCEJWKB
Develop>
Develop>
Develop>iptables -t nat -S | grep KUBE-SEP-NZ4ZVJ6J3XCEJWKB
-N KUBE-SEP-NZ4ZVJ6J3XCEJWKB
-A KUBE-SEP-NZ4ZVJ6J3XCEJWKB -s 192.128.0.51/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-NZ4ZVJ6J3XCEJWKB -p udp -m udp -j DNAT --to-destination 192.128.0.51:53
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-NZ4ZVJ6J3XCEJWKB
Develop>
Develop>
Develop>
这个IP地址就是配置在IPtables环境中，由于这个IP地址在PREROUTING链中作为目的地址进行匹配所以不能ping通，继续跟踪IPtables规则，发现规则最后把流量牵引到了“192.128.0.51:53”这个地址，结合endpoint内容可以判断这个配置属于Kubernetes的Service配置，然后再结合Service配置找到了这个Service的名字为kube-dns
Develop>kubectl get ep -A
NAMESPACE NAME ENDPOINTS AGE
default kubernetes 10.244.0.200:6443 2d2h
default mysql 192.128.0.47:3306,192.128.0.52:3306 2d2h
default mysql-read 192.128.0.47:3306,192.128.0.52:3306 2d2h
default nginx 192.128.0.55:10100 2d2h
default nginx-server 192.128.0.55:443,192.128.0.55:20103,192.128.0.55:20101 + 4 more... 2d2h
default uwsgi 192.128.0.54:8000,192.128.0.54:8001 5h20m
kube-system elasticsearch-logging 192.128.0.43:9200,192.128.0.49:9200 2d2h
kube-system kibana-logging 192.128.0.46:5601 2d2h
kube-system kube-controller-manager <none> 2d2h
kube-system kube-dns 192.128.0.48:53,192.128.0.51:53,192.128.0.48:53 + 3 more... 2d2h
kube-system kube-scheduler <none> 2d2h
Develop>
Develop>
Develop>kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.244.0.1 <none> 443/TCP 2d2h
default mysql ClusterIP None <none> 3306/TCP 2d2h
default mysql-read ClusterIP 10.244.27.22 <none> 3306/TCP 2d2h
default nginx ClusterIP 10.244.190.26 <none> 10100/TCP 2d2h
default nginx-server NodePort 10.244.155.83 <none> 443:443/TCP,20100:20100/TCP,20101:20101/TCP,20102:20102/TCP,20103:20103/TCP,10080:10080/TCP,10081:10081/TCP 2d2h
default uwsgi ClusterIP 10.244.144.63 <none> 8001/TCP,8000/TCP 5h20m
kube-system elasticsearch-logging ClusterIP 10.244.212.106 <none> 9200/TCP 2d2h
kube-system kibana-logging ClusterIP 10.244.117.119 <none> 5601/TCP 2d2h
kube-system kube-dns ClusterIP 10.244.0.10 <none> 53/UDP,53/TCP,9153/TCP 2d2h
Develop>
导出“kube-system”的yaml文件看到其中一个Service的Labels是“kube-dns”，并且它的Selector为“k8s-app: kube-dns”，继续导出coredns的yaml文件看到coredns的Labels有对应的“k8s-app: kube-dns”这个标签，以上就是Kubernetes Service的解析过程分析。
Develop>
Develop>kubectl get svc -n kube-system -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"elasticsearch-logging","kubernetes.io/cluster-service":"true","kubernetes.io/name":"Elasticsearch"},"name":"elasticsearch-logging","namespace":"kube-system"},"spec":{"ports":[{"port":9200,"protocol":"TCP","targetPort":"db"}],"selector":{"k8s-app":"elasticsearch-logging"}}}
creationTimestamp: "2019-09-02T09:56:16Z"
labels:
addonmanager.kubernetes.io/mode: Reconcile
k8s-app: elasticsearch-logging
kubernetes.io/cluster-service: "true"
kubernetes.io/name: Elasticsearch
name: elasticsearch-logging
namespace: kube-system
resourceVersion: "310"
selfLink: /api/v1/namespaces/kube-system/services/elasticsearch-logging
uid: 9f70b6ea-450b-4f6a-82ef-f80b4a3b01d8
spec:
clusterIP: 10.244.212.106
ports:
- port: 9200
protocol: TCP
targetPort: db
selector:
k8s-app: elasticsearch-logging
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
- apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kibana-logging","kubernetes.io/cluster-service":"true","kubernetes.io/name":"Kibana"},"name":"kibana-logging","namespace":"kube-system"},"spec":{"ports":[{"port":5601,"protocol":"TCP","targetPort":"ui"}],"selector":{"k8s-app":"kibana-logging"}}}
creationTimestamp: "2019-09-02T09:56:16Z"
labels:
addonmanager.kubernetes.io/mode: Reconcile
k8s-app: kibana-logging
kubernetes.io/cluster-service: "true"
kubernetes.io/name: Kibana
name: kibana-logging
namespace: kube-system
resourceVersion: "302"
selfLink: /api/v1/namespaces/kube-system/services/kibana-logging
uid: 59d4e98b-fa58-461d-a136-2e35814d554f
spec:
clusterIP: 10.244.117.119
ports:
- port: 5601
protocol: TCP
targetPort: ui
selector:
k8s-app: kibana-logging
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
- apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
creationTimestamp: "2019-09-02T09:56:10Z"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: KubeDNS
name: kube-dns
namespace: kube-system
resourceVersion: "206"
selfLink: /api/v1/namespaces/kube-system/services/kube-dns
uid: e875967b-0284-43b4-9dae-523ab56d24e4
spec:
clusterIP: 10.244.0.10
ports:
- name: dns
port: 53
protocol: UDP
targetPort: 53
- name: dns-tcp
port: 53
protocol: TCP
targetPort: 53
- name: metrics
port: 9153
protocol: TCP
targetPort: 9153
selector:
k8s-app: kube-dns
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Develop>
Develop>
Develop>
Develop>
Develop>
Develop>kubectl get pod -n kube-system coredns-6967fb4995-fnx28 -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2019-09-02T09:56:26Z"
generateName: coredns-6967fb4995-
labels:
k8s-app: kube-dns
pod-template-hash: 6967fb4995
name: coredns-6967fb4995-fnx28
namespace: kube-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: coredns-6967fb4995
uid: f4924630-a030-494b-981d-c652170c047c
resourceVersion: "75627"
selfLink: /api/v1/namespaces/kube-system/pods/coredns-6967fb4995-fnx28
uid: e71339f7-f8b4-46f7-9fa6-239e15e2bb05
spec:
containers:
- args:
- -conf
- /etc/coredns/Corefile
image: registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 5
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: coredns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8080
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/coredns
name: config-volume
readOnly: true
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: coredns-token-njjnj
readOnly: true
dnsPolicy: Default
enableServiceLinks: true
nodeName: ncss-i
nodeSelector:
beta.kubernetes.io/os: linux
priority: 2000000000
priorityClassName: system-cluster-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: coredns
serviceAccountName: coredns
terminationGracePeriodSeconds: 30
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
items:
- key: Corefile
path: Corefile
name: coredns
name: config-volume
- name: coredns-token-njjnj
secret:
defaultMode: 420
secretName: coredns-token-njjnj
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2019-09-02T09:56:35Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2019-09-03T02:53:46Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2019-09-03T02:53:46Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2019-09-02T09:56:35Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://a88841bec55520e4e6da31650d17f6ad1afa3bc4fe3f303dd996f994cff24bc6
image: registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1
imageID: docker://sha256:eb516548c180f8a6e0235034ccee2428027896af16a509786da13022fe95fe8c
lastState:
terminated:
containerID: docker://26acb782fbda7dae034b7aa20aa4342f06d298ef24906e4f7d66ddda786db2ed
exitCode: 2
finishedAt: "2019-09-03T02:51:02Z"
reason: Error
startedAt: "2019-09-02T10:32:37Z"
name: coredns
ready: true
restartCount: 3
state:
running:
startedAt: "2019-09-03T02:53:25Z"
hostIP: 10.66.250.138
phase: Running
podIP: 192.128.0.48
qosClass: Burstable
startTime: "2019-09-02T09:56:35Z"
Develop>
如果要宿主机使用Kubernetes的coredns解析K8S POD的DNS怎么办？
1、获取“kube-dns” Service对应的ClusterIP地址
Develop>kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-logging ClusterIP 10.244.212.106 <none> 9200/TCP 2d2h
kibana-logging ClusterIP 10.244.117.119 <none> 5601/TCP 2d2h
kube-dns ClusterIP 10.244.0.10 <none> 53/UDP,53/TCP,9153/TCP 2d2h
Develop>
2、把kube-dns Servce的Cluster IP地址写入系统的“cat /etc/resolv.conf”文件，并把search 配置来和Kubernetes环境中POD 中的“/etc/resolv.conf”的search 一致
Develop>cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 114.114.114.114
nameserver 10.244.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
options timeout:2 #域名解析超时时间
options attempts:2 #使用以上服务器重复解析次数