k8s集群部署DNS服务发现

环境：

• os: centos
• kubernetes: 1.5.2
• docker: 1.12.5

一、skydns-rc.yaml 文件

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v9
  namespace: default
  labels:
    k8s-app: kube-dns
    version: v9
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v9
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v9
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: 192.168.100.90:5000/duni/etcd-amd64:3.0.17
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        command:
        - /usr/local/bin/etcd
       # - --privileged=true
        - -data-dir
        - /home/data/etcd
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - mountPath: /home/data/etcd
          name: etcd-storage
      - name: kube2sky
        #image: gcr.io/google_containers/kube2sky:1.11
        image: 192.168.100.90:5000/duni/kube2sky:1.14
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        args:
        # command = "/kube2sky"
       # - -etcd-server=http://127.0.0.1:4001
        #- -kube_master_url=http://172.27.8.210:8080
        - --kube-master-url=http://192.168.100.27:8080
        - --domain=cluster.local
      - name: skydns
        #image: gcr.io/google_containers/skydns:2015-03-11-001
        image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8c
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://127.0.0.1:2379
        - -addr=0.0.0.0:53
        - -ns-rotate=false
        - -domain=cluster.local
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 1
          timeoutSeconds: 5
      - name: healthz
        #image: gcr.io/google_containers/exechealthz:1.0
        image: 192.168.100.90:5000/duni/exechealthz-amd64:latest
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default  # Don't use cluster DNS.

skydns-rc.yaml文件中需要根据自己的实际情况进行相关修改

1、把镜像改为自己私有仓库路径

$ cat skydns-rc.yaml | grep image

image: 192.168.100.90:5000/duni/etcd-amd64:3.0.17
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/kube2sky:1.11
image: 192.168.100.90:5000/duni/kube2sky:1.14
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/skydns:2015-03-11-001
image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8c
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/exechealthz:1.0
image: 192.168.100.90:5000/duni/exechealthz-amd64:latest
imagePullPolicy: IfNotPresent

建议镜像到阿里云容器镜像中查找，然后更改镜像tag，再push到自己搭建的私有仓库中，如何搭建自己的私有仓库

2、kube2sky容器中参数：- --kube-master-url=http://192.168.100.27:8080设为你自己的k8s集群master主机ip:port, - --domain=cluster.local 设置集群中service域名（可自行定义一个名字）

3、skydns容器中参数：- -domain=cluster.local，需跟kube2sky中设置的名称一致

二、skydns-svc.yaml 文件

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: default
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP:  10.254.0.100
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

此处可自行修改clusterIP: 10.254.0.100 ip

三、修改k8s集群中节点机的kubelet启动参数

$ vi /etc/kubernetes/kubelet    # centos系统，ubuntu系统在不同路径下

KUBELET_ARGS="--cluster_dns=10.254.0.100 --cluster_domain=cluster.local"

注：--cluster_dns必须跟skydns-svc.yaml文件中的clusterIP值相同，cluster_domain必须跟skydns-rc.yaml文件中skydns和kube2sky容器中的domain参数值相同

重启kubelet

systemctl restart kubelet

四、创建dns pod和service

kubectl create -f skydns-rc.yaml
kubectl create -f skydns-svc.yaml

查看pod和service状态

$ kubectl get pods --all-namespaces | grep kube-dns    
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
default       kube-dns-v9-vldgj                       4/4       Running   0          3h

$ kubectl get pods --all-namespaces | grep kube-dns    
NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
default       kube-dns               10.254.0.100     <none>        53/UDP,53/TCP    3h

五、验证dns

busybox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

创建busybox pod

$ kubectl create -f busybox.yaml
$ kubectl get pods --all-namespaces | grep busybox
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
default       busybox                                 1/1       Running   3          3h

验证dns解析

# 查看所有的service（以下是我master主机所有服务）
$ kbuectl get svc --all-namespaces
NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
default       frontend               10.254.204.132   <nodes>       80:30001/TCP     19h
default       kube-dns               10.254.0.100     <none>        53/UDP,53/TCP    3h
default       kubernetes             10.254.0.1       <none>        443/TCP          30d
default       mysql-service          10.254.246.96    <nodes>       3306:30365/TCP   21h
default       redis-master           10.254.172.30    <none>        6379/TCP         19h
default       redis-service          10.254.253.213   <none>        6379/TCP         19h
kube-system   heapster               10.254.145.32    <none>        80/TCP           18h
kube-system   kubernetes-dashboard   10.254.163.216   <nodes>       80:30009/TCP     20h
kube-system   monitoring-grafana     10.254.199.203   <none>        80/TCP           18h
kube-system   monitoring-influxdb    10.254.27.3      <none>        8086/TCP         18h

# 验证
$ kubectl exec -it busybox nslookup kubernetes
Server:    10.254.0.100
Address 1: 10.254.0.100

nslookup: can't resolve 'kubernetes'

$ kubectl exec -it busybox nslookup kubernetes.default
Server:    10.254.0.100
Address 1: 10.254.0.100

nslookup: can't resolve 'kubernetes.default': Try again

$ kubectl exec -it busybox nslookup www.baidu.com
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      www.baidu.com
Address 1: 14.215.177.38
Address 2: 14.215.177.37

好吧，所有pod运行正常，可dns就是不能根据本地域名解析其ip，发现解析百度，网易等却可以

六、查找dns不能根据域名解析其ip的原因

查看dns pod中各个容器的日志，查看skydns容器时，发现以下错误，请求超时

    $ kubectl logs --namespace=default $(kubectl get pods --namespace=default -l k8s-app=kube-dns -o name) -c skydns
    2017-04-26T07:15:35.141855000Z 2017/04/26 07:15:35 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:18:09.141845000Z 2017/04/26 07:18:09 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:21:53.045513000Z 2017/04/26 07:21:53 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:26:13.142510000Z 2017/04/26 07:26:13 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:28:25.045739000Z 2017/04/26 07:28:25 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"

于是谷歌，找到原因所在，192.168.100.1 这个dns服务地址并不可用，那我们就用谷歌公开的dns

修改集群master主机，新增谷歌dns

$ vi /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.100.1
# 新增下面两行
nameserver 8.8.8.8
nameserver 8.8.4.4

修改skydns-rc.yaml 中 skydns 容器的args参数

args:
# command = "/skydns"
- -machines=http://127.0.0.1:2379
- -addr=0.0.0.0:53
- -ns-rotate=false
- -domain=cluster.local
- -nameservers=8.8.8.8:53,8.8.4.4:53     # 新增这行

重启启动dns 和 busybox pod

$ kubectl delete rc kube-dns --namespace=default
$ kubectl delete src kube-dns --namespace=default
$ kubectl delete pods busybox --namespace=default
$ kubectl create -f skydns-rc.yaml
$ kubectl create -f skydns-svc.yaml
$ kubectl create -f busybox.yaml

七、再次验证dns

$ kubectl exec -it busybox nslookup kubernetes
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      kubernetes
Address 1: 10.254.0.1

$ kubectl exec -it busybox nslookup kubernetes.default
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      kubernetes.default
Address 1: 10.254.0.1

$ kubectl exec -it busybox nslookup heapster
Server:    10.254.0.100
Address 1: 10.254.0.100

nslookup: can't resolve 'heapster'

$ kubectl exec -it busybox nslookup heapster.kube-system
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      heapster.kube-system
Address 1: 10.254.145.32

注：我们的dns pod所在的命名空间（namespace）是在default，当我们需要查询的服务跟dns pod不在同一命名空间时，需通过域名.命名空间 方式进行查找，如上面的heapster.kube-system

友情提示：一定要学会查看pod中的容器日志，因为很多时候，我们一个pod中某个容器只要一个参数书写错误，则该容器就running失败了

kubectl logs –namespace=namespace_name $(kubectl get pods –namespace=namespace_name -l label_name=label_value -o name) -c container_name

如上面查找dns pod中skydns容器日志

kubectl logs –namespace=default $(kubectl get pods –namespace=default -l k8s-app=kube-dns -o name) -c skydns