k8s集羣部署DNS服務發現

環境：

• os: centos
• kubernetes: 1.5.2
• docker: 1.12.5

一、skydns-rc.yaml 文件

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v9
  namespace: default
  labels:
    k8s-app: kube-dns
    version: v9
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v9
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v9
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: 192.168.100.90:5000/duni/etcd-amd64:3.0.17
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        command:
        - /usr/local/bin/etcd
       # - --privileged=true
        - -data-dir
        - /home/data/etcd
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - mountPath: /home/data/etcd
          name: etcd-storage
      - name: kube2sky
        #image: gcr.io/google_containers/kube2sky:1.11
        image: 192.168.100.90:5000/duni/kube2sky:1.14
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        args:
        # command = "/kube2sky"
       # - -etcd-server=http://127.0.0.1:4001
        #- -kube_master_url=http://172.27.8.210:8080
        - --kube-master-url=http://192.168.100.27:8080
        - --domain=cluster.local
      - name: skydns
        #image: gcr.io/google_containers/skydns:2015-03-11-001
        image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8c
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://127.0.0.1:2379
        - -addr=0.0.0.0:53
        - -ns-rotate=false
        - -domain=cluster.local
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 1
          timeoutSeconds: 5
      - name: healthz
        #image: gcr.io/google_containers/exechealthz:1.0
        image: 192.168.100.90:5000/duni/exechealthz-amd64:latest
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default  # Don't use cluster DNS.

skydns-rc.yaml文件中需要根據自己的實際情況進行相關修改

1、把鏡像改爲自己私有倉庫路徑

$ cat skydns-rc.yaml | grep image

image: 192.168.100.90:5000/duni/etcd-amd64:3.0.17
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/kube2sky:1.11
image: 192.168.100.90:5000/duni/kube2sky:1.14
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/skydns:2015-03-11-001
image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8c
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/exechealthz:1.0
image: 192.168.100.90:5000/duni/exechealthz-amd64:latest
imagePullPolicy: IfNotPresent

建議鏡像到阿里雲容器鏡像中查找，然後更改鏡像tag，再push到自己搭建的私有倉庫中，如何搭建自己的私有倉庫

2、kube2sky容器中參數：- --kube-master-url=http://192.168.100.27:8080設爲你自己的k8s集羣master主機ip:port, - --domain=cluster.local 設置集羣中service域名（可自行定義一個名字）

3、skydns容器中參數：- -domain=cluster.local，需跟kube2sky中設置的名稱一致

二、skydns-svc.yaml 文件

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: default
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP:  10.254.0.100
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

此處可自行修改clusterIP: 10.254.0.100 ip

三、修改k8s集羣中節點機的kubelet啓動參數

$ vi /etc/kubernetes/kubelet    # centos系統，ubuntu系統在不同路徑下

KUBELET_ARGS="--cluster_dns=10.254.0.100 --cluster_domain=cluster.local"

注：--cluster_dns必須跟skydns-svc.yaml文件中的clusterIP值相同，cluster_domain必須跟skydns-rc.yaml文件中skydns和kube2sky容器中的domain參數值相同

重啓kubelet

systemctl restart kubelet

四、創建dns pod和service

kubectl create -f skydns-rc.yaml
kubectl create -f skydns-svc.yaml

查看pod和service狀態

$ kubectl get pods --all-namespaces | grep kube-dns    
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
default       kube-dns-v9-vldgj                       4/4       Running   0          3h

$ kubectl get pods --all-namespaces | grep kube-dns    
NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
default       kube-dns               10.254.0.100     <none>        53/UDP,53/TCP    3h

五、驗證dns

busybox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

創建busybox pod

$ kubectl create -f busybox.yaml
$ kubectl get pods --all-namespaces | grep busybox
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
default       busybox                                 1/1       Running   3          3h

驗證dns解析

# 查看所有的service（以下是我master主機所有服務）
$ kbuectl get svc --all-namespaces
NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
default       frontend               10.254.204.132   <nodes>       80:30001/TCP     19h
default       kube-dns               10.254.0.100     <none>        53/UDP,53/TCP    3h
default       kubernetes             10.254.0.1       <none>        443/TCP          30d
default       mysql-service          10.254.246.96    <nodes>       3306:30365/TCP   21h
default       redis-master           10.254.172.30    <none>        6379/TCP         19h
default       redis-service          10.254.253.213   <none>        6379/TCP         19h
kube-system   heapster               10.254.145.32    <none>        80/TCP           18h
kube-system   kubernetes-dashboard   10.254.163.216   <nodes>       80:30009/TCP     20h
kube-system   monitoring-grafana     10.254.199.203   <none>        80/TCP           18h
kube-system   monitoring-influxdb    10.254.27.3      <none>        8086/TCP         18h

# 驗證
$ kubectl exec -it busybox nslookup kubernetes
Server:    10.254.0.100
Address 1: 10.254.0.100

nslookup: can't resolve 'kubernetes'

$ kubectl exec -it busybox nslookup kubernetes.default
Server:    10.254.0.100
Address 1: 10.254.0.100

nslookup: can't resolve 'kubernetes.default': Try again

$ kubectl exec -it busybox nslookup www.baidu.com
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      www.baidu.com
Address 1: 14.215.177.38
Address 2: 14.215.177.37

好吧，所有pod運行正常，可dns就是不能根據本地域名解析其ip，發現解析百度，網易等卻可以

六、查找dns不能根據域名解析其ip的原因

查看dns pod中各個容器的日誌，查看skydns容器時，發現以下錯誤，請求超時

    $ kubectl logs --namespace=default $(kubectl get pods --namespace=default -l k8s-app=kube-dns -o name) -c skydns
    2017-04-26T07:15:35.141855000Z 2017/04/26 07:15:35 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:18:09.141845000Z 2017/04/26 07:18:09 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:21:53.045513000Z 2017/04/26 07:21:53 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:26:13.142510000Z 2017/04/26 07:26:13 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
    2017-04-26T07:28:25.045739000Z 2017/04/26 07:28:25 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"

於是谷歌，找到原因所在，192.168.100.1 這個dns服務地址並不可用，那我們就用谷歌公開的dns

修改集羣master主機，新增谷歌dns

$ vi /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.100.1
# 新增下面兩行
nameserver 8.8.8.8
nameserver 8.8.4.4

修改skydns-rc.yaml 中 skydns 容器的args參數

args:
# command = "/skydns"
- -machines=http://127.0.0.1:2379
- -addr=0.0.0.0:53
- -ns-rotate=false
- -domain=cluster.local
- -nameservers=8.8.8.8:53,8.8.4.4:53     # 新增這行

重啓啓動dns 和 busybox pod

$ kubectl delete rc kube-dns --namespace=default
$ kubectl delete src kube-dns --namespace=default
$ kubectl delete pods busybox --namespace=default
$ kubectl create -f skydns-rc.yaml
$ kubectl create -f skydns-svc.yaml
$ kubectl create -f busybox.yaml

七、再次驗證dns

$ kubectl exec -it busybox nslookup kubernetes
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      kubernetes
Address 1: 10.254.0.1

$ kubectl exec -it busybox nslookup kubernetes.default
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      kubernetes.default
Address 1: 10.254.0.1

$ kubectl exec -it busybox nslookup heapster
Server:    10.254.0.100
Address 1: 10.254.0.100

nslookup: can't resolve 'heapster'

$ kubectl exec -it busybox nslookup heapster.kube-system
Server:    10.254.0.100
Address 1: 10.254.0.100

Name:      heapster.kube-system
Address 1: 10.254.145.32

注：我們的dns pod所在的命名空間（namespace）是在default，當我們需要查詢的服務跟dns pod不在同一命名空間時，需通過域名.命名空間 方式進行查找，如上面的heapster.kube-system

友情提示：一定要學會查看pod中的容器日誌，因爲很多時候，我們一個pod中某個容器只要一個參數書寫錯誤，則該容器就running失敗了

kubectl logs –namespace=namespace_name $(kubectl get pods –namespace=namespace_name -l label_name=label_value -o name) -c container_name

如上面查找dns pod中skydns容器日誌

kubectl logs –namespace=default $(kubectl get pods –namespace=default -l k8s-app=kube-dns -o name) -c skydns