Table of Contents

Introduction

In some cases the k8s default DNS service does not match our needs, for example an Oracle RAC database has to resolve the public/private/vip/scan IPs for its nodes. This document describes how we can customize the DNS service including integrating our own DNS server.

This documents assumes you know the basic concepts of k8s and DNS.

This documents assumes you have read this page: multiple network interfaces for a pod, because we will touch this in later section.

k8s 1.9+ supports better DNS customization, but our environment is still 1.8 compatible, and uses kube-dns instead of CoreDNS.

Basics of k8s DNS

The default DNS record of a k8s pod is: a-b-c-d.sub-domain.my-namespace.pod.cluster.local

a-b-c-d is the IP string of the pod. For example, a pod my-pod with IP address 10.244.0.2 in default namespace, will have an A record of "10-244-0-2.default.pod.cluster.local". If the pod specifies a sub domain, such as ora-subdomain, it will be "10-244-0-2.ora-subdomain.default.pod.cluster.local". There is no pod name or hostname record in the DNS server. However, if we create a headless service (services without a clusterIP) with the name exactly same as the sub-domain, we will have a record "my-pod.ora-subdomin.default.svc.cluster.local" (note the suffix "pod.cluster.local" becomes "svc.cluster.local"). This is also where the StatefulSet's stable network identifier comes from, in that case, in addition to each pod that belongs to the StatefulSet, the service itself (i.e. "sub-domain.my-namespace.svc.cluster.local") will resolve to the IPs (a set) of the pods.

A service's DNS record is always my-svc.my-namespace.svc.cluster.local.

kube-dns customization

kube-dns can be extended to support additional DNS name servers. In short, consider the following config map:

apiVersion: v1

kind: ConfigMap

metadata:

name: kube-dns

namespace: kube-system

data:

stubDomains: |

{"example.com": ["1.2.3.4"]}

upstreamNameservers: |

["8.8.8.8", "8.8.4.4"]

The custom configurations are stubDomains and upstreamNameservers.

First, let's see what is the resolve process when these configurations does NOT exist:

The name resolves according to the default rule of k8s DNS, as stated in "Basics of k8s DNS" section.
If the name is not found (e.g. names with example.com domain ), it is forwarded to upstream name servers that inherited from the node.

This means the k8s node's /etc/hosts and /etc/resolv.conf functions inside the pod containers.

Now what if stubDomains and/or upstramNameservers are specified? The process is:

For names with the cluster suffix, i.e. ".cluster.local", the request is sent to kube-dns.
For names with the stub domain suffix, i.e. ".example.com", the request is sent to the configured custom DNS server, listening at 1.2.3.4.
For names without a matching suffix, for example "github.com", the request is forwarded to the upstream DNS, in this case 8.8.8.8 and 8.8.4.4 (These are public DNS servers of Google company).

Important: If upstreamNameservers is specified, we'll lost the node's resolving. A natural solution would be copying the node's resolve to the custom name server.

Setup our own name server

So what will be our custom name server? dnsmasq is an easy-to-configure, widely used one (kube-dns itself is using it). In this section, we will setup our own dnsmasq server from scratch.

Prepare the Dockerfile for dnsmasq image

First write the Dockerfile

Dockerfile

FROM oraclelinux:7

# Install dnsmasq

RUN yum install -y dnsmasq

# Install netcat

RUN yum install -y nc

# Pre-configure dnsmasq

RUN echo 'listen-address=__LOCAL_IP__' >> /etc/dnsmasq.conf

RUN echo 'resolv-file=/etc/resolv.dnsmasq.conf' >> /etc/dnsmasq.conf

RUN echo 'conf-dir=/etc/dnsmasq.d' >> /etc/dnsmasq.conf

RUN touch /etc/resolv.dnsmasq.conf

# Copied from node host /etc/resolv.conf, TODO: automate this

RUN echo 'nameserver my.internal.name.server1.ip' >> /etc/resolv.dnsmasq.conf

RUN echo 'nameserver my.internal.name.server2.ip' >> /etc/resolv.dnsmasq.conf

RUN echo 'nameserver my.internal.name.server3.ip' >> /etc/resolv.dnsmasq.conf

RUN echo 'search cn.my.com my.com mycorp.com' >> /etc/resolv.dnsmasq.conf

# This directory will usually be provided with the -v option.

# RUN echo 'address=/example.com/xx.xx.xx.xx' >> /etc/dnsmasq.d/0hosts

# On the other hand the above directory isn't reloaded with a SIGHUP. Instead

# we can use an --addn-hosts file, see run.sh.

RUN touch /etc/addn-hosts

ADD run.sh /root/run.sh

EXPOSE 22

EXPOSE 53

EXPOSE 12345

CMD /root/run.sh

Note that we explicitly specify "resolv-file=resolv.dnsmasq.conf" to replace the default /etc/resolv.conf and thus bypass the k8s pod default resolving. We copy the node host name servers and search domains, so our custom name server can still resolve the names out of the k8s cluster. Actually what we copy here is the kube-dns pod's /etc/resolv.conf, which in turn copies (partly) the file of the node:

bash-4.2$ kubectl get po -n kube-system | grep "kube-dns"

NAME READY STATUS RESTARTS AGE

kube-dns-545bc4bfd4-zjsz8 3/3 Running 4 4d

bash-4.2$ kubectl exec kube-dns-545bc4bfd4-zjsz8 -c kubedns -n kube-system cat /etc/resolv.conf

nameserver my.internal.name.server1.ip

nameserver my.internal.name.server2.ip

nameserver my.internal.name.server3.ip

search cn.my.com my.com mycorp.com

bash-4.2$ cat /etc/resolv.conf

options timout:1

options attempts:2

; generated by /usr/sbin/dhclient-script

search cn.my.com mycorp.com my.com

nameserver my.internal.name.server1.ip

nameserver my.internal.name.server2.ip

nameserver my.internal.name.server3.ip

The /etc/resolv.conf specified the "upstream" name servers of kube-dns, and explains how it inherited from the node's name resolving.

The run.sh specified in the CMD field of Dockerfile is:

#!/bin/bash

sed -i s/__LOCAL_IP__/$POD_IP/ /etc/dnsmasq.conf

dnsmasq --addn-hosts=/etc/addn-hosts &

echo "Config is /etc/dnsmasq.conf"

echo "--addn-hosts=/etc/addn-hosts"

# Start netcat, listen on port 12345 and reload the host line automatically on the fly

while [ 1 ];

do

#sleep 3600

m=$(nc -l 0.0.0.0 12345);

echo $m;

echo $m >> /etc/addn-hosts;

kill -HUP $(pgrep dnsmasq)

echo "-- Reloaded --"

done

put it to the same path of Dockerfile and build the image:

docker build --rm --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --build-arg no_proxy=$no_proxy -t my/dnsmasq .

Prepare our dnsmasq pod, i.e. custom name server

Yaml work:

dnsmasq.yaml

apiVersion: v1

kind: Pod

metadata:

name: mydns

labels:

app: mydns

spec:

containers:

- image: my/dnsmasq

name: mydns

command: ["/bin/sh", "-c", "/root/run.sh"]

imagePullPolicy: Never

securityContext:

privileged: true

env:

- name: POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

---

apiVersion: v1

kind: Service

metadata:

name: mydns-service

labels:

app: mydns-service

spec:

ports:

- port: 12345

name: inbound-port

- port: 53

name: dns-service-port

selector:

app: mydns

Then create the pod and service:

kubectl apply -f dnsmasq.yaml

Integrate our name server with kube-dns: create the ConfigMap

After the pod mydns starts, use "kubectl logs.." to see if it starts successfully, and then get its IP:

bash-4.2$ kubectl logs mydns

Config is /etc/dnsmasq.conf

--addn-hosts=/etc/addn-hosts

bash-4.2$ kubectl get po -o wide | grep mydns | awk '{print $6}'

10.244.0.11

Now write our ConfigMap yaml file using this IP:

cm.yaml

apiVersion: v1

kind: ConfigMap

metadata:

name: kube-dns

namespace: kube-system

data:

stubDomains: |

{"example.com": ["10.244.0.11"]}

upstreamNameservers: |

["10.244.0.11"]

Apply it

kubectl apply -f cm.yaml

Add entries to our DNS and test it out

As we stated previously, we assume you have read Multiple Network Interfaces for a k8s pod, now we are about to test our DNS server with some simple pod that enables multiple network interfaces. The testing pod is an ole7 container doing nothing but sleeping:

ole7.yaml

apiVersion: v1

kind: Pod

metadata:

name: ole7

annotations:

k8s.v1.cni.cncf.io/networks: macvlan-conf

spec:

volumes:

- name: tempdir

emptyDir: {}

initContainers:

- name: get-priv-ip

image: oraclelinux:7

command: ["/bin/bash", "-c", "ip a | grep 192.168 | awk '{ print substr($2, 1, index($2, \"/\") - 1) }' 2>&1 | tee /temp/PRIV_IP"]

volumeMounts:

- name: "tempdir"

mountPath: "/temp"

containers:

- name: ole7

command: ["/bin/bash", "-c", "sleep 2000000000000"]

image: oraclelinux:7

env:

- name: POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

volumeMounts:

- name: "tempdir"

mountPath: "/temp"

Create it by:

kubectl apply -f ole7.yaml

The pod has two IP addresses, one is the flannel default (displays in "kubectl get pod ...") and one is in file /temp/PRIV_IP in the container.

Once the pod is initiated, we can test it out. In the following example, I got both public and private IPs for pod "ole7", added them to our custom name server (by redirecting the echo output to /dev/tcp/mydns-service/12345) dynamically, giving them names. Next, I pinged all the names with or without the domain suffix, I also ping some hosts that out of the k8s cluster, e.g. vm09xxl and home.cn.my.com, and all the commands are successful.

Test commands

bash-4.2$ kubectl get po -o wide | grep ole7 | awk '{print $6}'

10.244.0.26

bash-4.2$ kubectl exec ole7 -- cat /temp/PRIV_IP

192.168.1.211

bash-4.2$ kubectl exec -ti ole7 -- bash

[root@ole7 /]# echo "10.244.0.26 ole.example.com ole" > /dev/tcp/mydns-service/12345

[root@ole7 /]# echo "192.168.1.211 ole-priv.example.com ole-priv" > /dev/tcp/mydns-service/12345

[root@ole7 /]# ping -c 3 ole

PING ole (10.244.0.26) 56(84) bytes of data.

64 bytes from ole7 (10.244.0.26): icmp_seq=1 ttl=64 time=0.057 ms

64 bytes from ole7 (10.244.0.26): icmp_seq=2 ttl=64 time=0.057 ms

64 bytes from ole7 (10.244.0.26): icmp_seq=3 ttl=64 time=0.038 ms

--- ole ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1999ms

rtt min/avg/max/mdev = 0.038/0.050/0.057/0.012 ms

[root@ole7 /]# ping -c 3 ole.example.com

PING ole.example.com (10.244.0.26) 56(84) bytes of data.

64 bytes from ole7 (10.244.0.26): icmp_seq=1 ttl=64 time=0.022 ms

64 bytes from ole7 (10.244.0.26): icmp_seq=2 ttl=64 time=0.032 ms

64 bytes from ole7 (10.244.0.26): icmp_seq=3 ttl=64 time=0.038 ms

--- ole.example.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2000ms

rtt min/avg/max/mdev = 0.022/0.030/0.038/0.009 ms

[root@ole7 /]# ping -c 3 ole-priv

PING ole-priv (192.168.1.211) 56(84) bytes of data.

64 bytes from ole7 (192.168.1.211): icmp_seq=1 ttl=64 time=0.023 ms

64 bytes from ole7 (192.168.1.211): icmp_seq=2 ttl=64 time=0.045 ms

64 bytes from ole7 (192.168.1.211): icmp_seq=3 ttl=64 time=0.043 ms

--- ole-priv ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2000ms

rtt min/avg/max/mdev = 0.023/0.037/0.045/0.009 ms

[root@ole7 /]# ping -c 3 ole-priv.example.com

PING ole-priv.example.com (192.168.1.211) 56(84) bytes of data.

64 bytes from ole7 (192.168.1.211): icmp_seq=1 ttl=64 time=0.026 ms

64 bytes from ole7 (192.168.1.211): icmp_seq=2 ttl=64 time=0.054 ms

64 bytes from ole7 (192.168.1.211): icmp_seq=3 ttl=64 time=0.038 ms

--- ole-priv.example.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2000ms

rtt min/avg/max/mdev = 0.026/0.039/0.054/0.012 ms

[root@ole7 /]# ping -c 3 vm09xxl

PING vm09xxl.cn.my.com (10.xxx.xxx.xxx) 56(84) bytes of data.

64 bytes from vm09xxl.cn.my.com (10.xxx.xxx.xxx): icmp_seq=1 ttl=57 time=0.703 ms

64 bytes from vm09xxl.cn.my.com (10.xxx.xxx.xxx): icmp_seq=2 ttl=57 time=0.716 ms

64 bytes from vm09xxl.cn.my.com (10.xxx.xxx.xxx): icmp_seq=3 ttl=57 time=0.591 ms

--- vm09xxl.cn.my.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2000ms

rtt min/avg/max/mdev = 0.591/0.670/0.716/0.056 ms

[root@ole7 /]# ping -c 3 home.cn.my.com

PING vmibfj.cn.my.com (10.xxx.x.xxx) 56(84) bytes of data.

64 bytes from vmibfj.cn.my.com (10.xxx.x.xxx): icmp_seq=1 ttl=57 time=0.602 ms

64 bytes from vmibfj.cn.my.com (10.xxx.x.xxx): icmp_seq=2 ttl=57 time=0.657 ms

64 bytes from vmibfj.cn.my.com (10.xxx.x.xxx): icmp_seq=3 ttl=57 time=0.684 ms

--- vmibfj.cn.my.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2000ms

rtt min/avg/max/mdev = 0.602/0.647/0.684/0.045 ms

We add two names for a single pod IP, e.g "ole.exmple.com" and "ole" to addn-hosts in our name server. When a name resolving request is sent to kube-dns, if the name has domain suffix i.e. example.com, the request is forwarded to the custom name server specified in the "stubDomains" attribute of the ConfigMap. If the name does not contain suffix, the request is firstly handled by kube-dns by adding the "...cluster.local" suffix, if the name could not be found in k8s cluster, kube-dns will forward it to "upstreamNameservers", in our case, this is the same custom name server, and the name is found in addn-hosts. On the other hand, a name such as "vm09xxl" follows the same resolving path, but cannot be found in addn-hosts, so the name server consults its /etc/resolv.dnsmasq.conf and forwards the request to its upstream name servers specified there.

Issues

Pod CIDR

Our current pod_network_cidr="10.244.0.0/16", the IP(s) allocated to pods conflict to those of our VMs'. For example, 10.244.0.25 is somevm1.cn.my.com, 10.244.0.26 is somevm2.cn.my.com. This may case confusion in our name resolving. We may need to change pod_network_cidr to another value.