Openshift v3.11 DNS配置及理解

前言

OpenShift在安裝的時候依賴於NetworkManager服務完成dnsmasq自動配置，請確保該服務保護啓動狀態，並保證在網卡配置文件中不要添加 NM_CONTROLLED=no參數,否則會導致一系列網絡問題。如果一定需要開啓此參數，請做好心裏準備，你要知道整個dns訪問配置流程，需要手動維護下列配置文件：
/etc/resolv.conf
/etc/dnsmasq.d/origin-dns.conf
/etc/dnsmasq.d/origin-upstream-dns.conf
/etc/origin/node/resolv.conf

三種DNS訪問類型

1. Pod訪問外網
2. 集羣內部服務發現
3. 外部服務訪問集羣內部

Pod訪問外網dns查詢流程：pod – dnsmasq – 上游dns(多個地址倒序查詢)
Pod訪問集羣內服務dns查詢流程：pod – dnsmasq – skydns（127.0.0.1:53) – 如果沒有本地緩存 – openshift api
SkyDns：用於解析service域名 ，調用 OpenShift API 服務來獲取主機名、IP地址等信息，然後封裝成標準 DNS 記錄並返回給查詢客戶端。

在node節點查看dns信息，可以看到dnsmasq啓了3個進程，openshift啓動了1個進程爲skydns
#netstat -atunlp|grep 53
tcp 0 0 10.10.6.136:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 172.17.0.1:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 10.131.0.1:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 8542/openshift

其中通過dnsmasq啓動的53端口監聽的網絡與route -n查看到的網段對應
10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
10.128.0.0 0.0.0.0 255.252.0.0 U 0 0 0 tun0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 tun0
在node節點查看skydns信息，通過openshift進程啓動。

#netstat -atunlp|grep 53
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 8542/openshift
node節點上的skydns從緩存或API獲取數據返回客戶端查詢
所有節點都安裝SkyDns，組成一個分佈式集羣，其中master節點額外啓動8053端口

所有DNS配置文件詳細信息

Pod DNS:
#cat /etc/resolv.conf
nameserver 10.10.6.136
search nginx-10-10-163-88.svc.cluster.local svc.cluster.local cluster.local localdomain
options ndots:5
註解：
nameserver是宿主機ip地址，所有dns請求會發送到宿主機53端口

宿主機DNS：
#cat /etc/resolv.conf
#nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
#Generated by NetworkManager
search cluster.local
nameserver 10.10.6.136

安裝啓動NetworkManager會觸發/etc/NetworkManager/dispatcher.d/99-origin-dns.sh腳本，修改 /etc/resolv.conf、設置搜索域，以及將宿主機的默認 IP 作爲 nameserver、創建 /etc/origin/node/resolv.conf、生成/etc/dnsmasq.d/origin-dns.conf,/etc/dnsmasq.d/origin-upstream-dns.conf兩個配置文件、啓動dnsmasq服務，其中origin-upstream-dns.conf 定義了上游DNS服務器，該地址通過DHCP獲取，我這裏沒有生成成功，手動配置並重啓dnsmasq

#cat origin-dns.conf
no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=10000
cache-size=10000
bind-dynamic
min-port=1024
except-interface=lo
#End of config

#cat origin-upstream-dns.conf
server=10.10.255.1
server=10.10.255.2
server=10.10.153.132

#systemctl restart dnsmasq
dnsmasq查詢順序是自下而上,即先查詢10.10.153.132

dnsmasq日誌開啓日誌調試及原理驗證

vi /etc/dnsmasq.conf
#For debugging purposes, log each DNS query as it passes through
#dnsmasq.
log-queries

#systemctl restart dnsmasq
dnsmasq查看日誌
#journalctl -f -u dnsmasq
Jan 14 11:10:51 openshift2 dnsmasq[14678]: setting upstream servers from DBus
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.255.2#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.255.1#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.153.132#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 127.0.0.1#53 for domain cluster.local
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: reply hawkular-metrics.openshift-infra.svc.cluster.local is 172.30.87.204
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: cached hawkular-metrics.openshift-infra.svc.cluster.local is 172.30.87.204
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4

根據日誌得知，dnsmasq把請求都轉發到127.0.0.1:53上的skydns中（封裝在openshift進程）node節點上的skydns從緩存或API獲取數據返回客戶端查詢，可以看到示例中是走的cached

openshift中dns配置信息：
master節點
vim /etc/origin/master/master-config.yaml
dnsConfig:
bindAddress: 0.0.0.0:8053
bindNetwork: tcp4

node節點
vim /etc/origin/node/node-config.yaml
dnsBindAddress: 127.0.0.1:53
dnsDomain: cluster.local
dnsIP: 0.0.0.0
dnsNameservers: null
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf

無論是在宿主機還是在pod內，均可以訪問service，因爲首先都要經過宿主機dns服務器127.0.0.1:53
#ping hawkular-metrics.openshift-infra.svc.cluster.local
PING hawkular-metrics.openshift-infra.svc.cluster.local (172.30.87.204) 56(84) bytes of data.

參考文檔:
http://www.cnblogs.com/sammyliu/p/10056035.html
https://www.cnblogs.com/ericnie/p/10216775.html
https://blog.cloudtechgroup.cn/Blog/2018/07/23/ocp-2018-07-23/