Table of Contents
Multiple Network Interfaces for a k8s pod
Introduction
Install multus binary on a VM
Configure multus
Try it out
Conclusion
Troubleshooting .etc
References
Introduction
In some cases we may need more than one network interface for a single k8s pod. One example is that if we'd like to make an Oracle RAC database into k8s cluster, one pod per instance, then we need at least two network interfaces for its public and private IP. This page describes how we can get this done by leveraging a 3rd party CNI (Container Network Interface) plug-in called "multus".
In short, multus is a CNI plug-in that takes over the k8s network and delegates the work to other existing CNI plug-ins such as flannel, wave-net, macvlan .etc, and thus makes applying multiple network CNIs possible. It is developed by Intel.
Install multus binary on a VM
First let's clone multus-cni repository (change "io" to your UID ):
|
bash-4.2$ cd /home/io/
bash-4.2$ mkdir intel
bash-4.2$ cd intel
bash-4.2$ mkdir multus-cni
bash-4.2$ cd multus-cni
bash-4.2$ git clone https://github.com/intel/multus-cni.git .
|
To install multus binary we need to install go, we can download it from google official site, e.g.
|
bash-4.2$ mkdir go
bash-4.2$ cd go
bash-4.2$ pwd
/home/io/go
bash-4.2$ su -
[root@myhost ~]# export http_proxy=...
[root@myhost ~]# export https_proxy=...
[root@myhost ~]# cd /home/io/go
[root@myhost go]# wget https://dl.google.com/go/go1.11.1.linux-amd64.tar.gz
[root@myhost go]# tar -C /usr/local -xzf go1.11.1.linux-amd64.tar.gz
[root@myhost go]# export PATH=$PATH:/usr/local/go/bin
[root@myhost go]# which go
/usr/local/go/bin/go
|
Now build our multus-cni plug-in:
|
[root@myhost go]# cd /home/io/intel/multus-cni
[root@myhost multus-cni]# ./build
|
Then copy the output binary to /opt/cni/bin/
|
[root@myhost multus-cni]# cd bin
[root@myhost bin]# ls
multus
[root@myhost bin]# cp multus /opt/cni/bin/
|
First create multus cluster role and cluster role binding (change "myhost" to your host name)
|
bash-4.2$ cd /home/io/intel/multus-cni/examples
bash-4.2$ kubectl apply -f clusterrole.yml
bash-4.2$ kubectl create clusterrolebinding myhost --clusterrole=multus-crd-overpowered --user=system:node:myhost
|
Delete any existing CNI config under directory /etc/cni/net.d/, e.g.
|
[root@myhost intel]# cd /etc/cni/net.d
[root@myhost net.d]# mv 10-flannel.conf /home/io/intel/
|
Then apply the multus DaemonSet:
|
bash-4.2$ cd /home/io/intel/multus-cni/examples
bash-4.2$ kubectl apply -f multus-with-flannel.yml
|
After that there should be a new file generated in /etc/cni/net.d:
|
bash-4.2$ ls /etc/cni/net.d/
10-multus-with-flannel.conf
|
Now create the CRD (Custom Resource Definition) and the flannel/macvlan instances
|
bash-4.2$ kubectl apply -f crd.yml
bash-4.2$ kubectl create -f flannel-conf.yml
bash-4.2$ kubectl create -f macvlan-conf.yml
|
At this point we probably have two flannel daemon processes running on our k8s node:
|
bash-4.2$ ps -ef | grep flanneld
root 69153 69135 0 Oct28 ? 00:00:06 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
root 69153 69135 0 Oct28 ? 00:00:06 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
io 103613 10149 0 02:05 pts/0 00:00:00 grep flanneld
|
This is because our k8s installation script had already started a flannel DaemonSet as our default network CNI plugin, and our multus DaemonSet launched a second one. We can delete the original flannel DaemonSet (though I don't see any problem with both of them running):
|
bash-4.2$ kubectl delete daemonset kube-flannel-ds -n kube-system
|
Try it out
Now let's create a simple pod (still under the dir /home/io/intel/multus-cni/examples):
|
bash-4.2$ kubectl create -f sample-pod.yml
|
Wait a while for the image to be pulled and the pod to be created.
As in sample-pod.yml we specifies the macvlan as its extra network interface (flannel is the default one), the pod should have two IP addresses:
|
bash-4.2$ kubectl exec -it samplepod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if629: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
link/ether 0a:58:0a:f4:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.0.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::8cbd:5aff:fee1:442/64 scope link
valid_lft forever preferred_lft forever
4: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 0a:58:c0:a8:01:c8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.1.200/24 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::858:c0ff:fea8:1c8/64 scope link
valid_lft forever preferred_lft forever
|
Create a file sample-pod-flannel.yml with following content (the only difference between it and sample-pod.yml is that we commented out the networks annotation) and apply it:
|
apiVersion: v1
kind: Pod
metadata:
name: samplepod2
annotations:
# k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod2
command: ["/bin/bash", "-c", "sleep 2000000000000"]
image: dougbtv/centos-network
ports:
- containerPort: 80
|
| |
Wait a while for the pod to create, and we will see that it has only the default flannel network address:
|
bash-4.2$ kubectl exec -it samplepod2 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
link/ether 0a:58:0a:f4:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.0.3/24 scope global eth0
valid_lft forever preferred_lft forever
|
Form samplepod2 we can ping samplepod by 10.244.0.2, but not 192.168.1.200, because samplepod2 is not in the macvlan network.
If we update samplepod2 adding back the annotation "k8s.v1.cni.cncf.io/networks: macvlan-conf", it will be able to ping samplepod by both of the IP addresses.
Conclusion
With multus CNI plugin (to substitute the default flannel one) and its CRD objects, we can customize our network interfaces for each pod.
Troubleshooting .etc
- Use command "docker inspect dougbtv/centos-network" to inspect the image of samplepod container.
- If see any problem, try to restart kubelet:
[root@myhost intel]# systemctl restart kubelet
- Use the following command to see pod schedule failures
[root@myhost intel]# journalctl -u kubelet
References
https://github.com/intel/multus-cni
