Open-vSwitch構建Docker跨主機網絡

環境說明：

主機名	操作系統	宿主機IP	Docker IP
ovs01	ubuntu 18.04	192.168.168.10	172.17.0.1
ovs02	ubuntu 18.04	192.168.168.11	172.17.1.2

 

 

 

---

安裝配置OVS網絡：

1、安裝docker－ce (安裝過程此處略)

2、設置docker0網段(ovs02同樣操作，bip不同)

$ sudo vi /etc/docker/daemon.json
{
    "bip":"172.17.0.1/24"
}

$ sudo systemctl restart docker

3、安裝openvswitch-switch和bridge-utils

$ sudo apt-get -y install openvswitch-switch bridge-utils

4、查看ovs運行狀態

$ sudo ps -ea | grep ovs
  1526 ?        00:00:00 ovsdb-server
  1593 ?        00:00:00 ovs-vswitchd

5、查看ovs版本信息和ovs支持的OpenFlow協議的版本

$ sudo ovs-appctl --version
ovs-appctl (Open vSwitch) 2.9.5

$ sudo ovs-ofctl --version
ovs-ofctl (Open vSwitch) 2.9.5
OpenFlow versions 0x1:0x5

6、創建br0網橋並激活

$ sudo ovs-vsctl add-br br0
$ sudo ip link set dev br0 up

7、創建gre隧道(remote_ip爲peer宿主機ip)

$ sudo ovs-vsctl add-port br0 gre0 -- set Interface gre0 type=gre options:remote_ip=192.168.168.11     //ovs01配置

$ sudo ovs-vsctl add-port br0 gre0 -- set Interface gre0 type=gre options:remote_ip=192.168.168.10     //ovs02配置

注：如有多臺docker主機需要構建網絡創建多個gre隧道

8、將br0作爲接口加入docker0網橋

$ sudo brctl addif docker0 br0
$ sudo brctl stp docker0 on

9、查看網橋配置

$ sudo ovs-vsctl show
cedc63c1-97d6-4e5e-bdf0-3efc0a5b7aa4
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "vxlan0"
            Interface "gre0"
                type: gre
                options: {remote_ip="192.168.168.11"}
    ovs_version: "2.9.5"

$ brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.02425f251c20       no              br0

10、添加靜態路由(網段地址爲peer Docker網段)

$ sudo ip route add 172.17.1.0/24 dev docker0  //ovs01添加peer docker net
$ sudo ip route add 172.17.0.0/24 dev docker0  //ovs02添加peer docker net

11、測試連通性

$ docker run -it busybox:1.28.3 /bin/sh       //ovs01測試
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/24 brd 172.17.0.255 scope global eth0
       valid_lft forever preferred_lft forever
/ # ping 172.17.1.2
PING 172.17.1.2 (172.17.1.2): 56 data bytes
64 bytes from 172.17.1.2: seq=0 ttl=63 time=3.302 ms
64 bytes from 172.17.1.2: seq=1 ttl=63 time=0.824 ms

$ docker run -it busybox:1.28.3 /bin/sh      //ovs02測試
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:11:01:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.2/24 brd 172.17.1.255 scope global eth0
       valid_lft forever preferred_lft forever
/ # ping 172.17.0.2
PING 172.17.0.2 (172.17.0.2): 56 data bytes
64 bytes from 172.17.0.2: seq=0 ttl=63 time=1.903 ms
64 bytes from 172.17.0.2: seq=1 ttl=63 time=0.765 ms

12、網橋配置和添加路由配置重啓宿主機後會失效，寫成shell腳本，重啓後執行

$ sudo cat > add_bridge.sh <<EOF   //ovs01配置
#!/bin/bash
sudo ip link set dev br0 up
sudo brctl addif docker0 br0
sudo ip route add 172.17.1.0/24 dev docker0
EOF
$ sudo chmod +x add_bridge.sh

$ sudo cat > add_bridge.sh <<EOF   //ovs02配置
#!/bin/bash
sudo ip link set dev br0 up
sudo brctl addif docker0 br0
sudo ip route add 172.17.0.0/24 dev docker0
EOF
$ sudo chmod +x add_bridge.sh

CentOS7 OVS安裝並生成RPM安裝包

1、安裝依賴包

yum -y install make gcc openssl-devel autoconf automake rpm-build redhat-rpm-config python-devel kernel-devel kernel-debug-devel libtool bridge-utils

2、下載OVS二進制安裝包

# mkdir -p ~/rpmbuild/SOURCES
# wget https://www.openvswitch.org/releases/openvswitch-2.5.9.tar.gz -P ~/rpmbuild/SOURCES

3、生成OVS RPM安裝包

# cd ~/rpmbuild/SOURCES
# tar -xvf openvswitch-2.5.9.tar.gz
# sed 's/openvswitch-kmod, //g' openvswitch-2.5.9/rhel/openvswitch.spec > openvswitch-2.5.9/rhel/openvswitch_no_kmod.spec

# rpmbuild -bb --nocheck openvswitch-2.12.0/rhel/openvswitch_no_kmod.spec

4、安裝OVS

# yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-2.5.9-1.x86_64.rpm

下載備份OVS RPM包，可在其它CentOS系統直接使用

5、啓動OVS服務

# service openvswitch start                                             
Starting openvswitch (via systemctl):                      [  OK  ]

# service openvswitch status
ovsdb-server is running with pid 7004
ovs-vswitchd is running with pid 7024

# chkconfig --add openvswitch
# chkconfig openvswitch on

# tail -50f /var/log/messages
Jan 19 11:07:39 ovs yum[6922]: Installed: openvswitch-2.5.9-1.x86_64
Jan 19 11:07:52 ovs systemd: Starting LSB: Open vSwitch switch...
Jan 19 11:07:52 ovs openvswitch: /etc/openvswitch/conf.db does not exist ... (warning).
Jan 19 11:07:52 ovs openvswitch: Creating empty database /etc/openvswitch/conf.db [  OK  ]
Jan 19 11:07:52 ovs openvswitch: Starting ovsdb-server [  OK  ]
Jan 19 11:07:52 ovs ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1
Jan 19 11:07:52 ovs ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.5.9 "external-ids:system-id=\"5aed6a14-bad2-438b-b012-c3dcbcb817fc\"" "system-type=\"unknown\"" "system-version=\"unknown\""
Jan 19 11:07:52 ovs openvswitch: Configuring Open vSwitch system IDs [  OK  ]
Jan 19 11:07:52 ovs kernel: nf_conntrack version 0.5.0 (7928 buckets, 31712 max)
Jan 19 11:07:52 ovs kernel: openvswitch: Open vSwitch switching datapath
Jan 19 11:07:52 ovs openvswitch: Inserting openvswitch module [  OK  ]
Jan 19 11:07:52 ovs openvswitch: Starting ovs-vswitchd [  OK  ]
Jan 19 11:07:52 ovs openvswitch: Enabling remote OVSDB managers [  OK  ]
Jan 19 11:07:52 ovs systemd: Started LSB: Open vSwitch switch.