flannel實戰

        docker swarm mode的出現是個里程碑，官方原生的編排調度看起來都成雛形了，但是swarm mode和容器外部系統的對接、網絡性能始終不盡人意，swarm mode下各種開源周邊不能使用，感覺swarm mode自成一個體系，網絡方面上篇調研了calico，本篇調研一下flannel，總體感覺大家都是在向K8S靠攏，docker原生這也是涼涼嘛.....

軟件信息

軟件	版本
OS	Ubuntu 16.04.3 LTS
Docker	18.03.0-ce
Etcd	3.3.9
Flannel	0.10.0

主機信息

ubuntu16.04-1	172.31.68.241	workload-A	docker、etcd、flannel
ubuntu16.04-2	172.31.68.242	workload-B	docker、flannel
ubuntu16.04-3	172.31.68.243	workload-C	docker、flannel

工作目錄

    /opt/programs：各種軟件的下載均在該目錄下

docker安裝

    下載

wget 'https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/amd64/docker-ce_18.03.0~ce-0~ubuntu_amd64.deb'

    安裝

dpkg -i docker-ce_18.03.0~ce-0~ubuntu_amd64.deb

    配置

        /etc/docker/daemon.json    

{
  "registry-mirrors": ["http://aa2fd190.m.daocloud.io"],
  "insecure-registries":["http://172.31.68.241"],
  "dns": ["202.96.209.5","202.96.209.133"],
  "experimental": true
}

    啓動

systemctl daemon-reload
systemctl restart docker.service

   啓動腳本

        /lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock 
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

    注意：以上步驟在三臺機器都需要執行

etcd安裝

    下載

wget 'https://github-production-release-asset-2e65be.s3.amazonaws.com/11225014/7c787e4c-8f2b-11e8-8a29-4db755239b18?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180813%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180813T090430Z&X-Amz-Expires=300&X-Amz-Signature=dc354d85304b135c99c4d0b606fa07bd75eaaa2fd46b19d43554d9112a4d83d8&X-Amz-SignedHeaders=host&actor_id=12913767&response-content-disposition=attachment%3B%20filename%3Detcd-v3.3.9-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream'

    安裝

tar zxvf etcd-v3.3.9-linux-amd64.tar.gz
mv etcd-v3.3.9-linux-amd64 etcd_3.3.9

    配置啓動腳本

        /etc/systemd/system/etcd.service

[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0 

ExecStart=/opt/programs/etcd_3.3.9/etcd --name ubuntu16.04-1 \
   --data-dir /var/lib/etcd \
   --listen-client-urls http://172.31.68.241:2379 \
   --listen-peer-urls http://172.31.68.241:2380 \
   --advertise-client-urls http://172.31.68.241:2379 \
   --initial-advertise-peer-urls http://172.31.68.241:2380

[Install]
WantedBy=multi-user.target

    啓動

systemctl daemon-reload
systemctl start etcd.service

    驗證

etcdctl --endpoints http://172.31.68.241:2379  cluster-health

    注意：以上操作只需要在ubuntu16.04-1上執行

flannel安裝

    下載

wget 'https://github-production-release-asset-2e65be.s3.amazonaws.com/21704134/596e76e2-002c-11e8-9359-36689058e7af?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180815%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180815T052949Z&X-Amz-Expires=300&X-Amz-Signature=5b4fda3ddd09110a2a9d17cd65d1af2adef7a608888fc95a2692623768b2daad&X-Amz-SignedHeaders=host&actor_id=12913767&response-content-disposition=attachment%3B%20filename%3Dflannel-v0.10.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream'

    安裝

mkdir /opt/programs/flannel_0.10.0
tar zxvf flannel-v0.10.0-linux-amd64.tar.gz -C /opt/programs/flannel_0.10.0

    etcd配置網段

etcdctl --endpoints http://172.31.68.241:2379 set /flannel/network/config '{"Network":"192.168.0.0/16","Backend":{"Type":"vxlan"}}'

    啓動flannel

flanneld -etcd-endpoints=http://172.31.68.241:2379 -ip-masq=true -etcd-prefix=/flannel/network

    配置啓動腳本

        /lib/systemd/system/flannel.service

[Unit]
Description=coreos flannel
Documentation=https://my.oschina.net/guol/blog/1928408
After=docker.service


[Service]
TimeoutSec=0
ExecStart=/opt/programs/flannel_0.10.0/flanneld -etcd-endpoints=http://172.31.68.241:2379 -ip-masq=true -etcd-prefix=/flannel/network
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
 

[Install]
WantedBy=multi-user.target

    生產docker配置

mk-docker-opts.sh

        ps：生成的文件在/run/docker_opts.env

    調整docker參數

        /lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock $DOCKER_OPTS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

    重啓docker

systemctl daemon-reload
systemctl start docker.service

    查看網卡

    注意：以上部署需要在三臺機器上都執行，上例是以ubuntu16.04-1爲例進行演示，"etcd配置網段"只需要執行一次即可

Etcd數據驗證

etcdctl --endpoints http://172.31.68.241:2379 ls /flannel/network/subnets/

啓動測試容器

    分別在三臺機器上啓動一個容器實例

docker run  --name workload-A -ti 172.31.68.241/library/xenial3
docker run  --name workload-B -ti 172.31.68.241/library/xenial3
docker run  --name workload-C -ti 172.31.68.241/library/xenial3

聯通測試

    容器間通訊

        workload-A --> workload-C

    容器宿主機通訊

        workload-C  --> ubuntu16.04-1

    主機容器通訊

        ubuntu16.04-3 --> workload-B