創建加密的ETCD docker集羣實戰記錄

ETCD 集羣在現代IT界面裏被廣泛用到，Mesos,Kubernetes, SpringCloud,
Ceph 這些知名的開源軟件都用到了etcd集羣。

之前的部署基本都是內部使用，沒有考慮到如何使用安全加密的ETCD,最近深入研究了k8s和Ceph, 發現etcd 基於TLS安全通訊的集羣模式是很重要的事情。下面是基於實踐的記錄，如何成功部署基於docker 的ETCD 集羣

前提條件:
1.有3臺虛擬機centos7，虛擬機已經安裝好了docker
2. 每一個虛擬機導入 k8s.gcr.io/etcd:3.3.10 鏡像
如果您無法下載可以下載 docker pull ascs/k8s-gcr-io-etcd:3.3.10

然後 docker image tag ascs/k8s-gcr-io-etcd:3.3.10 k8s.gcr.io/etcd:3.3.10

部署安全的etcd集羣。
本指南說明了如何設置高可用性etcd服務器集羣以及如何確保與TLS的通信安全。本指南改編自官方的etcd文檔，您可以在其中找到更多詳細信息。
證書生成要啓用TLS，您需要生成自簽名證書頒發機構和服務器證書。在此示例中，我們將考慮使用以下節點作爲etcd服務器。

主機名	FQDN	IP地址
etcd1	etcd1.linkaixin.com	10.6.4.31
etcd2	etcd2.linkaixin.com	10.6.4.32
etcd3	etcd3.linkaixin.com	10.6.4.33

注意：爲了獲得高可用性，最好使用奇數個服務器。添加更多服務器可提高高可用性，並可以提高讀取性能，但會降低寫入性能。建議使用3、5或7個服務器。爲了生成CA和服務器證書，我們按照官方文檔中的建議使用Cloudflare的cfssl。它可以很容易地安裝，如下所示：

第一步: 創建證書

mkdir ~/bin
curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x ~/bin/{cfssl,cfssljson}
export PATH=$PATH:~/bin

#創建一個目錄來保存您的證書和私鑰。如果需要生成更多證書，將來可能會需要它們，因此請確保將它們保存在具有受限訪權限的安全位置：

mkdir ~/etcd-ca
cd ~/etcd-ca

#生成CA證書
echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json

#對於每個etcd服務器，生成如下證書：

export NAME=etcd1
export DOMAIN="linkaixin.com"
export ADDRESS=10.6.4.31,$NAME.$DOMAIN,$NAME
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

export NAME=etcd2
export DOMAIN="linkaixin.com"
export ADDRESS=10.6.4.32,$NAME.$DOMAIN,$NAME
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

export NAME=etcd3
export DOMAIN="linkaixin.com"
export ADDRESS=10.6.4.33,$NAME.$DOMAIN,$NAME
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

在etcd1 機器上
依次執行上面的命令

得到如下文件列表:

[ceph@ceph1 etcd-ca]$ pwd
/home/ceph/etcd-ca
[ceph@ceph1 etcd-ca]$ ls -al
總用量 52
drwxrwxr-x 2 ceph ceph 239 10月 28 15:01 .
drwx------. 7 ceph ceph 168 10月 28 15:00 …
-rw-rw-r-- 1 ceph ceph 112 10月 28 15:00 ca-config.json
-rw-r–r-- 1 ceph ceph 883 10月 28 15:00 ca.csr
-rw------- 1 ceph ceph 1679 10月 28 15:00 ca-key.pem
-rw-rw-r-- 1 ceph ceph 1119 10月 28 15:00 ca.pem
-rw-r–r-- 1 ceph ceph 928 10月 28 15:01 etcd1.csr
-rw------- 1 ceph ceph 1679 10月 28 15:01 etcd1-key.pem
-rw-rw-r-- 1 ceph ceph 1220 10月 28 15:01 etcd1.pem
-rw-r–r-- 1 ceph ceph 928 10月 28 15:01 etcd2.csr
-rw------- 1 ceph ceph 1679 10月 28 15:01 etcd2-key.pem
-rw-rw-r-- 1 ceph ceph 1220 10月 28 15:01 etcd2.pem
-rw-r–r-- 1 ceph ceph 928 10月 28 15:01 etcd3.csr
-rw------- 1 ceph ceph 1679 10月 28 15:01 etcd3-key.pem
-rw-rw-r-- 1 ceph ceph 1220 10月 28 15:01 etcd3.pem

注意：
如果將通過其他IP或DNS別名訪問服務器，請確保在ADDRESS變量中引用它們。(當你的etcd機器上有多個IP時或多個域名指向時)

現在，您必須在每個服務器節點的/ etc / etcd /目錄中部署生成的密鑰和證書。

#在etcd-ca目錄下分別爲3個etcd 節點創建對應的證書

 mkdir -p deploy/etcd1
 mkdir -p deploy/etcd2
 mkdir -p deploy/etcd3


cd etcd-ca/deploy/etcd1

cp ../../ca.pem etcd-ca.crt
cp ../../etcd1.pem server.crt
cp ../../etcd1-key.pem server.key


cd etcd-ca/deploy/etcd2
cp ../../ca.pem etcd-ca.crt
cp ../../etcd2.pem server.crt
cp ../../etcd2-key.pem server.key


cd etcd-ca/deploy/etcd3
cp ../../ca.pem etcd-ca.crt
cp ../../etcd3.pem server.crt
cp ../../etcd3-key.pem server.key

確保通信安全：
要爲etcd配置安全的對等通信，請指定標誌 --peer-key-file=peer.key 和 --peer-cert-file=peer.cert,並使用https作爲URL架構。
同樣，要爲etcd配置安全的客戶端通信，請指定標誌 --key-file=k8sclient.key and --cert-file=k8sclient.cert, 並使用https作爲URL架構。

注意：
稍後將必須在承載pcocc的所有節點（前端和計算節點）上部署CA證書ca.pem。確保與整個etcd-ca目錄一起保留備份。
etcd配置：
需要在/etc/etcd/etcd.conf配置文件中的每個服務器節點上配置etcd。這是node1的示例：
ETCD_NAME=node1
ETCD_LISTEN_PEER_URLS=“https://10.19.213.101:2380”
ETCD_LISTEN_CLIENT_URLS=“https://10.19.213.101:2379”
ETCD_INITIAL_CLUSTER_TOKEN=“pcocc-etcd-cluster”
ETCD_INITIAL_CLUSTER=“node1=https://node1.mydomain.com:2380,node2=https://node2.mydomain.com:2380,node3=https://node3.mydomain.com:2380”
ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://node1.mydomain.com:2380”
ETCD_ADVERTISE_CLIENT_URLS=“https://node1.mydomain.com:2379”
ETCD_TRUSTED_CA_FILE=/etc/etcd/etcd-ca.crt
ETCD_CERT_FILE="/etc/etcd/server.crt"
ETCD_KEY_FILE="/etc/etcd/server.key"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/etcd-ca.crt
ETCD_PEER_KEY_FILE=/etc/etcd/server.key
ETCD_PEER_CERT_FILE=/etc/etcd/server.crt

注意
ETCD_NAME，ETCD_ADVERTISE_CLIENT_URLS，ETCD_INITIAL_ADVERTISE_PEER_URLS，ETCD_LISTEN_PEER_URLS和ETCD_LISTEN_CLIENT_URLS必須適合每個服務器節點。

最後，您可以在所有etcd節點上啓用並啓動服務：
在 etcd1,etcd2,etcd3 創建目錄/var/server/secure-etcd
mkdir -p /var/server/secure-etcd
拷貝對應的證書過去和對應的腳本過去。

/var/server/secure-etcd
[ceph@ceph1 secure-etcd]$ ls
data ssl start-etcd-cluster.sh
[ceph@ceph1 secure-etcd]$

[ceph@ceph2 secure-etcd]$ pwd
/var/server/secure-etcd
[ceph@ceph2 secure-etcd]$ ls -al
總用量 20
drwxrwxr-x 4 ceph ceph 125 10月 28 15:07 .
drwxr-xr-x. 7 ceph ceph 266 10月 28 11:30 …
drwxr-xr-x 3 root root 20 10月 28 13:49 data
-rw-r–r-- 1 ceph ceph 4096 10月 28 11:31 ._.DS_Store
-rw-r–r-- 1 ceph ceph 6148 10月 28 11:31 .DS_Store
drwxrwxr-x 2 ceph ceph 68 10月 28 12:02 ssl
-rw-r–r-- 1 ceph ceph 4096 10月 28 11:31 ._start-etcd-cluster.sh
-rwxr-xr-x 1 ceph ceph 1856 10月 28 13:49 start-etcd-cluster.sh
[ceph@ceph2 secure-etcd]$

/var/server/secure-etcd
[ceph@ceph3 secure-etcd]$ ls -al
總用量 20
drwxrwxr-x 4 root root 125 10月 28 15:07 .
drwxr-xr-x. 6 ceph ceph 255 10月 28 11:31 …
drwxr-xr-x 3 root root 20 10月 28 13:50 data
-rw-r–r-- 1 root root 4096 10月 28 11:31 ._.DS_Store
-rw-r–r-- 1 root root 6148 10月 28 11:31 .DS_Store
drwxrwxr-x 2 root root 68 10月 28 12:09 ssl
-rw-r–r-- 1 root root 4096 10月 28 11:31 ._start-etcd-cluster.sh
-rwxr-xr-x 1 ceph ceph 1856 10月 28 13:50 start-etcd-cluster.sh
[ceph@ceph3 secure-etcd]$

使用docker 啓動etcd

[ceph@ceph1 secure-etcd]$ more start-etcd-cluster.sh

#!/bin/bash
sudo systemctl start firewalld
sudo systemctl enable firewalld
firewall-cmd --add-port=2379/tcp --permanent --zone=public
firewall-cmd --add-port=2380/tcp --permanent --zone=public

current_file_path=$(cd "$(dirname "$0")"; pwd)
cd ${current_file_path}

ETCD_INITIAL_SECURE_CLUSTER="etcd1=https://10.6.4.31:2380,etcd2=https://10.6.4.32:2380,etcd3=https://10.6.4.333:2380"

ETCD_INITIAL_CLUSTER_STATE=new

#export currentHostIp=`ip -4 address show eth0 | grep 'inet' |  grep -v grep | awk '{print $2}' | cut -d '/' -f1`

firewall-cmd --reload
firewall-cmd --list-all

#注意防火牆出現奇怪問題,集羣可能還無法訪問,只能本機訪問,需要重現啓動可以解決問題. to do

docker stop etcd1
docker rm   etcd1

docker run \
  -d \
  --restart=always \
  --hostname=etcd1 \
  -p 2379:2379 \
  -p 2380:2380 \
  -v /etc/localtime:/etc/localtime \
  -v `pwd`/data:/data \
  -v `pwd`/ssl:/etc/etcd/ssl \
  --name etcd1 \
  nexus.linkaixin.com:2443/k8s.gcr.io/etcd:3.3.10 \
   etcd \
  -name etcd1 \
  --cert-file=/etc/etcd/ssl/etcd1-server.crt \
  --key-file=/etc/etcd/ssl/etcd1-server.key \
  --peer-cert-file=/etc/etcd/ssl/etcd1-server.crt \
  --peer-key-file=/etc/etcd/ssl/etcd1-server.key \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://10.6.4.31:2380 \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --advertise-client-urls https://10.6.4.31:2379 \
  --initial-cluster-token etcd-cluster-of-ceph \
  --initial-cluster etcd1=https://10.6.4.31:2380,etcd2=https://10.6.4.32:2380,etcd3=https://10.6.4.33:2380 \
  --initial-cluster-state new \
  --data-dir=/data

docker logs -f etcd1

etcd2 啓動腳本如下:

[ceph@ceph2 secure-etcd]$ more start-etcd-cluster.sh
#!/bin/bash
sudo systemctl start firewalld
sudo systemctl enable firewalld
firewall-cmd --add-port=2379/tcp --permanent --zone=public
firewall-cmd --add-port=2380/tcp --permanent --zone=public

current_file_path=$(cd "$(dirname "$0")"; pwd)
cd ${current_file_path}

#ETCD_INITIAL_CLUSTER="etcd1=http://10.6.4.31:2380,etcd2=http://10.6.4.32:2380,etcd3=http://10.6.4.333:2380"
ETCD_INITIAL_SECURE_CLUSTER="etcd1=https://10.6.4.31:2380,etcd2=https://10.6.4.32:2380,etcd3=https://10.6.4.333:2380"

ETCD_INITIAL_CLUSTER_STATE=new

#export currentHostIp=`ip -4 address show eth0 | grep 'inet' |  grep -v grep | awk '{print $2}' | cut -d '/' -f1`

firewall-cmd --reload
firewall-cmd --list-all

#注意防火牆出現奇怪問題,集羣可能還無法訪問,只能本機訪問,需要重現啓動可以解決問題. to do

docker stop etcd2
docker rm   etcd2

docker run \
  -d \
  --restart=always \
  --hostname=etcd2 \
  -p 2379:2379 \
  -p 2380:2380 \
  -v /etc/localtime:/etc/localtime \
  -v `pwd`/data:/data \
  -v `pwd`/ssl:/etc/etcd/ssl \
  --name etcd2 \
  nexus.linkaixin.com:2443/k8s.gcr.io/etcd:3.3.10 \
   etcd \
  -name etcd2 \
  --cert-file=/etc/etcd/ssl/etcd2-server.crt \
  --key-file=/etc/etcd/ssl/etcd2-server.key \
  --peer-cert-file=/etc/etcd/ssl/etcd2-server.crt \
  --peer-key-file=/etc/etcd/ssl/etcd2-server.key \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://10.6.4.32:2380 \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --advertise-client-urls https://10.6.4.32:2379 \
  --initial-cluster-token etcd-cluster-of-ceph \
  --initial-cluster etcd1=https://10.6.4.31:2380,etcd2=https://10.6.4.32:2380,etcd3=https://10.6.4.33:2380 \
  --initial-cluster-state existing \
  --data-dir=/data

docker logs -f etcd2

etcd3 啓動腳本如下:

[ceph@ceph3 secure-etcd]$ more start-etcd-cluster.sh
#!/bin/bash
sudo systemctl start firewalld
sudo systemctl enable firewalld
firewall-cmd --add-port=2379/tcp --permanent --zone=public
firewall-cmd --add-port=2380/tcp --permanent --zone=public

current_file_path=$(cd "$(dirname "$0")"; pwd)
cd ${current_file_path}

#ETCD_INITIAL_CLUSTER="etcd1=http://10.6.4.31:2380,etcd2=http://10.6.4.32:2380,etcd3=http://10.6.4.333:2380"
ETCD_INITIAL_SECURE_CLUSTER="etcd1=https://10.6.4.31:2380,etcd2=https://10.6.4.32:2380,etcd3=https://10.6.4.333:2380"

ETCD_INITIAL_CLUSTER_STATE=new

#export currentHostIp=`ip -4 address show eth0 | grep 'inet' |  grep -v grep | awk '{print $2}' | cut -d '/' -f1`

firewall-cmd --reload
firewall-cmd --list-all

#注意防火牆出現奇怪問題,集羣可能還無法訪問,只能本機訪問,需要重現啓動可以解決問題. to do

docker stop etcd3
docker rm   etcd3

docker run \
  -d \
  --restart=always \
  --hostname=etcd3 \
  -p 2379:2379 \
  -p 2380:2380 \
  -v /etc/localtime:/etc/localtime \
  -v `pwd`/data:/data \
  -v `pwd`/ssl:/etc/etcd/ssl \
  --name etcd3 \
  nexus.linkaixin.com:2443/k8s.gcr.io/etcd:3.3.10 \
   etcd \
  -name etcd3 \
  --cert-file=/etc/etcd/ssl/etcd3-server.crt \
  --key-file=/etc/etcd/ssl/etcd3-server.key \
  --peer-cert-file=/etc/etcd/ssl/etcd3-server.crt \
  --peer-key-file=/etc/etcd/ssl/etcd3-server.key \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://10.6.4.33:2380 \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --advertise-client-urls https://10.6.4.33:2379 \
  --initial-cluster-token etcd-cluster-of-ceph \
  --initial-cluster etcd1=https://10.6.4.31:2380,etcd2=https://10.6.4.32:2380,etcd3=https://10.6.4.33:2380 \
  --initial-cluster-state existing \
  --data-dir=/data

docker logs -f etcd3

檢查ETCD狀態
要檢查您的etcd服務器是否正常運行，可以執行以下操作：
$ etcdctl --endpoints=https://node1.mydomain.com:2379 --ca-file=~/etcd-ca/ca.pem member list
6c86f26914e6ace, started, Node2, https://node3.mydomain.com:2380, https://node3.mydomain.com:2379
1ca80865c0583c45, started, Node1, https://node2.mydomain.com:2380, https://node2.mydomain.com:2379
99c7caa3f8dfeb70, started, Node0, https://node1.mydomain.com:2380, https://node1.mydomain.com:2379

[ceph@ceph1 ~]$ docker exec etcd1 etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem member list
2e26c284a608e734: name=etcd1 peerURLs=https://10.6.4.31:2380 clientURLs=https://10.6.4.31:2379 isLeader=true
86110446cab57f89: name=etcd2 peerURLs=https://10.6.4.32:2380 clientURLs=https://10.6.4.32:2379 isLeader=false
ebbc05ecb174c1c3: name=etcd3 peerURLs=https://10.6.4.33:2380 clientURLs=https://10.6.4.33:2379 isLeader=false

瀏覽器查看狀態:

爲pcocc配置etcd
啓用身份驗證之前，請在etcd中配置root用戶：
etcdctl --endpoints=“https://node1.mydomain.com:2379” --ca-file=~/etcd-ca/ca.pem user add root

docker exec etcd1
/ # etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem user add root
New password:
User root created
/ # etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem user add etcd
New password:
User etcd created

docker exec etcd1 etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem user add root

警告
選擇一個安全密碼。您必須在pcocc配置文件中引用它。
啓用身份驗證：
etcdctl --endpoints=“https://node1.mydomain.com:2379” --ca-file=~/etcd-ca/ca.pem auth enable
etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem auth enable

刪除Guest角色:
$ etcdctl --endpoints=“https://node1.mydomain.com:2379” --ca-file=~/etcd-ca/ca.pem -u root: role remove guest
Role guest removed

etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem -u root:kaixin.com role remove guest

未經身份驗證，您將不再能夠訪問密鑰庫：
$ etcdctl --endpoints “https://node1.mydomain.com:2379” --ca-file=~/etcd-ca/ca.pem get /
Error: 110: The request requires user authentication (Insufficient credentials) [0]

測試集羣狀態：

#在ceph1 上操作添加一個key=test value=test123
etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem -u root:kaixin.com set test test123
test123
/ # etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem -u root:kaixin.com get test

測試集羣是非正常
[ceph@ceph2 secure-etcd]$ docker exec -ti etcd2 sh
/ # etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem -u root:kaixin.com get test
test123
/ # etcdctl --endpoints=https://10.6.4.31:2379 --ca-file=/etc/etcd/ssl/ca.pem get test
Error: 110: The request requires user authentication (Insufficient credentials) [0]

錯誤:
etcdmain: member has already been bootstrapped
解決辦法:
修改啓動shell腳本
–initial-cluster-state new 爲
–initial-cluster-state existing

創建加密的ETCD docker集羣實戰記錄

docker 掛載攝像頭出現的問題解決辦法

把docker狀態變成kafka實時數據流---一行代碼寫了2天的shell代碼

SpringCloud gateway 動態路由入門

簡單的命令快速下載遠程各種docker 鏡像 github 庫，各種軟件資料。

開源物聯網雲平臺 Thingsboard入門

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

創建加密的ETCD docker集羣 實戰記錄

創建加密的ETCD docker集羣實戰記錄