Etcd 簡介
Etcd 是一種分佈式 kv 存儲設施, 他具有一定的一致性,高性能,高可用的方案. 類似的 zookeeper, 但沒有 zookeeper 那麼重型,功能也沒有覆蓋那麼多. 簡單直接的應用就是配置中心
架構設計總覽
clients 爲多個需要 配置的服務, 中間層爲 多個 grpc-proxy 做均衡負載, 以免一個 proxy 掛了之後 導致單點問題. grpc-proxy 可以 同時訪問多個 etcd 服務器,進行 kv 的操作. 如果某一個 server 掛了,會自動訪問別的 集羣中的其他 server 以保證高可用.
etcd cluster 至少爲 3 臺, 如果小於 3 臺則無法進行選舉,造成集羣 不可用. (這裏需要用 promethus 進行監控預警)
etcd cluster 本地搭建
- 本地安裝 etcd,或者直接下載預編譯好的執行文件
https://github.com/etcd-io/etcd/releases - 獲取 etcd 源碼 (其實只是要一個配置文件 Procfile)
- 安裝 goreman -> go get github.com/mattn/goreman
- 修改源碼根目錄的 Procfile
# Use goreman to run `go get github.com/mattn/goreman` etcd1: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd2: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd3: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd4: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra4 --listen-client-urls http://127.0.0.1:42379 --advertise-client-urls http://127.0.0.1:42379 --listen-peer-urls http://127.0.0.1:42380 --initial-advertise-peer-urls http://127.0.0.1:42380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd5: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra5 --listen-client-urls http://127.0.0.1:52379 --advertise-client-urls http://127.0.0.1:52379 --listen-peer-urls http://127.0.0.1:52380 --initial-advertise-peer-urls http://127.0.0.1:52380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr proxy1: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd grpc-proxy start --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 --listen-addr=127.0.0.1:23790 --advertise-client-url=127.0.0.1:23790 --enable-pprof proxy2: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd grpc-proxy start --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 --listen-addr=127.0.0.1:23791 --advertise-client-url=127.0.0.1:23791 --enable-pprof |
- 啓動集羣 goreman -f Procfile start
配置解讀
配置了 5 個 etcd 服務模擬不同的 server, 配置了 2 個 proxy, 模擬兩個 grpc-proxy 當做均衡負載
–listen-client-urls etcd 監聽的地址
–listen-peer-urls 集羣內部 member 互相監聽的地址
–name 集羣成員的名字
–advertise-client-urls 集羣內成員對外發布信息的地址
–initial-advertise-peer-urls 對集羣內發佈信息的地址
–initial-cluster-state Set to new for all members present during initial static or DNS bootstrapping. If this option is set to existing, etcd will attempt to join the existing cluster. If the wrong value is set, etcd will attempt to start but fail safely.
–initial-cluster-token 集羣的標示
5 個 etcd server 形成 etcd-cluster-1, 2 個 proxy 對這個集羣進行代理
集羣檢查
ETCDCTL_API=3 ./etcdctl endpoint –endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 status -w table
λ ETCDCTL_API=3 ./etcdctl endpoint --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 status -w table +-----------------+------------------+---------+---------+-----------+-----------+------------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | +-----------------+------------------+---------+---------+-----------+-----------+------------+ | 127.0.0.1:2379 | 8211f1d0f64f3269 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:22379 | 91bc3c398fb3c146 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:32379 | fd422379fda50e48 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:42379 | 45d559f8148de837 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:52379 | c91263fe1e1dd3b5 | 3.3.9 | 25 kB | true | 2 | 12 | +-----------------+------------------+---------+---------+-----------+-----------+------------+ |
λ ./etcdctl cluster-health member 45d559f8148de837 is healthy: got healthy result from http://127.0.0.1:42379 member 8211f1d0f64f3269 is healthy: got healthy result from http://127.0.0.1:2379 member 91bc3c398fb3c146 is healthy: got healthy result from http://127.0.0.1:22379 member c91263fe1e1dd3b5 is healthy: got healthy result from http://127.0.0.1:52379 member fd422379fda50e48 is healthy: got healthy result from http://127.0.0.1:32379 cluster is healthy |
編寫測試客戶端進行 kv 操作
go get go.etcd.io/etcd
package main
import (
"context"
"fmt"
"log"
"strconv"
"time"
"go.etcd.io/etcd/clientv3"
"vincent.com/etcd/etcd/etcdserver/api/v3rpc/rpctypes"
)
func main() {
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"http://127.0.0.1:23790", "http://127.0.0.1:23791"},
DialTimeout: 5 * time.Second,
})
if err != nil {
// handle error!
log.Fatalln(err.Error())
}
defer cli.Close()
defer cli.Delete(context.Background(), "sample_key")
for index := 0; index < 10000; index++ {
insertKV(cli, "sample_key", "sample_value"+strconv.Itoa(index))
if err != nil {
break
}
}
resp, err := cli.Get(context.Background(), "sample_key")
if err != nil {
log.Fatalln("get key error", err)
}
fmt.Printf("get the sample_key: %v\n", resp.Kvs)
}
func insertKV(cli *clientv3.Client, key string, value string) (err error) {
time.Sleep(2 * time.Millisecond)
_, err = cli.Put(context.Background(), key, value)
// cancel()
if err != nil {
switch err {
case context.Canceled:
log.Fatalf("ctx is canceled by another routine: %v", err)
case context.DeadlineExceeded:
log.Fatalf("ctx is attached with a deadline is exceeded: %v", err)
case rpctypes.ErrEmptyKey:
log.Fatalf("client-side error: %v", err)
default:
log.Fatalf("bad cluster endpoints, which are not etcd servers: %v", err)
}
}
return
}
|
客戶端解讀
etcd 客戶端支持集羣, 所以直接可以連接兩個 proxy
這裏進行了 1w 此的寫入,每次寫入會有 x ms 的延遲(我怕我的 mbp 受不了)
客戶端測試
- 運行客戶端程序
- 模擬 etcd server 掛掉 goreman run stop etcd2
- 當集羣內健康機器少於 3 臺的時候,客戶端報錯,集羣整體不可用
集羣性能檢測
λ ETCDCTL_API=3 ./etcdctl check perf load="l" --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00%1m0s PASS: Throughput is 150 writes/s PASS: Slowest request took 0.388842s PASS: Stddev is 0.029037s PASS |
注意:
如果用命令行,記得加 ETCDCTL_API=3