交付redis 到k8s 集羣

1. 創建nfs 共享存儲

# 在所有主機上：
yum install nfs-utils rpcbind -y

在10.4.7.200 上部署 nfs server

cat[root@hdss7-200 nginxcurl]# cat /etc/exports
/data/nfs-volume 10.4.7.0/24(rw,no_root_squash)
/data/volumes 10.4.7.0/24(rw,no_root_squash)
/data/nfs-volume/redis/pv1 *(rw,all_squash)
/data/nfs-volume/redis/pv2 *(rw,all_squash)
/data/nfs-volume/redis/pv3 *(rw,all_squash)
/data/nfs-volume/redis/pv4 *(rw,all_squash)
/data/nfs-volume/redis/pv5 *(rw,all_squash)
/data/nfs-volume/redis/pv6 *(rw,all_squash)

chmod 777 /data/nfs-volume/redis/pv*   # 不加這個會出現 mount.nfs: access denied by server while mounting的報錯。

systemctl start rpcbind
systemctl start nfs

2. 創建PV

在 7.200 上創建6個pv，後面準備申請6個pod

cat pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv1
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv1"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv2
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv2"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv3
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv3"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv4
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv4"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv5
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv5"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv6
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv6"

kubectl create -f pv.yaml

3.創建Configmap

cat redis.conf 
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379

kubectl create configmap redis-conf --from-file=redis.conf  #用redis.conf 創建 redis-conf configmap

kubectl describe cm redis-conf  
Name:         redis-conf
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
redis.conf:
----
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379


Events:  <none>

4.創建Redis 集羣節點

有狀態服務用statefull 控制器

cat stateful.yaml 
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-app
spec:
  serviceName: "redis-service"
  replicas: 6
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis
              topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: "harbor.od.com/public/redis:6.0"
        command:
          - "redis-server"
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
            - name: redis
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
          - name: "redis-conf"
            mountPath: "/etc/redis"
          - name: "redis-data"   
            mountPath: "/var/lib/redis"
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-conf"
          items:
            - key: "redis.conf"
              path: "redis.conf"
  #這個就是pvc，關聯之前創建的pv。            
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 200M

　　volumemounts 指定兩個name 一個是 configmap 的，一個是pvc ，名字叫 redis-data, pvc 用來關聯pv，這裏用來存放redis 的數據的。

查看 redis 的pod

用 kubectl get pod -o wide --watch 可以創建的過程，會發現依次從0 到 5 逐個創建的，可以發現6個pod 都創建好了，但是現在pod 之間只能通過ip 訪問，statefull 設計宗旨是要通過域名訪問的，所以下面要

創建service，

 kubectl get pod -o wide --show-labels
NAME                                READY   STATUS              RESTARTS   AGE     IP            NODE                NOMINATED NODE   READINESS GATES   LABELS

pod-vol-pvc                         1/1     Running             0          10d     172.7.21.4    hdss7-21.host.com   <none>           <none>            <none>
redis-app-0                         1/1     Running             0          28h     172.7.21.8    hdss7-21.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-0
redis-app-1                         1/1     Running             0          28h     172.7.22.12   hdss7-22.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-1
redis-app-2                         1/1     Running             0          28h     172.7.21.9    hdss7-21.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-2
redis-app-3                         1/1     Running             0          28h     172.7.22.15   hdss7-22.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-3
redis-app-4                         1/1     Running             0          28h     172.7.21.10   hdss7-21.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-4
redis-app-5                         1/1     Running             0          28h     172.7.22.16   hdss7-22.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-5
web-0                               0/1     ContainerCreating   0          10d     <none>        hdss7-21.host.com   <none>           <none>            app=nginx,controller-revision-hash=web-7f49674c6,statefulset.kubernetes.io/pod-name=web-0

　　查看pv pvc，發現pvc 和pv 已經綁定了，pod 也關聯上了pv。

[root@hdss7-21 ~]# kubectl get pv,pvc -o wide 
NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                            STORAGECLASS   REASON   AGE   VOLUMEMODE
persistentvolume/nfs-pv1   200M       RWX            Retain           Bound         default/redis-data-redis-app-4                           44h   Filesystem
persistentvolume/nfs-pv3   200M       RWX            Retain           Bound         default/redis-data-redis-app-3                           44h   Filesystem
persistentvolume/nfs-pv4   200M       RWX            Retain           Bound         default/redis-data-redis-app-5                           44h   Filesystem
persistentvolume/nfs-pv5   200M       RWX            Retain           Bound         default/redis-data-redis-app-1                           44h   Filesystem
persistentvolume/nfs-pv6   200M       RWX            Retain           Available                                                              44h   Filesystem
persistentvolume/nfs-pv2   200M       RWX            Retain           Bound         default/redis-data-redis-app-2                           44h   Filesystem
persistentvolume/pv001     1Gi        RWO,RWX        Retain           Bound         default/redis-data-redis-app-0                           10d   Filesystem
persistentvolume/pv002     2Gi        RWO            Retain           Terminating   default/www-web-0                                        10d   Filesystem
persistentvolume/pv004     4Gi        RWO,RWX        Retain           Bound         default/mypvc                                            10d   Filesystem

NAME                                           STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
persistentvolumeclaim/mypvc                    Bound    pv004     4Gi        RWO,RWX                       10d   Filesystem
persistentvolumeclaim/redis-data-redis-app-0   Bound    pv001     1Gi        RWO,RWX                       46h   Filesystem
persistentvolumeclaim/redis-data-redis-app-1   Bound    nfs-pv5   200M       RWX                           28h   Filesystem
persistentvolumeclaim/redis-data-redis-app-2   Bound    nfs-vp2   200M       RWX                           46h   Filesystem
persistentvolumeclaim/redis-data-redis-app-3   Bound    nfs-pv3   200M       RWX                           44h   Filesystem
persistentvolumeclaim/redis-data-redis-app-4   Bound    nfs-pv1   200M       RWX                           44h   Filesystem
persistentvolumeclaim/redis-data-redis-app-5   Bound    nfs-pv4   200M       RWX                           44h   Filesystem

5. 創建redis 的service

用來實現pod 之間域名訪問， statefull 控制器的service 的 clusterIP: None ，這個要注意！service 通過標籤

app: redis
appCluster: redis-cluster   找到pod

[root@hdss7-200 redis]# cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    port: 6379
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster

查看 service

名字 redis-service 是剛創建的，這時候我們就可以通過 pod 名字. service 名字訪問pod 了，例如 redis-app-1.redis-service 這樣的。

kubectl get svc -o wide --show-labels
NAME                   TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE   SELECTOR                             LABELS
kubernetes             ClusterIP   192.168.0.1       <none>        443/TCP    71d   <none>                               component=apiserver,provider=kubernetes
nginx                  ClusterIP   None              <none>        80/TCP     11d   app=nginx                            app=nginx
redis-service          ClusterIP   None              <none>        6379/TCP   28h   app=redis,appCluster=redis-cluster   app=redis

　　新版busybox 的坑

由於redis 鏡像裏沒有nslookup 命令，這裏起一個busybox 鏡像來驗證。這裏有個坑，最新的busybox 鏡像會解析失敗，當時我還以爲我的coredns 有問題了，後來發現是 busybox 鏡像問題，用 1.28.4 就沒有問題了。

參考：

nslookup redis-app-1.redis-service  # 格式是 pod 名字.service名字.namespace名字，如果是default 可以不用謝namespace 名字。

kubectl run -i --tty --image busybox:1.28.4 dns-test --restart=Never --rm /bin/sh
/ # nslookup redis-app-1.redis-service
Server:    192.168.0.2
Address 1: 192.168.0.2 coredns.kube-system.svc.cluster.local

Name:      redis-app-1.redis-service
Address 1: 172.7.22.12 172-7-22-12.lightspeed.bcvloh.sbcglobal.net
/ # 
/ # nslookup redis-app-0.redis-service
Server:    192.168.0.2
Address 1: 192.168.0.2 coredns.kube-system.svc.cluster.local

Name:      redis-app-0.redis-service
Address 1: 172.7.21.8 172-7-21-8.lightspeed.bcvloh.sbcglobal.net
/ # cat /etc/resolv.conf 
nameserver 192.168.0.2
search default.svc.cluster.local svc.cluster.local cluster.local host.com
options ndots:5

　　發現都可以解析了。

6.初始化Redis集羣

先安裝 redis-trib.rb 命令的ruby 依賴環境，在10.4.7.21 上操作。 redis-trib.rb 在redis包的二進制文件裏面。

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
gem sources --add http://mirrors.aliyun.com/rubygems/ --remove https://rubygems.org/
yum install ruby rubygems  -y　 && gem install redis -v 3.3.5       //安裝版本3.3.5，針對redis3.0.6,redis-trib.rb reshard 操作可以成功
#gem list      //查看已經安裝的軟件列表

　　創建redis-cluster 集羣----- 有三個主從節點，

--replicas 1 表示集羣中的每個主節點創建一個從節點，前面三個是主節點，後面三個是slave 節點。

redis-trib.rb create --replicas 1 172.7.21.8:6379  172.7.21.9:6379 172.7.21.10:6379   172.7.22.14:6379 172.7.22.12:6379 172.7.22.15:6379

　　創建後隨便登錄一臺查看，可以看到集羣中有6個節點了。剩下的操作就和普通redis-cluster 操作一樣了。

redis-cli -h 172.7.22.15 -c 
172.7.22.15:6379> CLUSTER NODES
3b69afaaa38f700c057b3c0310765dc0082d71dd 172.7.22.12:6379@16379 slave cdb68a9c2c68657d344ce81a10529d0c760ad411 0 1593896053552 3 connected
cdb68a9c2c68657d344ce81a10529d0c760ad411 172.7.21.9:6379@16379 master - 0 1593896054564 3 connected 8192-16383
dec1a6f7cf8668ea987fe06b5ca05df5f054d198 172.7.22.15:6379@16379 myself,slave 4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 0 1593896054000 6 connected
ad855dc2ba9f2c3c409c05216068845e9a9f435e 172.7.21.10:6379@16379 slave 20c5a6057994439b84ac92ec3bbd8d76b1c80557 0 1593896053556 5 connected
20c5a6057994439b84ac92ec3bbd8d76b1c80557 172.7.21.8:6379@16379 master - 0 1593896054869 5 connected 286-8191
4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 172.7.22.14:6379@16379 master - 0 1593896053852 7 connected 0-285
172.7.22.15:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:7
cluster_stats_messages_ping_sent:9415
cluster_stats_messages_pong_sent:9545
cluster_stats_messages_sent:18960
cluster_stats_messages_ping_received:9545
cluster_stats_messages_pong_received:9415
cluster_stats_messages_update_received:2
cluster_stats_messages_received:18962

7. 創建提供給集羣內部的service

這裏是爲了外部訪問redis 用的。

cat ../redis/redissvc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: redis-access-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    protocol: "TCP"
    port: 6379
    targetPort: 6379
  selector:
    app: redis
    appCluster: redis-cluster

[root@hdss7-21 ~]# kubectl get svc -o wide --show-labels
NAME                   TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE    SELECTOR                             LABELS
kubernetes             ClusterIP   192.168.0.1       <none>        443/TCP    75d    <none>                               component=apiserver,provider=kubernetes
nginx                  ClusterIP   None              <none>        80/TCP     14d    app=nginx                            app=nginx
redis-access-service   ClusterIP   192.168.169.232   <none>        6379/TCP   5d3h   app=redis,appCluster=redis-cluster   app=redis
redis-service          ClusterIP   None              <none>        6379/TCP   3d8h   app=redis,appCluster=redis-cluster   app=redis

　　訪問 192.168.169.232 , 發現和連接redis pod 看到是一樣的效果！

redis-cli -c -h 192.168.169.232
192.168.169.232:6379> CLUSTER NODES
3b69afaaa38f700c057b3c0310765dc0082d71dd 172.7.22.12:6379@16379 slave cdb68a9c2c68657d344ce81a10529d0c760ad411 0 1593895154063 3 connected
cdb68a9c2c68657d344ce81a10529d0c760ad411 172.7.21.9:6379@16379 master - 0 1593895154594 3 connected 8192-16383
dec1a6f7cf8668ea987fe06b5ca05df5f054d198 172.7.22.15:6379@16379 myself,slave 4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 0 1593895154000 6 connected
ad855dc2ba9f2c3c409c05216068845e9a9f435e 172.7.21.10:6379@16379 slave 20c5a6057994439b84ac92ec3bbd8d76b1c80557 0 1593895155601 5 connected
20c5a6057994439b84ac92ec3bbd8d76b1c80557 172.7.21.8:6379@16379 master - 0 1593895155089 5 connected 286-8191
4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 172.7.22.14:6379@16379 master - 0 1593895154000 7 connected 0-285
192.168.169.232:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:7
cluster_stats_messages_ping_sent:8984
cluster_stats_messages_pong_sent:9113
cluster_stats_messages_sent:18097
cluster_stats_messages_ping_received:9113
cluster_stats_messages_pong_received:8984
cluster_stats_messages_update_received:2
cluster_stats_messages_received:18099

交付redis 到k8s 集羣

1. 創建nfs 共享存儲

2. 創建PV

3.創建Configmap

4.創建Redis 集羣節點

查看 redis 的pod

5. 創建redis 的service

查看 service

新版busybox 的坑

6.初始化Redis集羣

7. 創建提供給集羣內部的service

AI模型 Llama 3體驗筆記

【面試準備】又一次失敗的面試經歷，題目離譜～資深軟件測試工程師

dotnet 8 版本與銀河麒麟V10和UOS系統的 glibc 兼容性

用django 寫個用戶管理頁面

K8S flannel 網絡

交付redis 到k8s 集羣

交付dubbo 微服務到k8s 集羣

交付Jenkins 到k8s集羣

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結