elasticsearch7.x在k8s中的部署

目錄

一、說明

二、思路

三、部署

1、建nfs服務器

2、建持久卷

3、部署elasticsearch

四、附件

 pv.yaml內容

elasticsearch.yaml內容

---

一、說明

        本文章內容主要的參考來源是https://www.cnblogs.com/javashop-docs/p/12410845.html，但參考文獻中的elasticsearch是用的6.x版本，7.x與6.x有些配置上的差異，因此有必要在此基礎上記錄一下。

二、思路

• 數據的存儲

       在k8s中的持久化部署不可避免的要用到持久卷，我們採用nfs方式的持久捲來存儲es數據。持久卷的詳細介紹請見這裏：
https://kubernetes.io/docs/concepts/storage/persistent-volumes/

• 節點規劃

 默認啓動5個節點，3主2數據。根據es官方推薦每個節點的智能要分離，因此maseter節點不存儲數據，只用來協調。

• 3、多節點的權限問題

       es的數據目錄默認只允許一個節點訪問，但在k8s上採用了持久卷，所有節點的數據都存儲在這個捲上，這會導致es的訪問權限問題。報錯如下：

java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0

當然可以通過更改es的配置max_local_storage_nodes來允許多個節點訪問同一個數據目錄，但es官方不推薦這樣做。

所以我們的方案是更改每個節點的數據存儲目錄來解決 ps:指定es配置項path.data來實現。舉例說明：

節點名	存儲目錄
es-data-1	/usr/share/elasticsearch/data/es-data-1
es-data-2	/usr/share/elasticsearch/data/es-data-2

三、部署

1、建nfs服務器

對於持久卷的結構規劃如下：

目錄	內容
/nfs/data/esmaster	es master節點的數據
/nfs/data/esdata	es 數據節點的數據

關於索引的磁盤佔用：請根據業務的數據量情況來規劃持久卷硬件的情況，根據我們實際測算1000個商品大約需要1MB/每節點
       在默認的規劃中，我們使用k8s的master節點作爲nfs服務器，爲上述卷準備了10G的空間，請確保k8s master node 不少於10G的空閒磁盤。請根據您的具體業務情況選擇nfs服務器，如果條件允許最好是獨立的nfs服務器。根據如上規劃建立nfs服務：

#master節點安裝nfs
yum -y install nfs-utils
#創建nfs目錄
mkdir -p /nfs/data/{mqdata,esmaster,esdata}
#修改權限
chmod -R 777 /nfs/data/

#編輯export文件
vim /etc/exports

粘貼如下內容：
/nfs/data/esmaster *(rw,no_root_squash,sync)
/nfs/data/esdata *(rw,no_root_squash,sync)

#配置生效
exportfs -r
#查看生效
exportfs

#啓動rpcbind、nfs服務
systemctl restart rpcbind && systemctl enable rpcbind
systemctl restart nfs && systemctl enable nfs

#查看 RPC 服務的註冊狀況
rpcinfo -p localhost

#showmount測試，這裏的ip輸入master節點的局域網ip
showmount -e <your ip>

如果以看到可被掛載的目錄：
# showmount -e 172.17.14.73                                                            

 Export list for 172.17.14.73:
/nfs/data/esmaster *                                                                                                                                            
/nfs/data/mqdata   *
接下來，要在每一個節點上安裝nfs服務以便使k8s可以掛載nfs目錄
#所有node節點安裝客戶端
yum -y install nfs-utils
systemctl start nfs && systemctl enable nfs

這樣就爲k8s的持久卷做好了準備。

2、建持久卷

複製附件中的pv.yaml內容，修改其中的server配置爲nfs服務器的ip地址

  nfs:
           server: 192.168.0.186  #這裏請寫nfs服務器的ip 

在k8s  master節點上執行下面的命令創建namespace：
kubectl create namespace ns-elasticsearch
通過下面的命令建立持久卷：
kubectl create -f pv.yaml
通過以下命令查看持久卷是否建立成功：
kubectl get pv

3、部署elasticsearch

由於在elasticsearch.yaml中設置了node信息：nodeSelector:es: enable，因此需要將node添加標籤es:enable
kubectl label nodes <node-name1> es=enable
kubectl label nodes <node-name2> es=enable
kubectl label nodes <node-name3> es=enable
查看node標籤
 k get node --show-labels
複製附件elasticsearch.yaml的內，並執行下面的命令創建es集羣
kubectl create -f elasticsearch.yaml
通過以上部署我們建立了一個ns-elasticsearch的namespace，並在其中創建了相應的pvc、角色賬號，有狀態副本集以及服務。
kubectl get pods --namespace  ns-elasticsearch -o wide

NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
elasticsearch-data-0     1/1     Running   0          46h   10.244.2.45   vm188   <none>           <none>
elasticsearch-data-1     1/1     Running   2          45h   10.244.1.62   vm187   <none>           <none>
elasticsearch-master-0   1/1     Running   0          46h   10.244.2.44   vm188   <none>           <none>
elasticsearch-master-1   1/1     Running   0          46h   10.244.1.61   vm187   <none>           <none>
elasticsearch-master-2   1/1     Running   0          46h   10.244.0.14   vm186   <none>           <none>

服務
我們默認開啓了對外nodeport端口，對應關係：

32000->9200

32100->9300

k8s內部可以通過下面的服務名稱訪問：

elasticsearch-api-service.ns-elasticsearch:9300

elasticsearch-service.ns-elasticsearch:9200

等待容器都啓動成功後驗證。

注意：

es的最大內存和最小內存需要保持一致，默認的256m太小，可適當增加，我配置的是1024m。

es7.x參考：

https://blog.csdn.net/chengyuqiang/article/details/89841544

https://www.sohu.com/a/301517999_683048

四、附件

 pv.yaml內容

---
#es master節點的持久卷
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-es-master
  labels:
    pv: pv-es-master   
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: nfs
  nfs:
    server: 192.168.0.186  #這裏請寫nfs服務器的ip 
    path: /nfs/data/esmaster    
---
#es數據節點的持久卷
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-es-data
  labels:
    pv: pv-es-data    
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: nfs
  nfs:
    server: 192.168.0.186  #這裏請寫nfs服務器的ip 
    path: /nfs/data/esdata
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-es-master
  namespace: ns-elasticsearch
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
  selector:
    matchLabels:
      pv: pv-es-master  
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-es-data
  namespace: ns-elasticsearch
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nfs
  selector:
    matchLabels:
      pv: pv-es-data

elasticsearch.yaml內容

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    elastic-app: elasticsearch
  name: elasticsearch-admin
  namespace: ns-elasticsearch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elasticsearch-admin
  labels:
    elastic-app: elasticsearch
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: elasticsearch-admin
    namespace: ns-elasticsearch

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    elastic-app: elasticsearch
    role: master
  name: elasticsearch-master
  namespace: ns-elasticsearch
spec:
  serviceName: es-master
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      elastic-app: elasticsearch
      role: master
  template:
    metadata:
      labels:
        elastic-app: elasticsearch
        role: master
    spec:
      #將持久卷聲明  
      volumes:
       - name: pv-storage-elastic-master
         persistentVolumeClaim:
           claimName: pvc-es-master     
      nodeSelector:
        es: enable
      containers:
        - name: elasticsearch-master
          image: elasticsearch:7.6.2
          lifecycle:
            postStart:
              exec:
                command: ["/bin/bash", "-c", "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data;"]
          ports:
            - containerPort: 9200
              protocol: TCP
            - containerPort: 9300
              protocol: TCP
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            #修改es默認的數據存儲目錄，否則多個節點同時寫一個目錄es權限不允許                                           
            - name: "path.data"
              value: "/usr/share/elasticsearch/data/$(MY_POD_NAME)"            
            - name: "cluster.name"
              value: "elasticsearch-cluster"
            - name: "bootstrap.memory_lock"
              value: "true"
            - name: "discovery.seed_hosts" #7.x的配置方式
              value: "elasticsearch-discovery"
            - name: "node.master"
              value: "true"
            - name: "node.data"
              value: "false"
            - name: "node.ingest"
              value: "false"
            - name: "ES_JAVA_OPTS"
              value: "-Xms1024m -Xmx1024m"
            - name: "cluster.initial_master_nodes"
              value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"
          securityContext:
            privileged: true
          #將持久卷映射爲數據目錄的父目錄  
          volumeMounts:
            - name: pv-storage-elastic-master
              mountPath: /usr/share/elasticsearch/data/            
      imagePullSecrets:
      - name: aliyun-secret            
      serviceAccountName: elasticsearch-admin
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---
kind: Service
apiVersion: v1
metadata:
  labels:
    elastic-app: elasticsearch
  name: elasticsearch-discovery
  namespace: ns-elasticsearch
spec:
  ports:
    - port: 9300
      targetPort: 9300
  selector:
    elastic-app: elasticsearch
    role: master

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    elastic-app: elasticsearch
    role: data
  name: elasticsearch-data
  namespace: ns-elasticsearch
spec:
  serviceName: es-data  
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      elastic-app: elasticsearch
  template:
    metadata:
      labels:
        elastic-app: elasticsearch
        role: data
    spec:
      #將es-data持久卷聲明
      volumes:
       - name: pv-storage-elastic-data
         persistentVolumeClaim:
           claimName: pvc-es-data 
      nodeSelector:
        es: enable
      containers:
        - name: elasticsearch-data
          image: elasticsearch:7.6.2
          lifecycle:
            postStart:
              exec:
                command: ["/bin/bash", "-c", "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data;"]
          ports:
            - containerPort: 9200
              protocol: TCP
            - containerPort: 9300
              protocol: TCP
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            #修改es默認的數據存儲目錄，否則多個節點同時寫一個目錄es權限不允許                  
            - name: "path.data"
              value: "/usr/share/elasticsearch/data/$(MY_POD_NAME)"
            - name: "cluster.name"
              value: "elasticsearch-cluster"
            - name: "bootstrap.memory_lock"
              value: "true"
            - name: "discovery.seed_hosts"
              value: "elasticsearch-discovery"
            - name: "node.master"
              value: "false"
            - name: "node.data"
              value: "true"
            - name: "ES_JAVA_OPTS"
              value: "-Xms1024m -Xmx1024m"
          securityContext:
            privileged: true
          #將持久卷映射到數據目錄的父目錄  
          volumeMounts:
            - name: pv-storage-elastic-data
              mountPath: /usr/share/elasticsearch/data/
      # imagePullSecrets:
      # - name: aliyun-secret          
      serviceAccountName: elasticsearch-admin
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---
kind: Service
apiVersion: v1
metadata:
  labels:
    elastic-app: elasticsearch-service
  name: elasticsearch-service
  namespace: ns-elasticsearch
spec:
  ports:
    - port: 9200
      targetPort: 9200
      nodePort: 32000
  selector:
    elastic-app: elasticsearch
  type: NodePort

---
kind: Service
apiVersion: v1
metadata:
  labels:
    elastic-app: elasticsearch-service
  name: elasticsearch-api-service
  namespace: ns-elasticsearch
spec:
  ports:
    - port: 9300
      targetPort: 9300
      nodePort: 32100
  selector:
    elastic-app: elasticsearch
  type: NodePort