在Kubernetes上搭建EFK（Fluentd＋Elasticsearch＋Kibana）

1. 前言

kubernetes 默認會將容器的stdout和stderr錄入node(minion)的/var/log/containers目錄下,而kubernetes 組件的日誌默認放置在/var/log目錄下。

如果你是用kube-up啓動的kubernetes集羣，那麼恭喜你,你可以方便的啓動k8s的日誌功能。參考:http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/ 或者在k8s安裝包的解壓目錄kubernetes/cluster/addons/fluentd-elasticsearch中找到安裝文件。

如果你是通過命令或者腳本啓動的k8s集羣,那麼也恭喜你,折騰等着你。

2. 實現需求

將各個pod中容器的stdout和stderr中的日誌集中展示。

3. 部署結構

待傳。

4. Elasticsearch & Kibana

這兩個使用官網的鏡像。Elasticsearch如果需要集羣化,參考:

https://hub.docker.com/r/fabric8/elasticsearch/tags/

https://github.com/fabric8io/elasticsearch-cloud-kubernetes

由於本人的k8s環境並不通外網,需要先從官網下載鏡像，再打上私有倉庫的標籤。

docker pull elasticsearch:2.3

docker pull kibana:4.5

docker tag elasticsearch:2.3 10.10.50.161:5000/elasticsearch:2.3

docker tag kibana:4.5 10.10.50.161:5000/kibana:4.5

docker push 10.10.50.161:5000/elasticsearch:2.3

docker push 10.10.50.161:5000/kibana:4.5

生成rc和svc:

kubectl create -f elasticsearch-kibana-svc.yaml
kubectl create -f elasticsearch-kibana-rc.yaml

elasticsearch-kibana-rc.yaml:

apiVersion: v1
kind: ReplicationController
metadata:
 name: elasticsearch-kibana
 namespace: kube-system
 labels:
 k8s-app: elasticsearch-kibana
 version: v1
 kubernetes.io/cluster-service: "true"
spec:
 replicas: 1
 selector:
 k8s-app: elasticsearch-kibana
 version: v1
 template:
 metadata:
 labels:
 k8s-app: elasticsearch-kibana
 version: v1
 kubernetes.io/cluster-service: "true"
 spec:
 containers:
 - image: 10.10.50.161:5000/elasticsearch:2.3
 name: elasticsearch 
 resources:
 # keep request = limit to keep this container in guaranteed class
 limits:
 cpu: 100m
 requests:
 cpu: 100m
 ports:
 - containerPort: 9200
 name: http
 protocol: TCP
 - containerPort: 9300
 name: transport
 protocol: TCP
 volumeMounts:
 - name: es-persistent-storage
 mountPath: /usr/share/elasticsearch/data
 - image: 10.10.50.161:5000/kibana:4.5
 name: kibana
 resources:
 # keep request = limit to keep this container in guaranteed class
 limits:
 cpu: 100m
 requests:
 cpu: 100m
 ports:
 - containerPort: 5601
 name: ui
 protocol: TCP
 env:
 - name: ELASTICSEARCH_URL
 value: http://localhost:9200
 volumes:
 - name: es-persistent-storage
 emptyDir: {}

elasticsearch-kibana-svc.yaml:

apiVersion: v1
kind: Service
metadata:
 name: elasticsearch-kibana
 namespace: kube-system
 labels:
 k8s-app: elasticsearch-kibana
 kubernetes.io/cluster-service: "true"
 kubernetes.io/name: "elasticsearch-kibana"
spec:
 type: NodePort
 ports:
 - name: elasticsearch-http
 port: 9200
 protocol: TCP
 targetPort: http
 - name: elasticsearch-transport
 port: 9300
 protocol: TCP
 targetPort: transport
 - name: kibana
 port: 5601
 protocol: TCP
 targetPort: ui
 nodePort: 30016
 selector:
 k8s-app: elasticsearch-kibana

注意:這裏我把elasticsearch的日誌存儲放在了pod的empty volumns裏了，真正的運行環境應該使用nodeSelector選定一臺node，並把日誌存儲放在node裏的/usr/share/elasticsearch/data。

5. Fluentd

重點就是fluentd,addons裏的依賴kube.up裏引用的make ca製作認證。而我的環境進行認證時，使用了前面一同事部署heapster+influxdb+grafana時用的單域名認證,之後再進行多域名認證時,搞得亂七八糟，索性決定繞過k8s內部環境。

多域名認證參考:

https://coreos.com/kubernetes/docs/latest/openssl.html

http://www.linuxidc.com/Linux/2014-10/108222.htm

http://apetec.com/support/GenerateSAN-CSR.htm

https://certificates.heanet.ie/node/17

這裏使用fabric8的fabric8/fluentd-kubernetes鏡像,但需要重新制作鏡像。

fabric8/fluentd-kubernetes本身不依賴https和認證,但裏面的fluentd插件fluent-plugin-kubernetes_metadata_filter依賴了https和認證,這也是比較蛋疼的事。

首先pull官網的鏡像

docker pull fabric8/fluentd-kubernetes:v1.14
docker tag fabric8/fluentd-kubernetes:v1.14 10.10.50.161:5000/fabric8/fluentd-kubernetes:v1.14

製作鏡像:

mkdir myfluent
cd fluent-plugin
touch Dockerfile
touch start-fluentd

Dockerfile:

FROM 10.10.50.161:5000/fabric8/fluentd-kubernetes:v1.14

MAINTAINER miaobainian <[email protected]>

ADD start-fluentd /start-fluentd

start-fluentd:

#!/bin/sh

ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST:-es-logging.default.svc}
ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT:-9200}
ELASTICSEARCH_SCHEME=${ELASTICSEARCH_SCHEME:-http}

FLUENTD_FLUSH_INTERVAL=${FLUENTD_FLUSH_INTERVAL:-10s}
FLUENTD_FLUSH_THREADS=${FLUENTD_FLUSH_THREADS:-1}
FLUENTD_RETRY_LIMIT=${FLUENTD_RETRY_LIMIT:-10}
FLUENTD_DISABLE_RETRY_LIMIT=${FLUENTD_DISABLE_RETRY_LIMIT:-true}
FLUENTD_RETRY_WAIT=${FLUENTD_RETRY_WAIT:-1s}
FLUENTD_MAX_RETRY_WAIT=${FLUENTD_MAX_RETRY_WAIT:-60s}
FLUENTD_BUFFER_CHUNK_LIMIT=${FLUENTD_BUFFER_CHUNK_LIMIT:-8m}
FLUENTD_BUFFER_QUEUE_LIMIT=${FLUENTD_BUFFER_QUEUE_LIMIT:-8192}
FLUENTD_BUFFER_TYPE=${FLUENTD_BUFFER_TYPE:-memory}
FLUENTD_BUFFER_PATH=${FLUENTD_BUFFER_PATH:-/var/fluentd/buffer}
FLUENTD_LOGSTASH_FORMAT=${FLUENTD_LOGSTASH_FORMAT:-true}

KUBERNETES_PRESERVE_JSON_LOG=${KUBERNETES_PRESERVE_JSON_LOG:-true}


mkdir /etc/fluent

cat << EOF >> /etc/fluent/fluent.conf
<source>
 type tail
 path /var/log/containers/*.log
 pos_file /var/log/es-containers.log.pos
 time_format %Y-%m-%dT%H:%M:%S.%N
 tag kubernetes.*
 format json
 read_from_head true
 keep_time_key true
</source>

<filter kubernetes.**>
 type kubernetes_metadata
 preserve_json_log ${KUBERNETES_PRESERVE_JSON_LOG}
 kubernetes_url ${KUBERNETES_URL}
 verify_ssl ${VERIFY_SSL}
</filter>

<match **>
 type elasticsearch$([ "${ELASTICSEARCH_DYNAMIC}" == "true" ] && echo _dynamic)
 log_level info
 include_tag_key true
 time_key time
 host ${ELASTICSEARCH_HOST}
 port ${ELASTICSEARCH_PORT}
 scheme ${ELASTICSEARCH_SCHEME}
 $([ -n "${ELASTICSEARCH_USER}" ] && echo user ${ELASTICSEARCH_USER})
 $([ -n "${ELASTICSEARCH_PASSWORD}" ] && echo password ${ELASTICSEARCH_PASSWORD})
 buffer_type ${FLUENTD_BUFFER_TYPE}
 $([ "${FLUENTD_BUFFER_TYPE}" == "file" ] && echo buffer_path ${FLUENTD_BUFFER_PATH})
 buffer_chunk_limit ${FLUENTD_BUFFER_CHUNK_LIMIT}
 buffer_queue_limit ${FLUENTD_BUFFER_QUEUE_LIMIT}
 flush_interval ${FLUENTD_FLUSH_INTERVAL}
 retry_limit ${FLUENTD_RETRY_LIMIT}
 $([ "${FLUENTD_DISABLE_RETRY_LIMIT}" == "true" ] && echo disable_retry_limit)
 retry_wait ${FLUENTD_RETRY_WAIT}
 max_retry_wait ${FLUENTD_MAX_RETRY_WAIT}
 num_threads ${FLUENTD_FLUSH_THREADS}
 logstash_format ${FLUENTD_LOGSTASH_FORMAT}
 $([ -n "${FLUENTD_LOGSTASH_PREFIX}" ] && echo logstash_prefix ${FLUENTD_LOGSTASH_PREFIX})
 reload_connections false
EOF

cat << 'EOF' >> /etc/fluent/fluent.conf
</match>
EOF

exec je fluentd

注意這裏比官方的start-fluentd增加關鍵的兩行:

kubernetes_url ${KUBERNETES_URL}
verify_ssl ${VERIFY_SSL}

目的是繞過https://clusterIp:443的驗證。

docker build -t 10.10.50.161:5000/fabric8/fluentd-kubernetes:v1.15 .
docker push 10.10.50.161:5000/fabric8/fluentd-kubernetes:v1.15

ok,fluentd的鏡像定義好了之後，就可以生成pod了，可以使用static pod,但是推薦使用daemon sets

touch fluentd-daemon.yaml

fluentd-daemon.yaml :

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
 name: fluentd-elasticsearch
 namespace: kube-system
 labels:
 k8s-app: fluentd-logging
spec:
 template:
 metadata:
 labels:
 k8s-app: fluentd-logging
 spec:
 containers:
 - name: fluentd-elasticsearch
 image: 10.10.50.161:5000/fabric8/fluentd-kubernetes:v1.16
 resources:
 limits:
 cpu: 100m
 volumeMounts:
 - name: varlog
 mountPath: /var/log
 - name: varlibdockercontainers
 mountPath: /var/lib/docker/containers
 readOnly: true
 env:
 - name: KUBERNETES_URL
 value: "http://10.10.50.156:8080/api"
 - name: VERIFY_SSL
 value: "false"
 - name: ELASTICSEARCH_HOST
 value: elasticsearch-kibana
 - name: ELASTICSEARCH_PORT
 value: "9200"
 - name: FLUENTD_FLUSH_INTERVAL
 value: "300s"
 volumes:
 - name: varlog
 hostPath:
 path: /var/log
 - name: varlibdockercontainers
 hostPath:
 path: /var/lib/docker/containers

kubectl create -f fluentd-daemon.yaml

注意這裏的環境變量:

KUBERNETES_URL使用的是k8s master node的master api。

VERIFY_SSL爲false表示不驗證ca。

ELASTICSEARCH_HOST是前面部署的elasticsearch-kibana服務名,依賴於dns(服務名即dns名),如果沒有裝dns，你也可以使用kubectl get svc --namespace=kube-system找到elasticsearch-kibana的集羣ip,配置集羣ip也可以。但集羣ip是可變的,這個要注意。

ELASTICSEARCH_PORT是elasticsearch-kibana中elasticsearch的服務內部端口。
FLUENTD_FLUSH_INTERVAL用於標識收集時間間隔,設置爲300s是因爲第一次收集時，花費的時間較長，時間間隔不夠會導致elasticsearch不停的重新連接。

ok,現在all is already。

在瀏覽器打開:

http://10.10.50.155:30016/

10.10.50.155是我的k8s集羣中的一個node,30016是elasticsearch-kibana服務的node port。

進行kibana的界面。

默認進入Settings的indices界面。

將Index contains time-based events的打勾去掉。點擊下面的create。

ok，喝杯coffee or tea,等待k8s的日誌出現就可以了。

本文轉移開源中國-在Kubernetes上搭建EFK（Fluentd＋Elasticsearch＋Kibana）

在Kubernetes上搭建EFK（Fluentd＋Elasticsearch＋Kibana）

如何使用 JS 判斷用戶是否處於活躍狀態

通過HPA+CronHPA組合應對業務複雜彈性伸縮場景

DockOne微信分享（一八三）：滴滴彈性雲Kubernetes實踐

Rook：基於Ceph的Kubernetes存儲解決方案

超越Kubernetes：值得關注的5大雲原生技術

選擇Serverless還是Kubernetes？這種爭辯並沒有意義

Docker和Kubernetes如何讓DevOps更具效力

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結