採集kubernetes的容器日誌

需求

/var/log/containers下面的文件其實是軟鏈接

真正的日誌文件在/var/lib/docker/containers這個目錄

可選方案:

1. Logstash(過於消耗內存,儘量不要用這個)
2. fluentd
3. filebeat
4. 不使用docker-driver

日誌的格式

/var/log/containers

{
    "log": "17:56:04.176 [http-nio-8080-exec-5] INFO  c.a.goods.proxy.GoodsGetServiceProxy - ------ request_id=514136795198259200,zdid=42,gid=108908071,從緩存中獲取數據:失敗 ------\n",
    "stream": "stdout",
    "time": "2018-11-19T09:56:04.176713636Z"
}

{
    "log": "[][WARN ][2018-11-19 18:13:48,896][http-nio-10080-exec-2][c.h.o.p.p.s.impl.PictureServiceImpl][[msg:圖片不符合要求:null];[code:400.imageUrl.invalid];[params:https://img.alicdn.com/bao/uploaded/i2/2680224805/TB2w5C9bY_I8KJjy1XaXXbsxpXa_!!2680224805.jpg];[stack:{\"requestId\":\"514141260156502016\",\"code\":\"400.imageUrl.invalid\",\"msg\":\"\",\"stackTrace\":[],\"suppressedExceptions\":[]}];]\n",
    "stream": "stdout",
    "time": "2018-11-19T10:13:48.896892566Z"
}

Logstash

• filebeat.yml

filebeat:
  prospectors:
  - type: log
    //開啓監視，不開不採集
    enable: true
    paths:  # 採集日誌的路徑這裏是容器內的path
    - /var/log/elkTest/error/*.log
    # 日誌多行合併採集
    multiline.pattern: '^\['
    multiline.negate: true
    multiline.match: after
    # 爲每個項目標識,或者分組，可區分不同格式的日誌
    tags: ["java-logs"]
    # 這個文件記錄日誌讀取的位置，如果容器重啓，可以從記錄的位置開始取日誌
    registry_file: /usr/share/filebeat/data/registry

output:
  # 輸出到logstash中
  logstash:
    hosts: ["0.0.0.0:5044"]

注：6.0以上該filebeat.yml需要掛載到/usr/share/filebeat/filebeat.yml,另外還需要掛載/usr/share/filebeat/data/registry 文件，避免filebeat容器掛了後，新起的重複收集日誌。

• logstash.conf

input {
    beats {
        port => 5044
    }
}
filter {
    
   if "java-logs" in [tags]{ 
     grok {
        match => {
           "message" => "(?<date>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\]\[(?<level>[A-Z]{4,5})\]\[(?<thread>[A-Za-z0-9/-]{4,40})\]\[(?<class>[A-Za-z0-9/.]{4,40})\]\[(?<msg>.*)"
        }
        remove_field => ["message"]
     }
    }
    #if ([message] =~ "^\[") {
    #    drop {}
    #}
    # 不匹配正則，匹配正則用=~
    if [level] !~ "(ERROR|WARN|INFO)" {
        drop {}
    }
}

## Add your filters / logstash plugins configuration here

output {
    elasticsearch {
        hosts => "0.0.0.0:9200"
    }
}

fluentd

fluentd-es-image鏡像

Kubernetes-基於EFK進行統一的日誌管理

Docker Logging via EFK (Elasticsearch + Fluentd + Kibana) Stack with Docker Compose

filebeat+ES pipeline

定義pipeline

• 定義java專用的管道

PUT /_ingest/pipeline/java
{
    "description": "[0]java[1]nginx[last]通用規則",
    "processors": [{
        "grok": {
            "field": "message",
            "patterns": [
                "\\[%{LOGLEVEL:level}\\s+?\\]\\[(?<date>\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3})\\]\\[(?<thread>[A-Za-z0-9/-]+?)\\]\\[%{JAVACLASS:class}\\]\\[(?<msg>[\\s\\S]*?)\\]\\[(?<stack>.*?)\\]"
            ]
        },"remove": {
              "field": "message"
            }
    }]
}

PUT /_ingest/pipeline/nginx
{
    "description": "[0]java[1]nginx[last]通用規則",
    "processors": [{
        "grok": {
            "field": "message",
            "patterns": [
                "%{IP:client} - - \\[(?<date>.*?)\\] \"(?<method>[A-Za-z]+?) (?<url>.*?)\" %{NUMBER:statuscode} %{NUMBER:duration} \"(?<refer>.*?)\" \"(?<user-agent>.*?)\"" 
            ]
        },"remove": {
              "field": "message"
            }
    }]
}

PUT /_ingest/pipeline/default
{
    "description": "[0]java[1]nginx[last]通用規則",
    "processors": []
}

PUT /_ingest/pipeline/all
{
    "description": "[0]java[1]nginx[last]通用規則",
    "processors": [{
        "grok": {
            "field": "message",
            "patterns": [
                "\\[%{LOGLEVEL:level}\\s+?\\]\\[(?<date>\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3})\\]\\[(?<thread>[A-Za-z0-9/-]+?)\\]\\[%{JAVACLASS:class}\\]\\[(?<msg>[\\s\\S]*?)\\]\\[(?<stack>.*?)\\]",
                
                "%{IP:client} - - \\[(?<date>.*?)\\] \"(?<method>[A-Za-z]+?) (?<url>.*?)\" %{NUMBER:statuscode} %{NUMBER:duration} \"(?<refer>.*?)\" \"(?<user-agent>.*?)\"",
                
                ".+"
            ]
        }
    }]
}

filebeat.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
        inputs:
          # Mounted `filebeat-inputs` configmap:
          path: ${path.config}/inputs.d/*.yml
          # Reload inputs configs as they change:
          reload.enabled: false
        modules:
          path: ${path.config}/modules.d/*.yml
          # Reload module configs as they change:
          reload.enabled: false
    setup.template.settings:
        index.number_of_replicas: 0

    # https://www.elastic.co/guide/en/beats/filebeat/6.5/filebeat-reference-yml.html
    # https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         templates:
             config:
               - type: docker
                 containers.ids:
                  # - "${data.kubernetes.container.id}" 
                  - "*"
                 enable: true
                 processors:
                  - add_kubernetes_metadata:
                      # include_annotations:
                      #   - annotation_to_include        
                      in_cluster: true
                  - add_cloud_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output:
      elasticsearch:
        hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
        # username: ${ELASTICSEARCH_USERNAME}
        # password: ${ELASTICSEARCH_PASSWORD}
        # pipelines:          
        #   - pipeline: "nginx"
        #     when.contains:
        #       kubernetes.container.name: "nginx-"
        #   - pipeline: "java"
        #     when.contains:
        #       kubernetes.container.name: "java-"              
        #   - pipeline: "default"  
        #     when.contains:
        #       kubernetes.container.name: ""
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      tolerations:
        - key: "elasticsearch-exclusive"
          operator: "Exists"
          effect: "NoSchedule"         
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          imagePullPolicy: Always          
          image: 'filebeat:6.6.0'
          args: [
            "-c", 
            "/etc/filebeat.yml",
            "-e",
          ]         
          env:
          - name: ELASTICSEARCH_HOST
            value: 0.0.0.0
          - name: ELASTICSEARCH_PORT
            value: "9200"
          # - name: ELASTICSEARCH_USERNAME
          #   value: elastic
          # - name: ELASTICSEARCH_PASSWORD
          #   value: changeme
          # - name: ELASTIC_CLOUD_ID
          #   value:
          # - name: ELASTIC_CLOUD_AUTH
          #   value:
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
          - name: config
            mountPath: /etc/filebeat.yml
            readOnly: true
            subPath: filebeat.yml
          - name: data
            mountPath: /usr/share/filebeat/data
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
            readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat

如果output是單節點elasticsearch,可以通過修改模板把導出的filebeat*設置爲0個副本

curl -X PUT "10.10.10.10:9200/_template/template_log" -H 'Content-Type: application/json' -d'
{
    "index_patterns" : ["filebeat*"],
    "order" : 0,
    "settings" : {
        "number_of_replicas" : 0
    }
}
'

參考鏈接:

1. running-on-kubernetes
2. ELK+Filebeat 集中式日誌解決方案詳解
3. filebeat.yml（中文配置詳解）
4. Elasticsearch Pipeline 詳解
5. es number_of_shards和number_of_replicas

其他方案

有些是sidecar模式,sidecar模式可以做得比較細緻.

1. 使用filebeat收集kubernetes中的應用日誌
2. 使用Logstash收集Kubernetes的應用日誌

阿里雲的方案

1. Kubernetes日誌採集流程

跟隨docker啓動

1. docker驅動

kubectl delete po $pod  -n kube-system
kubectl get po -l k8s-app=fluentd-es -n kube-system
pod=`kubectl get po -l k8s-app=fluentd-es -n kube-system | grep -Eoi 'fluentd-es-([a-z]|-|[0-9])+'` && kubectl logs $pod -n kube-system
kubectl get events -n kube-system | grep $pod