基于Kubernetes实现的大数据采集与存储实践总结

原創

2020-07-12 16:23

[^ ]*) (?[^ ]*)(?: \"(?[^\\\"]*)\" \"(?[^\\\"]*)\")?$\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name apache2\n Format regex\n Regex ^(?[^ ]*) [^ ]* (?[^ ]*) \\[(?[^\\]]*)\\] \"(?\\S+)(?: +(?[^ ]*) +\\S*)?\" (?[^ ]*) (?[^ ]*)(?: \"(?[^\\\"]*)\" \"(?[^\\\"]*)\")?$\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name apache_error\n Format regex\n Regex ^\\[[^ ]* (?[^\\]]*)\\] \\[(?[^\\]]*)\\](?: \\[pid (?[^\\]]*)\\])?( \\[client (?[^\\]]*)\\])? (?.*)$\n [PARSER]\n Name nginx\n Format regex\n Regex ^(?[^ ]*) (?[^ ]*) (?[^ ]*) \\[(?[^\\]]*)\\] \"(?\\S+)(?: +(?[^\\\"]*?)(?: +\\S*)?)?\" (?[^ ]*) (?[^ ]*)(?: \"(?[^\\\"]*)\" \"(?[^\\\"]*)\")?$\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name json-test\n Format json\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name docker\n Format json\n Time_Key time\n Time_Format %Y-%m-%dT%H:%M:%S.%L\n Time_Keep On\n [PARSER]\n Name syslog\n Format regex\n Regex ^\\[0-9]+)\\>(?[^ ]* {1,2}[^ ]* [^ ]*) (?[^ ]*) (?[a-zA-Z0-9_\\/\\.\\-]*)(?:\\[(?[0-9]+)\\])?(?:[^\\:]*\\:)? *(?.*)$\n Time_Key time\n Time_Format %b %d %H:%M:%S"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"部署文件"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"apiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: fluentbit\n namespace: logging\n labels:\n k8s-app: fluentbit-logging\n version: v1\n kubernetes.io/cluster-service: \"true\"\nspec:\n selector:\n matchLabels:\n k8s-app: fluentbit-logging\n version: v1\n template:\n metadata:\n labels:\n k8s-app: fluentbit-logging\n version: v1\n kubernetes.io/cluster-service: \"true\"\n spec:\n containers:\n - name: fluentbit\n image: fluent/fluent-bit:1.2.1\n env:\n - name: FLUENTD_HOST\n value: \"fluentd\"\n - name: FLUENTD_PORT\n value: \"24224\"\n resources:\n limits:\n memory: 100Mi\n requests:\n cpu: 100m\n memory: 100Mi\n volumeMounts:\n - name: config\n mountPath: /fluent-bit/etc\n - name: varlog\n mountPath: /var/log\n - name: varlibdockercontainers\n mountPath: /var/lib/docker/containers\n readOnly: true\n - name: minikube\n mountPath: /mnt/sda1/var/lib/docker/containers\n readOnly: true\n terminationGracePeriodSeconds: 10\n volumes:\n - name: config\n configMap:\n name: fluentbit-config\n - name: varlog\n hostPath:\n path: /var/log\n - name: varlibdockercontainers\n hostPath:\n path: /var/lib/docker/containers\n - name: minikube\n hostPath:\n path: /mnt/sda1/var/lib/docker/containers\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"5.3 日志聚合器Fluentd"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    Fluentd作为日志聚合器负责数据的聚合来自FluentBit的数据，并将其存储到AWS S3中，需要准备项目配置以及deployment文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"项目配置"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"kind: ConfigMap\napiVersion: v1\nmetadata:\n name: fluentd-config\n namespace: logging\ndata:\n fluent.conf: |-\n \n log_level debug\n \n\n \n @type http\n port 9880\n bind 0.0.0.0\n body_size_limit 32m\n keepalive_timeout 10s\n \n \n @type forward\n port 24224\n bind 0.0.0.0\n tag kube.forward\n \n \n @type s3\npath \"year=%Y/month=%m/day=%d/hour=%H\"\n s3_bucket \n s3_region \n "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"部署文件"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\napiVersion: v1\nkind: Service\nmetadata:\n name: fluentd\n namespace: logging\n labels:\n app: fluentd\nspec:\n ports:\n - name: http\n port: 9880\n protocol: TCP\n - name: forword\n port: 24224\n protocol: TCP\n selector:\n app: fluentd\n type: NodePort\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: fluentd\n namespace: logging\n labels:\n app: fluentd\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: fluentd\n strategy:\n rollingUpdate:\n maxSurge: 25%\n maxUnavailable: 25%\n type: RollingUpdate\n template:\n metadata:\n labels:\n app: fluentd\n spec:\n containers:\n - name: fluentd\n image: ghgt/fluentd-es-s3-rewrite:v1.7.3-alpine-es7-s3-rewrite-1.1\n env:\n - name: FLUENTD_ARGS\n value: --no-supervisor -q\n resources:\n limits:\n memory: 500Mi\n requests:\n cpu: 100m\n memory: 200Mi\n ports:\n - containerPort: 8080\n - containerPort: 24224\n volumeMounts:\n - name: varlog\n mountPath: /var/log\n - name: config-volume\n mountPath: /fluentd/etc\n terminationGracePeriodSeconds: 30\n volumes:\n - name: varlog\n hostPath:\n path: /var/log\n - name: config-volume\n configMap:\n name: fluentd-config"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  到此，编码完成！启动并在集群中运行三个服务，随后检查你的AWS S3存储桶吧！"}]},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    写到最后，相信你已经了解如何使用FluentBit、Fluentd等工具结合Kubernetes实现对大数据的采集和存储流程，笔者从基础原理到方案架构，再到实践过程，通过简单的Sample示例带领大家领悟企业级的项目实现，希望读完文章的你有所收获。在接下来的文章中，笔者将继续带领大家关注云原生领域的知识，敬请期待~"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1a/1a6175a84b764112b853cc7b7ebd3e1f.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}