背景
在Skywalking官方部署资料中，支持多种模式模式，例如：混合部署、L1和L2分级部署模式，数据可以通过配置存储模式，直接写入ES。但是有一个共同的问题，当客户端日志量很大时，为了提高系统的稳定性，同时减少因直接写入ES导致系统压力过大，可以在日志搜集到写入之间，采用Kafka进行数据缓冲，这样及时后端ES存储全部宕机，也不至于数据丢失。
本文采用技术栈如下：
- Skywalking 8.5.0
- Kafka 2.1.0
- ES 6.1.0
配置部署
agent端部署
如何开启Agent端数据直接写入Kafka，官方地址：https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/how-to-enable-kafka-reporter/
The Kafka reporter plugin support report traces, JVM metrics, Instance Properties, and profiled snapshots to Kafka cluster, which is disabled in default. Move the jar of the plugin, kafka-reporter-plugin-x.y.z.jar, from agent/optional-reporter-plugins to agent/plugins for activating.
Notice, currently, the agent still needs to configure GRPC receiver for delivering the task of profiling. In other words, the following configure cannot be omitted.
# Backend service addresses.
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800}
# Kafka producer configuration
plugin.kafka.bootstrap_servers=${SW_KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
plugin.kafka.producer_config[delivery.timeout.ms]=12000
plugin.kafka.get_topic_timeout=${SW_GET_TOPIC_TIMEOUT:10}
Kafka reporter plugin support to customize all configurations of listed in here.
Before you activated the Kafka reporter, you have to make sure that Kafka fetcher has been opened in service.
其中plugin.kafka.bootstrap_servers默认是关闭状态，需要将agent.properties文件中的 # 注释去掉。
plugin.kafka.producer_config[delivery.timeout.ms] 此参数表示上报数据超时时间（毫秒），可以根据实际需要自行修改
plugin.kafka.get_topic_timeout 此参数表示获取TOPIC超时时间，酌情考虑
异常处理
因我本地kafka服务端配置较差，在启动agent端服务时，在 ~/logs/skywalking-api.log 日志中出现报错：
ERROR 2021-06-21 21:58:39:541 SkywalkingAgent-6-JVMService-consume-0 JVMService : JVMService consumes and upload failure.
java.lang.NullPointerException
at org.apache.skywalking.apm.agent.core.kafka.KafkaJVMMetricsSender.run(KafkaJVMMetricsSender.java:69)
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
可以直接把agent.properties中如下配置修改阈值，或者直接去掉（有风险）
plugin.kafka.producer_config[delivery.timeout.ms]=12000
plugin.kafka.get_topic_timeout=${SW_GET_TOPIC_TIMEOUT:10}
说明: agent的参数，如果不想直接在agent.properties中配置，也可以在应用启动时，动态传入，例如此处启动应用时示例：
-javaagent:D:/gongstring/dev/skywalking/agent/skywalking-agent.jar -DSW_AGENT_NAMESPACE=gongstring -DSW_AGENT_NAME=dubbo.consumer -Dplugin.dubbo.collect_provider_arguments=true -Dplugin.dubbo.collect_consumer_arguments=true
OAP Server端配置
服务端需要单独开启Kafka-Fetcher，文档地址：https://skywalking.apache.org/docs/main/v8.6.0/en/setup/backend/backend-fetcher/#kafka-fetcher
修改${SW_KAFKA_FETCHER:-} 修改配置，例如本文中配置分组使用default，可以直接修改如下配置：
vim ./config/application.yml
kafka-fetcher:
selector: ${SW_KAFKA_FETCHER:default}
default:
bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:192.168.0.4:9093,192.168.0.4:9094,192.168.0.4:9095}
partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
enableMeterSystem: ${SW_KAFKA_FETCHER_ENABLE_METER_SYSTEM:false}
enableLog: ${SW_KAFKA_FETCHER_ENABLE_LOG:false}
isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:false}
consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:""}
kafkaHandlerThreadPoolSize: ${SW_KAFKA_HANDLER_THREAD_POOL_SIZE:-1}
kafkaHandlerThreadPoolQueueSize: ${SW_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:-1}
备注:SW_KAFKA_FETCHER_SERVERS 也可以采用环境变量等多种模式进行配置
修改完服务端后，需要重启服务端程序方可生效。启动的时候，会自动检查kafka中topic是否存在，如果不存在会自动创建。当然如果此处需要自定义topic（例如关闭了创建topic权限），需要提前手工创建topic信息。