Skywalking使用Kafka從Agent向服務端傳輸數據

背景

在Skywalking官方部署資料中，支持多種模式模式，例如：混合部署、L1和L2分級部署模式，數據可以通過配置存儲模式，直接寫入ES。但是有一個共同的問題，當客戶端日誌量很大時，爲了提高系統的穩定性，同時減少因直接寫入ES導致系統壓力過大，可以在日誌蒐集到寫入之間，採用Kafka進行數據緩衝，這樣及時後端ES存儲全部宕機，也不至於數據丟失。

本文采用技術棧如下：

• Skywalking 8.5.0
• Kafka 2.1.0
• ES 6.1.0

配置部署

agent端部署

如何開啓Agent端數據直接寫入Kafka，官方地址：https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/how-to-enable-kafka-reporter/

The Kafka reporter plugin support report traces, JVM metrics, Instance Properties, and profiled snapshots to Kafka cluster, which is disabled in default. Move the jar of the plugin, kafka-reporter-plugin-x.y.z.jar, from agent/optional-reporter-plugins to agent/plugins for activating.

Notice, currently, the agent still needs to configure GRPC receiver for delivering the task of profiling. In other words, the following configure cannot be omitted.

# Backend service addresses.
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800}

# Kafka producer configuration
plugin.kafka.bootstrap_servers=${SW_KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
plugin.kafka.producer_config[delivery.timeout.ms]=12000
plugin.kafka.get_topic_timeout=${SW_GET_TOPIC_TIMEOUT:10}

Kafka reporter plugin support to customize all configurations of listed in here.

Before you activated the Kafka reporter, you have to make sure that Kafka fetcher has been opened in service.

其中plugin.kafka.bootstrap_servers默認是關閉狀態，需要將agent.properties文件中的 # 註釋去掉。

plugin.kafka.producer_config[delivery.timeout.ms] 此參數表示上報數據超時時間（毫秒），可以根據實際需要自行修改

plugin.kafka.get_topic_timeout 此參數表示獲取TOPIC超時時間，酌情考慮

異常處理

因我本地kafka服務端配置較差，在啓動agent端服務時，在 ~/logs/skywalking-api.log 日誌中出現報錯：

ERROR 2021-06-21 21:58:39:541 SkywalkingAgent-6-JVMService-consume-0 JVMService : JVMService consumes and upload failure. 
java.lang.NullPointerException
	at org.apache.skywalking.apm.agent.core.kafka.KafkaJVMMetricsSender.run(KafkaJVMMetricsSender.java:69)
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

可以直接把agent.properties中如下配置修改閾值，或者直接去掉（有風險）

plugin.kafka.producer_config[delivery.timeout.ms]=12000
plugin.kafka.get_topic_timeout=${SW_GET_TOPIC_TIMEOUT:10}

說明: agent的參數，如果不想直接在agent.properties中配置，也可以在應用啓動時，動態傳入，例如此處啓動應用時示例：

-javaagent:D:/gongstring/dev/skywalking/agent/skywalking-agent.jar -DSW_AGENT_NAMESPACE=gongstring -DSW_AGENT_NAME=dubbo.consumer -Dplugin.dubbo.collect_provider_arguments=true -Dplugin.dubbo.collect_consumer_arguments=true

OAP Server端配置

服務端需要單獨開啓Kafka-Fetcher，文檔地址：https://skywalking.apache.org/docs/main/v8.6.0/en/setup/backend/backend-fetcher/#kafka-fetcher

修改${SW_KAFKA_FETCHER:-} 修改配置，例如本文中配置分組使用default，可以直接修改如下配置：

vim ./config/application.yml

kafka-fetcher:
  selector: ${SW_KAFKA_FETCHER:default}
  default:
    bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:192.168.0.4:9093,192.168.0.4:9094,192.168.0.4:9095}
    partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
    replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
    enableMeterSystem: ${SW_KAFKA_FETCHER_ENABLE_METER_SYSTEM:false}
    enableLog: ${SW_KAFKA_FETCHER_ENABLE_LOG:false}
    isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:false}
    consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:""}
    kafkaHandlerThreadPoolSize: ${SW_KAFKA_HANDLER_THREAD_POOL_SIZE:-1}
    kafkaHandlerThreadPoolQueueSize: ${SW_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:-1}

備註:SW_KAFKA_FETCHER_SERVERS 也可以採用環境變量等多種模式進行配置

修改完服務端後，需要重啓服務端程序方可生效。啓動的時候，會自動檢查kafka中topic是否存在，如果不存在會自動創建。當然如果此處需要自定義topic（例如關閉了創建topic權限），需要提前手工創建topic信息。