服務追蹤調研

OpenTracing

簡介

OpenTracing 提供了一套平臺無關、廠商無關的 API，這樣不同的組織或者開發人員就能夠更加方便的添加或更換追蹤系統的實現。 OpenTracing API 中的一些概念和術語，在不同的語言環境下都是共享。

接口規範

• Tracer Interface
• Span Interface
• SpanContext Interface
• Global and No-op Tracers

jaeger install

# run jaeger
docker run -d \
  --rm \
  --name jaeger \
  -p6831:6831/udp \
  -p16686:16686 \
  jaegertracing/all-in-one:1.16.0
  
# run demo  
docker run -d \
  --name hotrod\
  --rm \
  --link jaeger \
  --env JAEGER_AGENT_HOST=jaeger \
  --env JAEGER_AGENT_PORT=6831 \
  -p8080-8083:8080-8083 \
  jaegertracing/example-hotrod:1.16.0 \
  all

jaeger 組件簡介

jaeger-agent

jaeger client sdk 產生的數據由 jaeger-agent負責採集併發送到jaeger-collector

Port	Protocol	Function
6831	UDP	accept jaeger.thrift in compact Thrift protocol used by most current Jaeger clients
6832	UDP	accept jaeger.thrift in binary Thrift protocol used by Node.js Jaeger client (because thriftrw npm package does not support compact protocol)
5778	HTTP	serve configs, sampling strategies
5775	UDP	accept zipkin.thrift in compact Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
14271	HTTP	Healthcheck at / and metrics at /metrics

jaeger-collector

jaeger-collector 收集 agent發送過來的數據並分發給後端存儲

Port	Protocol	Function
14267	TChanel	used by jaeger-agent to send spans in jaeger.thrift format
14250	gRPC	used by jaeger-agent to send spans in model.proto format
14268	HTTP	can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411	HTTP	can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
14269	HTTP	Healthcheck at / and metrics at /metrics

Storage Backends

通過環境變量 SPAN_STORAGE_TYPE 來設置後端存儲

• badger和Memory 僅在all-in-one安裝模式下使用
• elasticsearch
• cassandra noSql數據庫
• kafka 僅作爲緩存，將數據分發到es或者Cassandra

Storage plugin

jaeger 支持 gRPC通信的後端存儲

• InfluxDB
• Logz.io

jaeger-ingester

jaeger-ingester可以消費kafka內的數據分發到es或者cassandra

Port	Protocol	Function
14270	HTTP	Healthcheck at / and metrics at /metrics

jaeger-query

該服務是無狀態應用，一般部署在負載均衡（nginx）後面

Port	Protocol	Function
16686	HTTP	/api/* endpoints and Jaeger UI at /
16687	HTTP	Healthcheck at / and metrics at /metrics

spark-dependencies

生產環境下需要部署spark-dependencies 分析調用鏈信息

驗證Jaeger Client自動埋點

java-specialagent

java-specialagent是java應用自動埋點第三方引擎庫，詳細說明請點擊github傳送門

• 目前測試情況是springboot web應用和採用okhttp和Dropwizard框架編譯的java應用都可以實現自動埋點，但是阿里的sofarpc無法自動埋點。

• 添加自動埋點的java應用會延長啓動時間

本地運行 jaeger all-in-one應用

docker run -d --rm \
    --name jaeger \
    -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 14250:14250 \
    -p 9411:9411 \
    jaegertracing/all-in-one:1.16.0

運行 springboot demo

驗證可行，能夠在jaeger ui中查詢到服務追蹤信息

java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO  -DJAEGER_SERVICE_NAME="springboot" -jar springboot-helloworld-1.0-SNAPSHOT.jar --server.port=9000

運行PRC demo

驗證可行，能夠在jaeger ui中查詢到服務追蹤信息

java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO  -DJAEGER_SERVICE_NAME="formatter" -jar jaeger-demo/formatter/target/formatter-1.0-SNAPSHOT.jar

java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO  -DJAEGER_SERVICE_NAME="publisher" -jar jaeger-demo/publisher/target/publisher-1.0-SNAPSHOT.jar

java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO  -DJAEGER_SERVICE_NAME="callserver" -jar jaeger-demo/callserver/target/callserver-1.0-SNAPSHOT.jar

運行sofarpc demo

驗證失敗，在jaeger ui中沒有查詢到追蹤信息

• rpcclient 第一次發起請求會報錯,後續請求可正常收到響應
• 報錯信息如下

2020-03-03 14:45:50.730  INFO 34856 --- [           main] com.citicbank.RpcclientApplication       : Started RpcclientApplication in 7.443 seconds (JVM running for 19.898)
com.alipay.sofa.rpc.context.RpcInvokeContext@5013c6bc{timeout=null, targetURL='null', targetGroup='null', responseCallback=null, future=null, map={}, requestBaggage={service=mosn}, responseBaggage={}}
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: com.alipay.sofa.rpc.core.exception.SofaTimeOutException: com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:12200
	at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.convertToRpcException(BoltClientTransport.java:327)
	at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.syncSend(BoltClientTransport.java:251)
	at com.alipay.sofa.rpc.client.AbstractCluster.doSendMsg(AbstractCluster.java:535)
	at com.alipay.sofa.rpc.client.AbstractCluster.sendMsg(AbstractCluster.java:506)
	at com.alipay.sofa.rpc.filter.ConsumerInvoker.invoke(ConsumerInvoker.java:60)
	at com.alipay.sofa.rpc.filter.sofatracer.ConsumerTracerFilter.invoke(ConsumerTracerFilter.java:66)
	at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98)
	at com.alipay.sofa.rpc.filter.RpcReferenceContextFilter.invoke(RpcReferenceContextFilter.java:80)
	at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98)
	at com.alipay.sofa.rpc.filter.ConsumerExceptionFilter.invoke(ConsumerExceptionFilter.java:37)
	at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98)
	at com.alipay.sofa.rpc.filter.FilterChain.invoke(FilterChain.java:262)
	at com.alipay.sofa.rpc.client.AbstractCluster.filterChain(AbstractCluster.java:499)
	at com.alipay.sofa.rpc.client.FailoverCluster.doInvoke(FailoverCluster.java:67)
	at com.alipay.sofa.rpc.client.AbstractCluster.invoke(AbstractCluster.java:288)
	at com.alipay.sofa.rpc.client.ClientProxyInvoker.invoke(ClientProxyInvoker.java:83)
	at com.citicbank.service.RpcDemoService_proxy_0.sayHello(RpcDemoService_proxy_0.java)
	at com.citicbank.service.RpcDemoClientImpl.sayClientAnnotation(RpcDemoClientImpl.java:19)
	at com.citicbank.RpcclientApplication.main(RpcclientApplication.java:22)
	... 8 more
Caused by: com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:12200
	at com.alipay.remoting.rpc.RpcResponseResolver.preProcess(RpcResponseResolver.java:83)
	at com.alipay.remoting.rpc.RpcResponseResolver.resolveResponseObject(RpcResponseResolver.java:54)
	at com.alipay.remoting.rpc.RpcRemoting.invokeSync(RpcRemoting.java:186)
	at com.alipay.remoting.rpc.RpcClientRemoting.invokeSync(RpcClientRemoting.java:64)
	at com.alipay.remoting.rpc.RpcClient.invokeSync(RpcClient.java:355)
	at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.doInvokeSync(BoltClientTransport.java:275)
	at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.syncSend(BoltClientTransport.java:248)
	... 25 more
Unknown instrumentable state for thread: DestroyJavaVM

運行命令

java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO  -DJAEGER_SERVICE_NAME="client" -jar rpcclient-0.0.1-SNAPSHOT.jar

java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO  -DJAEGER_SERVICE_NAME="server" -jar rpcserver-0.0.1-SNAPSHOT.jar

java-specialagent 支持的插件

OpenTracing Plugin (link to plugin implementation)	SpecialAgent Rule (link to rule implementation)	Min Version (min supported)	Max Version (max supported)
Akka Actor	[akka:actor][akka-actor]	2.5.0	2.6.1
Akka Http	[akka:http][akka-http]	10.1.0	10.1.10
Apache Camel	[camel][camel]	2.24.0	2.24.2
Apache HttpClient	[apache:httpclient][apache-httpclient]	4.4	4.5.9
Async Http Client	[asynchttpclient][asynchttpclient]	2.7.0	2.10.4
AWS SDK	[aws:sdk:1][aws-sdk-1]	1.11.79	1.11.5.1
	[aws:sdk:2][aws-sdk-2]	2.1.4	2.7.15
Cassandra Driver	[cassandra:driver:3][cassandra-driver-3]	3.0.0	3.7.2
	[cassandra:driver:4][cassandra-driver-4]	4.0.0	4.2.0
Couchbase Client	[couchbase-client][couchbase-client]	2.7.3	2.7.11
Dynamic	[dynamic (configurable)][dynamic]	*	*
Elasticsearch Client	[elasticsearch:client-transport][elasticsearch-7-transport-client] [elasticsearch:client-rest][elasticsearch-7-rest-client]	6.4.0	7.3.1
Feign	[feign][feign]	9.0.0	10.4.0
Google Http Client	[google-http-client][google-http-client]	1.19.0	1.33.0
Grizzly AsyncHttpClient	[grizzly:ahc][grizzly-ahc]	1.15	1.15
Grizzly HTTP Server	[grizzly:http-server][grizzly-http-server]	2.3.35	2.3.35
GRPC	[grpc][grpc]	1.6.0	1.23.0
Hazelcast	[hazelcast][hazelcast]	3.12	3.12.4
Java Concurrent API [java.util.concurrent]	[concurrent][concurrent]	1.5	11
[Java JDBC API [java.sql]][java-jdbc]	[jdbc (configurable)][jdbc]	3.1	4.3
[Java JMS API [javax.jms]][java-jms]	[jms:1][jms-1]	1.1	1.1
	[jms:2][jms-2]	2.0	2.0a
Java Servlet API [javax.servlet]	[servlet (configurable)][servlet]	2.3	3.1
Jetty		7.6.21.v20160908	9.2.15.v20160210
Tomcat		7.0.65	9.0.27
Java Thread [java.lang.Thread]	[thread][thread]	1.0	11
HttpURLConnection [java.net.HttpURLConnection]	[httpurlconnection][httpurlconnection]	1.1	11
JAX-RS Client	[jax-rs][jaxrs]	2.0	2.1
Jedis Client	[jedis][jedis]	2.7.0	3.1.0
Kafka Client	[kafka:client][kafka-client]	1.1.0	2.3.0
Lettuce Client	[lettuce:5.0][lettuce-5.0]	5.0.0.RELEASE	5.0.5.RELEASE
	[lettuce:5.1][lettuce-5.1]	5.1.0.M1	5.1.8.RELEASE
	[lettuce:5.2][lettuce-5.2]	5.2.0.RELEASE	5.2.0.RELEASE
MongoDB Driver	[mongo:driver][mongo-driver]	3.9.0	3.11.0
Neo4j Driver	[neo4j:driver][neo4j-driver]	4.0.0	4.0.0
Netty	[netty][netty]	4.1.0	4.1.44
[OkHttp][java-okhttp]	[okhttp][okhttp]	3.5.0	4.2.2
Play Framework	[play][play]	2.6.0	2.7.3
Play WS	[play:ws][play-ws]	2.0.0	2.0.7
Pulsar Client	[pulsar:client][pulsar-client]	2.2.0	2.4.2
Pulsar Functions	[pulsar-functions (configurable)][pulsar-functions]	2.2.0	2.4.2
RabbitMQ Client	[rabbitmq:client][rabbitmq-client]	5.0.0	5.7.3
Reactor	[reactor][reactor]	3.2.3.RELEASE	3.2.3.RELEASE
Redisson	[redisson][redisson]	3.11.0	3.11.5
RxJava 2	[rxjava:2][rxjava-2]	2.1.0	2.2.12
Spring JMS	[spring:jms][spring-jms]	5.0.0.RELEASE	5.2.2.RELEASE
Spring Kafka	[spring:kafka][spring-kafka]	2.2.0.RELEASE	2.3.3.RELEASE
Spring Messaging	[spring:messaging][spring-messaging]	5.1.0.RELEASE	5.2.2.RELEASE
Spring RabbitMQ	[spring:rabbitmq][spring-rabbitmq]	2.0.0.RELEASE	2.2.1.RELEASE
Spring WebFlux	[spring:webflux][spring-webflux]	5.1.0.RELEASE	5.2.1.RELEASE
Spring Boot WebSocket STOMP	[spring:websocket][spring-websocket]	2.1.0.RELEASE	2.1.8.RELEASE
Spring [@Async and @Scheduled]	[spring:scheduling][spring-scheduling]	5.0.0.RELEASE	5.2.1.RELEASE
Spring Web	[spring:web:3][spring-web-3]	3.0.3.RELEASE	3.2.18.RELEASE
	[spring:web:4.0][spring-web-4.0]	4.0.0.RELEASE	4.0.9.RELEASE
	[spring:web:4.x][spring-web-4]	4.1.0.RELEASE	4.3.25.RELEASE
	[spring:web:5][spring-web-5]	5.0.0.RELEASE	5.2.1.RELEASE
Spring Web MVC	[spring:webmvc:3][spring-webmvc-3]	3.0.2.RELEASE	3.2.18.RELEASE
	[spring:webmvc:4][spring-webmvc-4]	4.0.0.RELEASE	4.3.25.RELEASE
	[spring:webmvc:5][spring-webmvc-5]	5.0.0.RELEASE	5.2.1.RELEASE
Spymemcached	[spymemcached][spymemcached]	2.11.0	2.12.3
Thrift	[thrift][thrift]	0.10.0	0.12.0
Zuul	[zuul][zuul]	1.0.0	1.3.1

關於skywalking

Skywalking 是一個APM系統，即應用性能監控系統，爲微服務架構和雲原生架構系統設計。完全支持opentracing接口，它通過探針自動收集所需的指標，並進行分佈式追蹤。通過這些調用鏈路以及指標，Skywalking APM會感知應用間關係和服務間關係，並進行相應的指標統計。目前支持鏈路追蹤和監控應用組件如下，基本涵蓋主流框架和容器，包括sofarpc、 dubbo等。詳細列表如下

• HTTP Server
  • Tomcat 7
  • Tomcat 8
  • Tomcat 9
  • Spring Boot Web 4.x
  • Spring MVC 3.x, 4.x 5.x with servlet 3.x
  • Nutz Web Framework 1.x
  • Struts2 MVC 2.3.x -> 2.5.x
  • Resin 3 (Optional¹)
  • Resin 4 (Optional¹)
  • Jetty Server 9
  • Spring WebFlux 5.x
  • Undertow 1.3.0.Final -> 2.0.27.Final
  • RESTEasy 3.1.0.Final -> 3.7.0.Final
  • Play Framework 2.6.x -> 2.8.x
  • Light4J Microservices Framework 1.6.x -> 2.x
  • Netty SocketIO 1.x
• HTTP Client
  • Feign 9.x
  • Netflix Spring Cloud Feign 1.1.x, 1.2.x, 1.3.x
  • Okhttp 3.x
  • Apache httpcomponent HttpClient 2.0 -> 3.1, 4.2, 4.3
  • Spring RestTemplete 4.x
  • Jetty Client 9
  • Apache httpcomponent AsyncClient 4.x
• HTTP Gateway
  • Spring Cloud Gateway 2.1.x.RELEASE (Optional²)
• JDBC
  • Mysql Driver 5.x, 6.x, 8.x
  • Oracle Driver (Optional¹)
  • H2 Driver 1.3.x -> 1.4.x
  • Sharding-JDBC 1.5.x
  • ShardingSphere 3.0.0, 4.0.0-RC1
  • PostgreSQL Driver 8.x, 9.x, 42.x
• RPC Frameworks
  • Dubbo 2.5.4 -> 2.6.0
  • Dubbox 2.8.4
  • Apache Dubbo 2.7.0
  • Motan 0.2.x -> 1.1.0
  • gRPC 1.x
  • Apache ServiceComb Java Chassis 0.1 -> 0.5,1.x
  • SOFARPC 5.4.0
  • Armeria 0.63.0 -> 0.98.0
  • Apache Avro 1.7.0 - 1.8.x
• MQ
  • RocketMQ 4.x
  • Kafka 0.11.0.0 -> 1.0
  • ActiveMQ 5.10.0 -> 5.15.4
  • RabbitMQ 5.x
  • Pulsar 2.2.x -> 2.4.x
• NoSQL
  • Redis
    • Jedis 2.x
    • Redisson Easy Java Redis client 3.5.2+
    • Lettuce 5.x
  • MongoDB Java Driver 2.13-2.14, 3.4.0-3.11.1
  • Memcached Client
    • Spymemcached 2.x
    • Xmemcached 2.x
  • Elasticsearch
    • transport-client 5.2.x-5.6.x
    • rest-high-level-client 6.7.1-6.8.4
    • rest-high-level-client 7.0.0-7.5.2
  • Solr
    • SolrJ 7.x
  • Cassandra 3.x
    • cassandra-java-driver 3.7.0-3.7.2
• Service Discovery
  • Netflix Eureka
• Distributed Coordination
  • Zookeeper 3.4.x (Optional² & Except 3.4.4)
• Spring Ecosystem
  • Spring Bean annotations(@Bean, @Service, @Component, @Repository) 3.x and 4.x (Optional²)
  • Spring Core Async SuccessCallback/FailureCallback/ListenableFutureCallback 4.x
  • Spring Transaction 4.x and 5.x (Optional²)
• Hystrix: Latency and Fault Tolerance for Distributed Systems 1.4.20 -> 1.5.12
• Scheduler
  • Elastic Job 2.x
• OpenTracing community supported
• Canal: Alibaba mysql database binlog incremental subscription & consumer components 1.0.25 -> 1.1.2
• JSON
  • GSON 2.8.x (Optional²)
• Vert.x Ecosystem
  • Vert.x Eventbus 3.2+
  • Vert.x Web 3.x
• Thread Schedule Framework
  • Spring @Async 4.x and 5.x
• Cache
  • Ehcache 2.x
• Kotlin
  • Coroutine 1.0.1 -> 1.3.x (Optional²)

skywalking demo 實例

[北京服務器] http://122.112.182.72:8080/