微服務全鏈路跟蹤:jaeger集成istio,併兼容uber-trace-id與b3
公司有自己的一套基於k8s的paas系統,並且集成了istio,這裏主要是想講解下springcloud服務如何集成istio
jaeger跨進程傳遞
在基於HTTP協議的分佈式調用中,通常會使用HTTP Header來傳遞SpanContext的內容。
常見的Wire Protocol包含Zipkin使用的b3 HTTP header,Jaeger使用的uber-trace-id HTTP Header,LightStep使用的"x-ot-span-context" 等。
Istio1.0支持b3 header和x-ot-span-context header,可以和Zipkin,Jaeger及LightStep對接;istio1.4以上支持uber-trace-id。
注:請參考github官方說明:https://github.com/istio/istio/issues/12400
uber-trace-id
圖中可以看到其中traceId、spanId等字段都拼接到一個key中了
b3
istio的b3頭詳情可以參考:https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-request-id
b3並沒有將字段都拼接,二十單個傳遞,下面是案例:
X-B3-TraceId:427fde2dc7edb084
X-B3-ParentSpanId:427fde2dc7edb084
X-B3-SpanId:827e270489aafbd7
X-B3-Sampled:1
變更jaeger傳輸爲b3
如果需要集成istio的jaeger,則需要將傳輸方式修改爲b3
微服務全鏈路跟蹤:springcloud集成jaeger該章中已經描述瞭如何集成jaeger,這裏只需要修改一個配置enable-b3-propagation,如下
opentracing:
jaeger:
enable-b3-propagation: true// 默認爲false
udp-sender:
host: localhost
port: 6831
remote-reporter:
flush-interval: 1000
max-queue-size: 5000
log-spans: true
probabilistic-sampler:
sampling-rate: 1
這樣springboot服務就可以與istio中的jaeger信息串起來形成完整全鏈路。
兼容uber-trace-id與b3
現在遇到了另外一個問題,公司已經很早就搭建了一套全鏈路jaeger,並且已經接入了大部分系統,採用的是默認的header傳輸,即:uber-trace-id
而下游有很多paas內部系統都是非java的不方便接入jaeger,只是注入了istio,並自動注入了jaeger-agent,這裏使用的是b3頭傳輸,這就導致了部分鏈路上下游無法串聯起來。而如果需要統一傳輸方式暫不現實,首先如果都改成b3,則需要上游很多已接入的系統修改配置爲b3,如果是都改成uber-trace-id,istio當前版本不支持,如果需要升級istio,則需要升級kubernetes,風險比較大,所以這裏根據實際情況先採用集成兩種頭,即上游都是uber-trace-id,到中間層服務時手動注入b3相關頭如下。
grpc注入b3頭
這裏需要使用grpc的攔截器
import com.google.common.collect.ImmutableMap;
import io.grpc.*;
import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.propagation.Format;
import io.opentracing.propagation.TextMap;
import lombok.extern.slf4j.Slf4j;
import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.Set;
import java.util.concurrent.TimeUnit;
/**
* An intercepter that applies tracing via OpenTracing to all client requests.
*/
@Slf4j
public class ClientTracingInterceptor implements ClientInterceptor {
private final Tracer tracer;
private final OperationNameConstructor operationNameConstructor;
private final boolean streaming;
private final boolean verbose;
private final Set<ClientRequestAttribute> tracedAttributes;
private final ActiveSpanSource activeSpanSource;
private final Metadata.Key<String> b3TraceIdKey = Metadata.Key.of("X-B3-TraceId", Metadata.ASCII_STRING_MARSHALLER);
private final Metadata.Key<String> b3SpanIdKey = Metadata.Key.of("X-B3-SpanId", Metadata.ASCII_STRING_MARSHALLER);
private final Metadata.Key<String> b3ParentSpanIdKey = Metadata.Key.of("X-B3-ParentSpanId", Metadata.ASCII_STRING_MARSHALLER);
private final Metadata.Key<String> b3SampledKey = Metadata.Key.of("X-B3-Sampled", Metadata.ASCII_STRING_MARSHALLER);
/**
* @param
*/
public ClientTracingInterceptor(Tracer tracer) {
this.tracer=tracer;
this.operationNameConstructor = OperationNameConstructor.DEFAULT;
this.streaming = false;
this.verbose = false;
this.tracedAttributes = new HashSet<ClientRequestAttribute>();
this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT;
}
private ClientTracingInterceptor(Tracer tracer, OperationNameConstructor operationNameConstructor, boolean streaming,
boolean verbose, Set<ClientRequestAttribute> tracedAttributes, ActiveSpanSource activeSpanSource) {
this.tracer = tracer;
this.operationNameConstructor = operationNameConstructor;
this.streaming = streaming;
this.verbose = verbose;
this.tracedAttributes = tracedAttributes;
this.activeSpanSource = activeSpanSource;
}
/**
* Use this intercepter to trace all requests made by this client channel.
* @param channel to be traced
* @return intercepted channel
*/
public Channel intercept(Channel channel) {
return ClientInterceptors.intercept(channel, this);
}
@Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
MethodDescriptor<ReqT, RespT> method,
CallOptions callOptions,
Channel next
) {
final String operationName = operationNameConstructor.constructOperationName(method);
Span activeSpan = this.activeSpanSource.getActiveSpan();
final Span span = createSpanFromParent(activeSpan, operationName);
for (ClientRequestAttribute attr : this.tracedAttributes) {
switch (attr) {
case ALL_CALL_OPTIONS:
span.setTag("grpc.call_options", callOptions.toString());
break;
case AUTHORITY:
if (callOptions.getAuthority() == null) {
span.setTag("grpc.authority", "null");
} else {
span.setTag("grpc.authority", callOptions.getAuthority());
}
break;
case COMPRESSOR:
if (callOptions.getCompressor() == null) {
span.setTag("grpc.compressor", "null");
} else {
span.setTag("grpc.compressor", callOptions.getCompressor());
}
break;
case DEADLINE:
if (callOptions.getDeadline() == null) {
span.setTag("grpc.deadline_millis", "null");
} else {
span.setTag("grpc.deadline_millis", callOptions.getDeadline().timeRemaining(TimeUnit.MILLISECONDS));
}
break;
case METHOD_NAME:
span.setTag("grpc.method_name", method.getFullMethodName());
break;
case METHOD_TYPE:
if (method.getType() == null) {
span.setTag("grpc.method_type", "null");
} else {
span.setTag("grpc.method_type", method.getType().toString());
}
break;
case HEADERS:
break;
}
}
return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {
@Override
public void start(Listener<RespT> responseListener, Metadata headers) {
if (verbose) {
span.log("Started call");
}
if (tracedAttributes.contains(ClientRequestAttribute.HEADERS)) {
span.setTag("grpc.headers", headers.toString());
}
tracer.inject(span.context(), Format.Builtin.HTTP_HEADERS, new TextMap() {
@Override
public void put(String key, String value) {
log.info("jaeger key:{},value:{}",key,value);
Metadata.Key<String> headerKey = Metadata.Key.of(key, Metadata.ASCII_STRING_MARSHALLER);
headers.put(headerKey, value);
String[] mm=value.split("%3A");
if("uber-trace-id".equals(key)&&mm.length==4){
headers.put(b3TraceIdKey,mm[0]);
log.info("jaeger traceId:{}",mm[0]);
headers.put(b3SpanIdKey,mm[1]);
headers.put(b3ParentSpanIdKey,mm[2]);
headers.put(b3SampledKey,mm[3]);
}
}
@Override
public Iterator<Entry<String, String>> iterator() {
throw new UnsupportedOperationException(
"TextMapInjectAdapter should only be used with Tracer.inject()");
}
});
Listener<RespT> tracingResponseListener = new ForwardingClientCallListener
.SimpleForwardingClientCallListener<RespT>(responseListener) {
@Override
public void onHeaders(Metadata headers) {
if (verbose) { span.log(ImmutableMap.of("Response headers received", headers.toString())); }
delegate().onHeaders(headers);
}
@Override
public void onMessage(RespT message) {
if (streaming || verbose) { span.log("Response received"); }
delegate().onMessage(message);
}
@Override
public void onClose(Status status, Metadata trailers) {
if (verbose) {
if (status.getCode().value() == 0) { span.log("Call closed"); }
else { span.log(ImmutableMap.of("Call failed", status.getDescription())); }
}
span.finish();
delegate().onClose(status, trailers);
}
};
delegate().start(tracingResponseListener, headers);
}
@Override
public void cancel(@Nullable String message, @Nullable Throwable cause) {
String errorMessage;
if (message == null) {
errorMessage = "Error";
} else {
errorMessage = message;
}
if (cause == null) {
span.log(errorMessage);
} else {
span.log(ImmutableMap.of(errorMessage, cause.getMessage()));
}
delegate().cancel(message, cause);
}
@Override
public void halfClose() {
if (streaming) { span.log("Finished sending messages"); }
delegate().halfClose();
}
@Override
public void sendMessage(ReqT message) {
if (streaming || verbose) { span.log("Message sent"); }
delegate().sendMessage(message);
}
};
}
private Span createSpanFromParent(Span parentSpan, String operationName) {
if (parentSpan == null) {
return tracer.buildSpan(operationName).startManual();
} else {
return tracer.buildSpan(operationName).asChildOf(parentSpan).startManual();
}
}
/**
* Builds the configuration of a ClientTracingInterceptor.
*/
public static class Builder {
private Tracer tracer;
private OperationNameConstructor operationNameConstructor;
private boolean streaming;
private boolean verbose;
private Set<ClientRequestAttribute> tracedAttributes;
private ActiveSpanSource activeSpanSource;
/**
* @param tracer to use for this intercepter
* Creates a Builder with default configuration
*/
public Builder(Tracer tracer) {
this.tracer = tracer;
this.operationNameConstructor = OperationNameConstructor.DEFAULT;
this.streaming = false;
this.verbose = false;
this.tracedAttributes = new HashSet<ClientRequestAttribute>();
this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT;
}
/**
* @param operationNameConstructor to name all spans created by this intercepter
* @return this Builder with configured operation name
*/
public Builder withOperationName(OperationNameConstructor operationNameConstructor) {
this.operationNameConstructor = operationNameConstructor;
return this;
}
/**
* Logs streaming events to client spans.
* @return this Builder configured to log streaming events
*/
public Builder withStreaming() {
this.streaming = true;
return this;
}
/**
* @param tracedAttributes to set as tags on client spans
* created by this intercepter
* @return this Builder configured to trace attributes
*/
public Builder withTracedAttributes(ClientRequestAttribute... tracedAttributes) {
this.tracedAttributes = new HashSet<ClientRequestAttribute>(
Arrays.asList(tracedAttributes));
return this;
}
/**
* Logs all request life-cycle events to client spans.
* @return this Builder configured to be verbose
*/
public Builder withVerbosity() {
this.verbose = true;
return this;
}
/**
* @param activeSpanSource that provides a method of getting the
* active span before the client call
* @return this Builder configured to start client span as children
* of the span returned by activeSpanSource.getActiveSpan()
*/
public Builder withActiveSpanSource(ActiveSpanSource activeSpanSource) {
this.activeSpanSource = activeSpanSource;
return this;
}
/**
* @return a ClientTracingInterceptor with this Builder's configuration
*/
public ClientTracingInterceptor build() {
return new ClientTracingInterceptor(this.tracer, this.operationNameConstructor,
this.streaming, this.verbose, this.tracedAttributes, this.activeSpanSource);
}
}
public enum ClientRequestAttribute {
METHOD_TYPE,
METHOD_NAME,
DEADLINE,
COMPRESSOR,
AUTHORITY,
ALL_CALL_OPTIONS,
HEADERS
}
}
主要改動點在header那一塊
feign注入b3頭
@Configuration
public class FeignConfig implements RequestInterceptor {
@Autowired
private final Tracer tracer;
@Override
public void apply(RequestTemplate requestTemplate) {
ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
if(attributes!=null&&attributes.getRequest()!=null){
HttpServletRequest request = attributes.getRequest();
JaegerSpanContext context=(JaegerSpanContext) tracer.activeSpan().context();
requestTemplate.header("X-B3-TraceId",String.valueOf(context.getTraceId()));
requestTemplate.header("X-B3-SpanId", String.valueOf(context.getSpanId()));
requestTemplate.header("X-B3-ParentSpanId", String.valueOf(context.getParentId()));
requestTemplate.header("X-B3-Sampled", context.isSampled()?"1":"0");
}
}
@Bean
Logger.Level feignLoggerLevel() {
return Logger.Level.FULL;
}
}
經過手動注入後,就可以實現上下游串起來,暫時達到目標,後面的方案是統一傳輸方式,慢慢升級。