SpringCloud升級之路2020.0.x版-45. 實現公共日誌記錄

本系列代碼地址：https://github.com/JoJoTec/spring-cloud-parent

我們這一節在前面實現的帶有鏈路信息的 Publisher 的工廠的基礎上，實現公共日誌記錄的 GlobalFilter。回顧下我們的需求：

我們需要在網關記錄每個請求的：

HTTP 相關元素：
- URL 相關信息
- 請求信息，例如 HTTP HEADER，請求時間等等
- 某些類型的請求體
- 響應信息，例如響應碼
- 某些類型響應的響應體
鏈路信息

記錄請求與響應的 Body 需要注意的地方

前面的章節我們提到過，對於請求與響應的 body 處理，如果用其結果放入主鏈路的話，會造成 Spring Cloud Sleuth 的鏈路信息丟失。還有兩個要注意的地方是：

TCP 粘包拆包導致一個請求體分割成好幾份或者一個包包含幾個請求
讀取後要釋放原本的請求 body 讀取出來的 DataBuffer

爲何要釋放原本的請求 body 讀取出來的 DataBuffer？因爲讀取出來後佔用的 DataBuffer 如果手動不釋放那麼底層的計數一直不歸零會造成內存泄漏。可以參考框架代碼看出，這裏的 DataBuffer 是需要手動釋放的，參考源碼：

ByteBufferDecoder.java

@Override
public ByteBuffer decode(DataBuffer dataBuffer, ResolvableType elementType,
		@Nullable MimeType mimeType, @Nullable Map<String, Object> hints) {

	int byteCount = dataBuffer.readableByteCount();
	ByteBuffer copy = ByteBuffer.allocate(byteCount);
	copy.put(dataBuffer.asByteBuffer());
	copy.flip();
	DataBufferUtils.release(dataBuffer);
	if (logger.isDebugEnabled()) {
		logger.debug(Hints.getLogPrefix(hints) + "Read " + byteCount + " bytes");
	}
	return copy;
}

我們是想把可以輸出到日誌的 body 轉換成字符串進行輸出，爲了代碼簡潔防止出錯，我們使用一個工具類來完成將 DataBuffer 讀取成字符串並釋放的操作：

package com.github.jojotech.spring.cloud.apigateway.common;

import com.google.common.base.Charsets;

import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferUtils;

public class BufferUtil {
	public static String dataBufferToString(DataBuffer dataBuffer) {
		byte[] content = new byte[dataBuffer.readableByteCount()];
		dataBuffer.read(content);
		DataBufferUtils.release(dataBuffer);
		return new String(content, Charsets.UTF_8);
	}
}

編寫實現公共日誌記錄 GlobalFilter

前面鋪墊了那麼多，我們終於可以着手開始寫這個日誌 GlobalFilter 了：

package com.github.jojotech.spring.cloud.apigateway.filter;

import java.net.URI;
import java.util.Set;

import com.alibaba.fastjson.JSON;
import com.github.jojotech.spring.cloud.apigateway.common.BufferUtil;
import com.github.jojotech.spring.cloud.apigateway.common.TracedPublisherFactory;
import lombok.extern.log4j.Log4j2;
import org.reactivestreams.Publisher;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.GlobalFilter;
import org.springframework.core.Ordered;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferFactory;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.server.reactive.ServerHttpRequest;
import org.springframework.http.server.reactive.ServerHttpRequestDecorator;
import org.springframework.http.server.reactive.ServerHttpResponse;
import org.springframework.http.server.reactive.ServerHttpResponseDecorator;
import org.springframework.stereotype.Component;
import org.springframework.web.server.ServerWebExchange;

@Log4j2
@Component
public class CommonLogFilter implements GlobalFilter, Ordered {
	//可以輸出的 body 格式
	public static final Set<MediaType> legalLogMediaTypes = Set.of(
			MediaType.TEXT_XML,
			MediaType.TEXT_PLAIN,
			MediaType.APPLICATION_XML,
			MediaType.APPLICATION_JSON
	);

	@Autowired
	private TracedPublisherFactory tracedPublisherFactory;

	@Override
	public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
		long startTime = System.currentTimeMillis();
		ServerHttpRequest request = exchange.getRequest();
		ServerHttpResponse response = exchange.getResponse();
		//獲取用於拆包處理聚合讀取請求和響應 body 的 buffer 的 factory
		DataBufferFactory dataBufferFactory = response.bufferFactory();
		//請求 http 頭
		HttpHeaders requestHeaders = request.getHeaders();
		//請求 body 類型
		MediaType requestContentType = requestHeaders.getContentType();
		//請求 uri
		String uri = request.getURI().toString();
		//請求 http 方法
		HttpMethod method = request.getMethod();
		log.info("{} -> {}: header: {}", method, uri, JSON.toJSONString(requestHeaders));
		Flux<DataBuffer> dataBufferFlux = tracedPublisherFactory.getTracedFlux(request.getBody(), exchange)
				//使用 buffer 在這裏將所有 body 讀取完避免拆包影響
				.buffer()
				.map(dataBuffers -> {
					//將所有 buffer 粘合在一起
					DataBuffer dataBuffer = dataBufferFactory.join(dataBuffers);
					//只有在 debug 開啓的時候，纔會輸出 body
					if (log.isDebugEnabled()) {
						//只有特定的 body 類型纔會輸出具體的
						if (legalLogMediaTypes.contains(requestContentType)) {
							try {
								//將 body 轉化爲 String 進行輸出，同時注意，原始的 buffer 需要被釋放，因爲 body 流已經被讀取出來，但是沒有地方回收
								//參考
								String s = BufferUtil.dataBufferToString(dataBuffer);
								log.debug("body: {}", s);
								dataBuffer = dataBufferFactory.wrap(s.getBytes());
							}
							catch (Exception e) {
								log.error("error read request body: {}", e.getMessage(), e);
							}
						}
						else {
							log.debug("body: {}", request);
						}
					}
					return dataBuffer;
				});
		return chain.filter(exchange.mutate().request(new ServerHttpRequestDecorator(request) {
			@Override
			public Flux<DataBuffer> getBody() {
				return dataBufferFlux;
			}
		}).response(new ServerHttpResponseDecorator(response) {
			@Override
			public Mono<Void> writeWith(Publisher<? extends DataBuffer> body) {
				HttpHeaders responseHeaders = super.getHeaders();
				//調用這裏的是寫響應回客戶端的 HttpClientConnect 的回寫，已經跳出了 Spring Cloud Sleuth 的鏈路 Span，所以沒有鏈路追蹤信息
				//但是我們在 CommonTraceFilter 我們將鏈路信息放入了響應 Header 中，所以這裏我們就不用手動增加鏈路信息了
				log.info("response: {} -> {} {} header: {}, time: {}ms", method, uri, getStatusCode(), JSON.toJSONString(responseHeaders), System.currentTimeMillis() - startTime);
				final MediaType contentType = responseHeaders.getContentType();
				if (contentType != null && body instanceof Flux && legalLogMediaTypes.contains(contentType) && log.isDebugEnabled()) {
					//有TCP粘包拆包問題，這個body是多次寫入的，一次調用拿不到完整的body，所以這裏轉換成fluxBody利用其中的buffer來接受完整的body
					Flux<? extends DataBuffer> fluxBody = tracedPublisherFactory.getTracedFlux(Flux.from(body), exchange);
					return super.writeWith(fluxBody.buffer().map(buffers -> {
						DataBuffer buffer = dataBufferFactory.join(buffers);
						try {
							String s = BufferUtil.dataBufferToString(buffer);
							log.debug("response: body: {}", s);
							return dataBufferFactory.wrap(s.getBytes());
						} catch (Exception e) {
							log.error("error read response body: {}", e.getMessage(), e);
						}
						return buffer;
					}));
				}
				// if body is not a flux. never got there.
				return super.writeWith(body);
			}
		}).build());
	}

	@Override
	public int getOrder() {
		//指定順序，在 CommonTraceFilter（這個Filter是讀取鏈路信息，最好在所有 Filter 之前） 之後
		return new CommonTraceFilter().getOrder() + 1;
	}
}

需要注意的點都在註釋當中明確標出了，請大家參考。

查看日誌

我們通過加入下面的日誌配置，打開 body 的日誌，這樣日誌就全了：

<AsyncLogger name="com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter" level="debug" additivity="false" includeLocation="true">
    <appender-ref ref="console" />
</AsyncLogger>

發送一個 POST 帶 body 的請求，從日誌中就能看到：

2021-11-29 14:08:42,231  INFO [sports,8481ce2786b686fa,8481ce2786b686fa] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter:59]:POST -> http://127.0.0.1:8181/test-ss/anything?test=1: header: {"Content-Type":["text/plain"],"User-Agent":["PostmanRuntime/7.28.4"],"Accept":["*/*"],"Postman-Token":["666b17c9-0789-46e6-b515-9a4538803308"],"Host":["127.0.0.1:8181"],"Accept-Encoding":["gzip, deflate, br"],"Connection":["keep-alive"],"content-length":["8"]}
2021-11-29 14:08:42,233 DEBUG [sports,8481ce2786b686fa,8481ce2786b686fa] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter:74]:body: ifasdasd
2021-11-29 14:08:42,463  INFO [sports,,] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter$1:96]:response: POST -> http://127.0.0.1:8181/test-ss/anything?test=1 200 OK header: {"traceId":["8481ce2786b686fa"],"spanId":["8481ce2786b686fa"],"Date":["Mon, 29 Nov 2021 14:08:43 GMT"],"Content-Type":["application/json"],"Server":["gunicorn/19.9.0"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Credentials":["true"],"content-length":["886"]}, time: 232ms
2021-11-29 14:08:42,466 DEBUG [sports,8481ce2786b686fa,8481ce2786b686fa] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter$1:105]:response: body: {
  "args": {
    "test": "1"
  }, 
  "data": "ifasdasd", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Content-Length": "8", 
    "Content-Type": "text/plain", 
    "Forwarded": "proto=http;host=\"127.0.0.1:8181\";for=\"127.0.0.1:57526\"", 
    "Host": "httpbin.org", 
    "Postman-Token": "666b17c9-0789-46e6-b515-9a4538803308", 
    "User-Agent": "PostmanRuntime/7.28.4", 
    "X-Amzn-Trace-Id": "Root=1-61a4deeb-3d016ff729306d862edcca0b", 
    "X-B3-Parentspanid": "8481ce2786b686fa", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "5def545b28a7a842", 
    "X-B3-Traceid": "8481ce2786b686fa", 
    "X-Forwarded-Host": "127.0.0.1:8181", 
    "X-Forwarded-Prefix": "/test-ss"
  }, 
  "json": null, 
  "method": "POST", 
  "origin": "127.0.0.1, 61.244.202.46", 
  "url": "http://127.0.0.1:8181/anything?test=1"
}

2021-11-29 14:08:42,474  INFO [sports,,] [24916] [reactor-http-nio-2][reactor.util.Loggers$Slf4JLogger:269]:8481ce2786b686fa,8481ce2786b686fa -> 127.0.0.1:57526 - - [2021-11-29T14:08:42.230008Z[Etc/GMT]] "POST /test-ss/anything?test=1 HTTP/1.1" 200 886 243 ms

微信搜索“我的編程喵”關注公衆號，每日一刷，輕鬆提升技術，斬獲各種offer：

SpringCloud升級之路2020.0.x版-45. 實現公共日誌記錄

記錄請求與響應的 Body 需要注意的地方

編寫實現公共日誌記錄 GlobalFilter

查看日誌

公司新來一個幹練小夥，把 MyBatis 替換成 MyBatis-Plus，上線後哭暈在廁所。。。

10分鐘本地運行llama3及初體驗

Testin雲測上線華爲Pura 70系列真機測試服務！

手寫協議報文 c語言手法

甲骨文(Oracle)宣佈將以74億美元收購Sun公司

golang 表格

conda安裝使用問題

界面控件DevExpress VCL v24.1預覽 - 支持RAD Studio 12.1、圖表新功能

「Java開發指南」如何利用MyEclipse啓用Spring DSL？（二）

“企業創新新引擎”數據庫專項賦能會，讓雲原生技術普惠千行百業！

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結