記一次排查InputStream不可重複讀問題

背景

筆者在做一個項目，需要把用戶瀏覽頁面信息上報到後端做PV，UV統計展示。之前是在POST請求體中直接明文傳輸用戶瀏覽信息，現在希望用Base64編碼一下，這樣即使用戶打開瀏覽器控制檯的Network也看不出上傳了什麼。（其實這個感覺多此一舉，普通的用戶根本不會查看Network，會查看的人，稍微看一下就知道是Base64編碼）

由於升級涉及到安卓、IOS、web端，有的端已經改好要上線，有的端還沒改好，希望可以同時支持明文和編碼後的密文上傳一段時間，等三端都升級後，再去掉明文處理方式。

步驟

1.編寫註解
對RequestBody進行增強，原來的Controller方法如下：

  @CrossOrigin(origins = "*")
  @PostMapping("/page")
  @RequestBodyNeedDecrypt
  public SimpleResponse queryFile(@RequestBody PageView pageView, HttpServletRequest request) {
    LocalDateTime now = LocalDateTime.now();
    pageView.setUploadTime(TimeUtil.getDateTimeStr(now));

    collectorService.save(pageView);
    collectorService.sendMessage(pageTopic, pageView);
    return new SimpleResponse(0);
  }

其中的@RequestBodyNeedDecrypt是筆者自定義的註解，用於標識RequestBody需要解碼的Controller方法，只有被這個註解修飾的方法，纔會對RequestBody進行Base64解密。
註解代碼如下：

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * 標識請求體是否需要解密的註解
 * @create: 2018-07-19 17:37
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface RequestBodyNeedDecrypt {

}

2.編寫自己的增強邏輯

import cn.superid.collector.annotation.RequestBodyNeedDecrypt;
import cn.superid.collector.util.EncryptionUtil;
import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.MethodParameter;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.lang.Nullable;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.servlet.mvc.method.annotation.RequestBodyAdvice;

import java.io.IOException;
import java.io.InputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;

/**
 * 對請求body進行解碼
 *
 * @create: 2018-07-19 16:37
 */
@RestControllerAdvice
public class MyRequestBodyAdvice implements RequestBodyAdvice {


    private final static Logger logger = LoggerFactory.getLogger(MyRequestBodyAdvice.class);

    /**
     * Invoked first to determine if this interceptor applies.
     *
     * @param methodParameter the method parameter
     * @param type            the target type, not necessarily the same as the method
     *                        parameter type, e.g. for {@code HttpEntity<String>}.
     * @param aClass          the selected converter type
     * @return whether this interceptor should be invoked or not
     */
    @Override
    public boolean supports(MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) {
        Annotation[] methodAnnotations = methodParameter.getExecutable().getDeclaredAnnotations();
        if (methodAnnotations == null) {
            return false;
        }

        for (Annotation annotation : methodAnnotations) {
            //只要controller中的方法上有RequestBodyNeedDecrypt註解，就執行beforeBodyRead方法對其requestbody內容進行解碼
            if (annotation.annotationType() == RequestBodyNeedDecrypt.class) {
                return true;
            }
        }

        return false;
    }

    @Override
    public HttpInputMessage beforeBodyRead(HttpInputMessage httpInputMessage, MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) throws IOException {

        try {
            return new MyHttpInputMessage(httpInputMessage);
        } catch (Exception e) {
            e.printStackTrace();
            return httpInputMessage;
        }
    }

    @Override
    public Object afterBodyRead(Object o, HttpInputMessage httpInputMessage, MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) {
        return o;
    }

    @Override
    public Object handleEmptyBody(@Nullable Object o, HttpInputMessage httpInputMessage, MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) {
        return null;
    }


    class MyHttpInputMessage implements HttpInputMessage {
        private HttpHeaders headers;
        private InputStream body;

        public MyHttpInputMessage(HttpInputMessage inputMessage) throws Exception {
            this.headers = inputMessage.getHeaders();
            //在這兒進行base64解碼
            byte[] b = EncryptionUtil.base64Decode(IOUtils.toString(inputMessage.getBody(), "UTF-8"));
            this.body = IOUtils.toInputStream(IOUtils.toString(b, "UTF-8"), "UTF-8");
        }

        @Override
        public InputStream getBody() throws IOException {
            return body;
        }

        @Override
        public HttpHeaders getHeaders() {
            return headers;
        }
    }
}

3.對明文進行兼容
按照上述的方式已經可以解決RequestBody中是base64編碼的內容的解碼處理了。現在需要對明文做一下兼容處理。一開始我以爲加一個if…else…就可以了。
因爲Json字符串開頭都是{“，base64加密後的文字都是”ey”開頭，如：

{"需求":"請把APP背景設置成隨用戶手機殼顏色改變而改變"}

經過base64加密後爲(下面文字可以水平滑動到後面看，有等號)：

eyLpnIDmsYIiOiLor7fmiopBUFDog4zmma/orr7nva7miJDpmo/nlKjmiLfmiYvmnLrlo7PpopzoibLmlLnlj5jogIzmlLnlj5gifQ==

再比如：

{"name":"小二黑","age":28}

編碼後爲：

eyJuYW1lIjoi5bCP5LqM6buRIiwiYWdlIjoyOH0=

於是筆者想，把請求body的內容拿出來看看，是否是”ey”開頭，來判斷是否是base64編碼後的，如果是就解碼，不是就直接放過去。於是把Advice類中的邏輯改成如下：

    @Override
    public HttpInputMessage beforeBodyRead(HttpInputMessage httpInputMessage, MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) throws IOException {

        try {
            String bodyStr = IOUtils.toString(httpInputMessage.getBody(), "UTF-8");

            System.out.println("bodyStr="+bodyStr);
            //目前先兼容明文和base64編碼的
            //base64編碼的
            if (bodyStr.startsWith("ey")) {
                System.out.println("base64編碼的");
                return new MyHttpInputMessage(httpInputMessage);
            } else {//非base64編碼的明文
                System.out.println("明文的");
                return httpInputMessage;
            }


        } catch (Exception e) {
            e.printStackTrace();
            return httpInputMessage;
        }
    }

然而，奇怪的是，不管是明文還是密文，控制檯總會報錯：

{"timestamp":"2018-08-02T10:00:15.801+0000","status":400,"error":"Bad Request","message":"JSON parse error: 
No content to map due to end-of-input; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: 
No content to map due to end-of-input
 at [Source: (PushbackInputStream); line: 1, column: 0]","path":"/collector/page"}

後來發現是

String bodyStr = IOUtils.toString(httpInputMessage.getBody(), "UTF-8");

這行代碼影響到了後續的處理，這個toString方法，追蹤下去，在IOUtils中

    public static long copyLarge(final Reader input, final Writer output, final char[] buffer) throws IOException {
        long count = 0;
        int n;
        while (EOF != (n = input.read(buffer))) {
            output.write(buffer, 0, n);
            count += n;
        }
        return count;
    }

這兒有從包裝了InputStream的Reader input中讀取內容到buffer字符數組中的操作。因爲InputStream讀完之後就不能再讀了，筆者加了幾行輸出：

          System.out.println("before available ="+httpInputMessage.getBody().available());
          System.out.println("markSuported ="+httpInputMessage.getBody().markSupported());
          String bodyStr = IOUtils.toString(httpInputMessage.getBody(), "UTF-8");
          System.out.println("after available ="+httpInputMessage.getBody().available());

輸出的內容如下：

before available =119
markSuported =false
after available =0

可以看到調用了IOUtils.toString方法後，available字節數就變成了0，而且這個輸入流不支持mark，沒法通過reset方法重新再讀一遍流了。InputStream又沒有重寫Object的clone方法，於是只能自己寫一個緩存類。由於這個緩存類別的地方也用不到，筆者就用做增強類的內部類了。總算可以完美的解決流的重複讀了。

最後完整增強類的代碼如下：

import cn.superid.collector.annotation.RequestBodyNeedDecrypt;
import cn.superid.collector.util.EncryptionUtil;
import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.MethodParameter;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.lang.Nullable;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.servlet.mvc.method.annotation.RequestBodyAdvice;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;

/**
 * 對請求body進行解碼
 *
 * @create: 2018-07-19 16:37
 */
@RestControllerAdvice
public class MyRequestBodyAdvice implements RequestBodyAdvice {


    private final static Logger logger = LoggerFactory.getLogger(MyRequestBodyAdvice.class);

    /**
     * Invoked first to determine if this interceptor applies.
     *
     * @param methodParameter the method parameter
     * @param type            the target type, not necessarily the same as the method
     *                        parameter type, e.g. for {@code HttpEntity<String>}.
     * @param aClass          the selected converter type
     * @return whether this interceptor should be invoked or not
     */
    @Override
    public boolean supports(MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) {
        Annotation[] methodAnnotations = methodParameter.getExecutable().getDeclaredAnnotations();
        if (methodAnnotations == null) {
            return false;
        }

        for (Annotation annotation : methodAnnotations) {
            //只要controller中的方法上有RequestBodyNeedDecrypt註解，就執行beforeBodyRead方法對其requestbody內容進行解碼
            if (annotation.annotationType() == RequestBodyNeedDecrypt.class) {
                return true;
            }
        }

        return false;
    }

    @Override
    public HttpInputMessage beforeBodyRead(HttpInputMessage httpInputMessage, MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) throws IOException {

        try {
            return new MyHttpInputMessage(httpInputMessage);
        } catch (Exception e) {
            e.printStackTrace();
            return httpInputMessage;
        }
    }

    @Override
    public Object afterBodyRead(Object o, HttpInputMessage httpInputMessage, MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) {
        return o;
    }

    @Override
    public Object handleEmptyBody(@Nullable Object o, HttpInputMessage httpInputMessage, MethodParameter methodParameter, Type type, Class<? extends HttpMessageConverter<?>> aClass) {
        return null;
    }


    class MyHttpInputMessage implements HttpInputMessage {
        private HttpHeaders headers;

        private InputStream body;

        public MyHttpInputMessage(HttpInputMessage inputMessage) throws Exception {
            this.headers = inputMessage.getHeaders();

            InputStreamHolder holder = new InputStreamHolder(inputMessage.getBody());

            String bodyStr = IOUtils.toString(holder.getInputStream(), "UTF-8");

            //目前先兼容明文和base64編碼的
            //base64編碼的
            if (bodyStr.startsWith("ey")) {
                byte[] b = EncryptionUtil.base64Decode(bodyStr);
                this.body = IOUtils.toInputStream(IOUtils.toString(b, "UTF-8"), "UTF-8");
            } else {//非base64編碼的明文
                this.body = holder.getInputStream();
            }

        }

        @Override
        public InputStream getBody() throws IOException {
            return body;
        }

        @Override
        public HttpHeaders getHeaders() {
            return headers;
        }
    }

    /**
     * 緩存InputStream的容器，可以把輸入流拿出來重複使用
     */
    public class InputStreamHolder {

        private ByteArrayOutputStream byteArrayOutputStream = null;

        public InputStreamHolder(InputStream source) throws IOException {

            byteArrayOutputStream = new ByteArrayOutputStream();
            //輸入流中的字節數組長度
            int length = source.available();
            byte[] holder = new byte[length];
            //把輸入流中的字節數組讀取到holder中，讀取完，輸入流source也就不能再用了
            source.read(holder, 0, length);

            //把holder中的字節寫到byteArrayOutputStream中
            byteArrayOutputStream.write(holder, 0, length);
            byteArrayOutputStream.flush();

        }

        /**
         * 獲取輸入流，可以重複使用的方法
         * @return
         */
        public InputStream getInputStream() {
            return new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
        }
    }
}

感悟

真的只有在踩到坑的時候，才能迅速成長，有時候看似簡單的問題背後可能蘊含着java底層的很多知識，如果不清楚底層實現，很容易遇到自己意料之外的事情。還是要多實踐，才能成長。

記一次排查InputStream不可重複讀問題

背景

步驟

感悟

教你用Perl實現Smgp協議

如何通過前端表格控件在10分鐘內完成一張分組報表？

win11關閉自動檢測病毒刪文件

千兆寬帶實際網速能到達多少？

規避Debezium master purged GTID問題

解決Spark Arrays.toString(Dataset.collect())報錯

IDEA2018.1.6打開Scala文件decompiled.class file bytecode version50(java 6)解決

解決Spark standalone部署模式cores爲0的問題

Kafka Connect 日誌配置

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結