Java實現AWS S3 V4 Authorization自定義驗證

前言

最近在開發文件存儲服務，需要符合s3的協議標準，可以直接接入aws-sdk，本文針對sdk發出請求的鑑權信息進行重新組合再簽名驗證有效性，sdk版本如下

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>2.20.45</version>
        </dependency>

算法解析

首先對V4版本簽名算法的數據結構及簽名流程進行拆解分析，以請求頭簽名爲示例講解

signature = doSign(waitSignString)

簽名示例

請求頭簽名

AWS4-HMAC-SHA256 Credential=admin/20230530/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=6f50628a101b46264c7783937be0366762683e0d319830b1844643e40b3b0ed

Url簽名

http://localhost:8001/s3/kkk/test.docx?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230531T024715Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=admin%2F20230531%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=038e2ea71073761aa0370215621599649e9228177c332a0a79f784b1a6d9ee39

數據結構

waitSignString = doHex（【第一部分】+【第二部分】+【第三部分】+【第四部分】），每部分使用\n換行符連接，第四部分不要加上換行符

第一部分

Algorithm – 用於創建規範請求的哈希的算法，對於 SHA-256，算法是 AWS4-HMAC-SHA256，則這部分的內容固定爲

"AWS4-HMAC-SHA256" + "\n"

第二部分

RequestDateTime – 在憑證範圍內使用的日期和時間，這個時間爲請求發出的時間，直接從請求頭獲取x-amz-date即可，這部分內容爲

request.getHeader("x-amz-date") + "\n"

第三部分

CredentialScope – 憑證範圍，這會將生成的簽名限制在指定的區域和服務範圍內，該字符串採用以下格式：YYYYMMDD/region/service/aws4_request

這部分由4個內容信息拼接組成

• 請求時間的YYYYMMDD格式
• 存儲區域
• 存儲服務
• 請求頭

這些信息我們都可以從請求頭的Authorization憑證提取出Credential部分進行拆分重新組合

        String[] parts = authorization.trim().split("\\,");
        String credential = parts[0].split("\\=")[1];
        String[] credentials = credential.split("\\/");
        String accessKey = credentials[0];
        if (!accessKeyId.equals(accessKey)) {
            return false;
        }
        String date = credentials[1];
        String region = credentials[2];
        String service = credentials[3];
        String aws4Request = credentials[4];

這部分內容爲

date + "/" + region + "/" + service + "/" + aws4Request + "\n"

第四部分

HashedCanonicalRequest – 規範請求的哈希

這部分內容爲

doHex(canonicalRequest)

canonicalRequest具體拆解又可以6小部分組成，每部分使用\n換行符連接，最後不要加上換行符

<HTTPMethod>\n
<CanonicalURI>\n
<CanonicalQueryString>\n
CanonicalHeaders>\n
<SignedHeaders>\n
<HashedPayload>

• HTTPMethod

  代表請求的HTTP方法，例如GET，POST，DELETE，PUT等，直接從request獲取即可

  這部分內容爲

  String HTTPMethod = request.getMethod() + "\n"
  

• CanonicalURI

  代表請求的路由部分，例如完成請求爲http://localhost:8001/s3/aaaa/ccc.txt，則該部分爲/s3/aaaa/ccc.txt

  需要進行encode操作，我這裏直接獲取則省略了這部分

  這部分內容爲

  String CanonicalURI = request.getRequestURI().split("\\?")[0] + "\n";
  

• CanonicalQueryString

  代表請求參數的拼接成字符串key1=value1&key2=value2這種形式，拼接的key需要按照字母排序

  value需要進行encode操作，我這裏直接獲取則省略了這部分

          String queryString = ConvertOp.convert2String(request.getQueryString());
          if(!StringUtil.isEmpty(queryString)){
              Map<String, String> queryStringMap =  parseQueryParams(queryString);
              List<String> keyList = new ArrayList<>(queryStringMap.keySet());
              Collections.sort(keyList);
              StringBuilder queryStringBuilder = new StringBuilder("");
              for (String key:keyList) {
                  queryStringBuilder.append(key).append("=").append(queryStringMap.get(key)).append("&");
              }
              queryStringBuilder.deleteCharAt(queryStringBuilder.lastIndexOf("&"));
          }
  
      public static Map<String, String> parseQueryParams(String queryString) {
          Map<String, String> queryParams = new HashMap<>();
          try {
              if (queryString != null && !queryString.isEmpty()) {
                  String[] queryParamsArray = queryString.split("\\&");
  
                  for (String param : queryParamsArray) {
                      String[] keyValue = param.split("\\=");
                      if (keyValue.length == 1) {
                          String key = keyValue[0];
                          String value = "";
                          queryParams.put(key, value);
                      }
                      else if (keyValue.length == 2) {
                          String key = keyValue[0];
                          String value = keyValue[1];
                          queryParams.put(key, value);
                      }
                  }
              }
          } catch (Exception e) {
              e.printStackTrace();
          }
          return queryParams;
      }
  

  這部分內容爲

  String CanonicalQueryString = queryStringBuilder.toString() + "\n"
  

• CanonicalHeaders

  代表請求頭拼接成字符串key:value的形式，每個head部分使用\n換行符連接，拼接的key需要按照字母排序

  簽名的請求頭從Authorization解析獲取

          String signedHeader = parts[1].split("\\=")[1];
          String[] signedHeaders = signedHeader.split("\\;");
  

          String headString = "";
          for (String name : signedHeaders) {
              headString += name + ":" + request.getHeader(name) + "\n";
          }
  

  這部分內容爲

  String CanonicalHeaders = headString + "\n"
  

• SignedHeaders

  代表請求頭的key部分，使用;隔開

  這部分內容爲從Authorization解析中獲取

  這部分內容爲

  String SignedHeaders = signedHeader + "\n"
  

• HashedPayload

  代表請求body部分的簽名，直接從requet的head提取x-amz-content-sha256內容

  這部分內容爲

  String HashedPayload = Stringrequest.getHeader("x-amz-content-sha256")
  

doHex

本部分只是一個字符串轉16進制的一個操作

    private String doHex(String data) {
        MessageDigest messageDigest;
        try {
            messageDigest = MessageDigest.getInstance("SHA-256");
            messageDigest.update(data.getBytes("UTF-8"));
            byte[] digest = messageDigest.digest();
            return String.format("%064x", new java.math.BigInteger(1, digest));
        } catch (NoSuchAlgorithmException | UnsupportedEncodingException e) {
            e.printStackTrace();
        }
        return null;
    }

簽名流程

doSign 的流程爲doBytesToHex（doHmacSHA256（signatureKey，waitSignString ））

doBytesToHex爲byte轉16進制操作

    private String doBytesToHex(byte[] bytes) {
        char[] hexChars = new char[bytes.length * 2];
        for (int j = 0; j < bytes.length; j++) {
            int v = bytes[j] & 0xFF;
            hexChars[j * 2] = hexArray[v >>> 4];
            hexChars[j * 2 + 1] = hexArray[v & 0x0F];
        }
        return new String(hexChars).toLowerCase();
    }

doHmacSHA256爲簽名算法

    private byte[] doHmacSHA256(byte[] key, String data) throws Exception {
        String algorithm = "HmacSHA256";
        Mac mac = Mac.getInstance(algorithm);
        mac.init(new SecretKeySpec(key, algorithm));
        return mac.doFinal(data.getBytes("UTF8"));
    }

signatureKey簽名密鑰由secretAccessKey，請求時間，存儲區域，存儲服務，請求頭這5個要素進行疊加簽名生成

        byte[] kSecret = ("AWS4" + secretAccessKey).getBytes("UTF8");
        byte[] kDate = doHmacSHA256(kSecret, date);
        byte[] kRegion = doHmacSHA256(kDate, region);
        byte[] kService = doHmacSHA256(kRegion, service);
        byte[] signatureKey = doHmacSHA256(kService, aws4Request);

將最終生成的再簽名與Authorization中解析出的Signature進行比較，一致則鑑權成功

調試位置

調試過程中需要驗證每部分的簽名是否拼接編碼正確，我們需要和sdk生成的內容進行比對找出問題

調試software.amazon.awssdk.auth.signer.internal包下AbstractAws4Signer類的doSign類，獲取stringToSign與你待簽名字符串比對差異，源碼如下

    protected Builder doSign(SdkHttpFullRequest request, Aws4SignerRequestParams requestParams, T signingParams, ContentChecksum contentChecksum) {
        Builder mutableRequest = request.toBuilder();
        AwsCredentials sanitizedCredentials = this.sanitizeCredentials(signingParams.awsCredentials());
        if (sanitizedCredentials instanceof AwsSessionCredentials) {
            this.addSessionCredentials(mutableRequest, (AwsSessionCredentials)sanitizedCredentials);
        }

        this.addHostHeader(mutableRequest);
        this.addDateHeader(mutableRequest, requestParams.getFormattedRequestSigningDateTime());
        mutableRequest.firstMatchingHeader("x-amz-content-sha256").filter((h) -> {
            return h.equals("required");
        }).ifPresent((h) -> {
            mutableRequest.putHeader("x-amz-content-sha256", contentChecksum.contentHash());
        });
        this.putChecksumHeader(signingParams.checksumParams(), contentChecksum.contentFlexibleChecksum(), mutableRequest, contentChecksum.contentHash());
        AbstractAws4Signer.CanonicalRequest canonicalRequest = this.createCanonicalRequest(request, mutableRequest, contentChecksum.contentHash(), signingParams.doubleUrlEncode(), signingParams.normalizePath());
        String canonicalRequestString = canonicalRequest.string();
        String stringToSign = this.createStringToSign(canonicalRequestString, requestParams);
        byte[] signingKey = this.deriveSigningKey(sanitizedCredentials, requestParams);
        byte[] signature = this.computeSignature(stringToSign, signingKey);
        mutableRequest.putHeader("Authorization", this.buildAuthorizationHeader(signature, sanitizedCredentials, requestParams, canonicalRequest));
        this.processRequestPayload(mutableRequest, signature, signingKey, requestParams, signingParams, contentChecksum.contentFlexibleChecksum());
        return mutableRequest;
    }

代碼示例

通過攔截器進行驗證的過程，完整代碼如下，兼容了普通請求的頭部驗證和文件下載url的簽名驗證

@Component
public class S3Intecept implements HandlerInterceptor {
    @Autowired
    private SystemConfig systemConfig;

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        boolean flag = false;
        String authorization = request.getHeader("Authorization");
        if(!StringUtil.isEmpty(authorization)){
            flag = validAuthorizationHead(request, systemConfig.getUsername(), systemConfig.getPassword());
        }else{
            authorization = request.getParameter("X-Amz-Credential");
            if(!StringUtil.isEmpty(authorization)){
                flag = validAuthorizationUrl(request, systemConfig.getUsername(), systemConfig.getPassword());
            }
        }
        if(!flag){
            response.setStatus(HttpStatus.UNAUTHORIZED.value());
        }
        return flag;
    }

    public boolean validAuthorizationHead(HttpServletRequest request, String accessKeyId, String secretAccessKey) throws Exception {
        String authorization = request.getHeader("Authorization");
        String requestDate = request.getHeader("x-amz-date");
        String contentHash = request.getHeader("x-amz-content-sha256");
        String httpMethod = request.getMethod();
        String uri = request.getRequestURI().split("\\?")[0];
        String queryString = ConvertOp.convert2String(request.getQueryString());
        //示例
        //AWS4-HMAC-SHA256 Credential=admin/20230530/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=6f50628a101b46264c7783937be0366762683e0d319830b1844643e40b3b0ed

        ///region authorization拆分
        String[] parts = authorization.trim().split("\\,");
        //第一部分-憑證範圍
        String credential = parts[0].split("\\=")[1];
        String[] credentials = credential.split("\\/");
        String accessKey = credentials[0];
        if (!accessKeyId.equals(accessKey)) {
            return false;
        }
        String date = credentials[1];
        String region = credentials[2];
        String service = credentials[3];
        String aws4Request = credentials[4];
        //第二部分-簽名頭中包含哪些字段
        String signedHeader = parts[1].split("\\=")[1];
        String[] signedHeaders = signedHeader.split("\\;");
        //第三部分-生成的簽名
        String signature = parts[2].split("\\=")[1];
        ///endregion

        ///region 待簽名字符串
        String stringToSign = "";
        //簽名由4部分組成
        //1-Algorithm – 用於創建規範請求的哈希的算法。對於 SHA-256，算法是 AWS4-HMAC-SHA256。
        stringToSign += "AWS4-HMAC-SHA256" + "\n";
        //2-RequestDateTime – 在憑證範圍內使用的日期和時間。
        stringToSign += requestDate + "\n";
        //3-CredentialScope – 憑證範圍。這會將生成的簽名限制在指定的區域和服務範圍內。該字符串採用以下格式：YYYYMMDD/region/service/aws4_request
        stringToSign += date + "/" + region + "/" + service + "/" + aws4Request + "\n";
        //4-HashedCanonicalRequest – 規範請求的哈希。
        //<HTTPMethod>\n
        //<CanonicalURI>\n
        //<CanonicalQueryString>\n
        //<CanonicalHeaders>\n
        //<SignedHeaders>\n
        //<HashedPayload>
        String hashedCanonicalRequest = "";
        //4.1-HTTP Method
        hashedCanonicalRequest += httpMethod + "\n";
        //4.2-Canonical URI
        hashedCanonicalRequest += uri + "\n";
        //4.3-Canonical Query String
        if(!StringUtil.isEmpty(queryString)){
            Map<String, String> queryStringMap =  parseQueryParams(queryString);
            List<String> keyList = new ArrayList<>(queryStringMap.keySet());
            Collections.sort(keyList);
            StringBuilder queryStringBuilder = new StringBuilder("");
            for (String key:keyList) {
                queryStringBuilder.append(key).append("=").append(queryStringMap.get(key)).append("&");
            }
            queryStringBuilder.deleteCharAt(queryStringBuilder.lastIndexOf("&"));

            hashedCanonicalRequest += queryStringBuilder.toString() + "\n";
        }else{
            hashedCanonicalRequest += queryString + "\n";
        }
        //4.4-Canonical Headers
        for (String name : signedHeaders) {
            hashedCanonicalRequest += name + ":" + request.getHeader(name) + "\n";
        }
        hashedCanonicalRequest += "\n";
        //4.5-Signed Headers
        hashedCanonicalRequest += signedHeader + "\n";
        //4.6-Hashed Payload
        hashedCanonicalRequest += contentHash;
        stringToSign += doHex(hashedCanonicalRequest);
        ///endregion

        ///region 重新生成簽名
        //計算簽名的key
        byte[] kSecret = ("AWS4" + secretAccessKey).getBytes("UTF8");
        byte[] kDate = doHmacSHA256(kSecret, date);
        byte[] kRegion = doHmacSHA256(kDate, region);
        byte[] kService = doHmacSHA256(kRegion, service);
        byte[] signatureKey = doHmacSHA256(kService, aws4Request);
        //計算簽名
        byte[] authSignature = doHmacSHA256(signatureKey, stringToSign);
        //對簽名編碼處理
        String strHexSignature = doBytesToHex(authSignature);
        ///endregion

        if (signature.equals(strHexSignature)) {
            return true;
        }
        return false;
    }

    public boolean validAuthorizationUrl(HttpServletRequest request, String accessKeyId, String secretAccessKey) throws Exception {
        String requestDate = request.getParameter("X-Amz-Date");
        String contentHash = "UNSIGNED-PAYLOAD";
        String httpMethod = request.getMethod();
        String uri = request.getRequestURI().split("\\?")[0];
        String queryString = ConvertOp.convert2String(request.getQueryString());
        //示例
        //"http://localhost:8001/s3/kkk/%E6%B1%9F%E5%AE%81%E8%B4%A2%E6%94%BF%E5%B1%80%E9%A1%B9%E7%9B%AE%E5%AF%B9%E6%8E%A5%E6%96%87%E6%A1%A3.docx?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230531T024715Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=admin%2F20230531%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=038e2ea71073761aa0370215621599649e9228177c332a0a79f784b1a6d9ee39

        ///region 參數準備
        //第一部分-憑證範圍
        String credential =request.getParameter("X-Amz-Credential");
        String[] credentials = credential.split("\\/");
        String accessKey = credentials[0];
        if (!accessKeyId.equals(accessKey)) {
            return false;
        }
        String date = credentials[1];
        String region = credentials[2];
        String service = credentials[3];
        String aws4Request = credentials[4];
        //第二部分-簽名頭中包含哪些字段
        String signedHeader = request.getParameter("X-Amz-SignedHeaders");
        String[] signedHeaders = signedHeader.split("\\;");
        //第三部分-生成的簽名
        String signature = request.getParameter("X-Amz-Signature");
        ///endregion

        ///region 驗證expire
        String expires = request.getParameter("X-Amz-Expires");
        DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'");
        LocalDateTime startDate = LocalDateTime.parse(requestDate,formatter);
        ZoneId zoneId = ZoneId.systemDefault();
        ZonedDateTime localDateTime = startDate.atZone(ZoneId.of("UTC")).withZoneSameInstant(zoneId);
        startDate = localDateTime.toLocalDateTime();
        LocalDateTime endDate = startDate.plusSeconds(ConvertOp.convert2Int(expires));
        if(endDate.isBefore(LocalDateTime.now())){
            return false;
        }
        ///endregion

        ///region 待簽名字符串
        String stringToSign = "";
        //簽名由4部分組成
        //1-Algorithm – 用於創建規範請求的哈希的算法。對於 SHA-256，算法是 AWS4-HMAC-SHA256。
        stringToSign += "AWS4-HMAC-SHA256" + "\n";
        //2-RequestDateTime – 在憑證範圍內使用的日期和時間。
        stringToSign += requestDate + "\n";
        //3-CredentialScope – 憑證範圍。這會將生成的簽名限制在指定的區域和服務範圍內。該字符串採用以下格式：YYYYMMDD/region/service/aws4_request
        stringToSign += date + "/" + region + "/" + service + "/" + aws4Request + "\n";
        //4-HashedCanonicalRequest – 規範請求的哈希。
        //<HTTPMethod>\n
        //<CanonicalURI>\n
        //<CanonicalQueryString>\n
        //<CanonicalHeaders>\n
        //<SignedHeaders>\n
        //<HashedPayload>
        String hashedCanonicalRequest = "";
        //4.1-HTTP Method
        hashedCanonicalRequest += httpMethod + "\n";
        //4.2-Canonical URI
        hashedCanonicalRequest += uri + "\n";
        //4.3-Canonical Query String
        if(!StringUtil.isEmpty(queryString)){
            Map<String, String> queryStringMap =  parseQueryParams(queryString);
            List<String> keyList = new ArrayList<>(queryStringMap.keySet());
            Collections.sort(keyList);
            StringBuilder queryStringBuilder = new StringBuilder("");
            for (String key:keyList) {
                if(!key.equals("X-Amz-Signature")){
                    queryStringBuilder.append(key).append("=").append(queryStringMap.get(key)).append("&");
                }
            }
            queryStringBuilder.deleteCharAt(queryStringBuilder.lastIndexOf("&"));

            hashedCanonicalRequest += queryStringBuilder.toString() + "\n";
        }else{
            hashedCanonicalRequest += queryString + "\n";
        }
        //4.4-Canonical Headers
        for (String name : signedHeaders) {
            hashedCanonicalRequest += name + ":" + request.getHeader(name) + "\n";
        }
        hashedCanonicalRequest += "\n";
        //4.5-Signed Headers
        hashedCanonicalRequest += signedHeader + "\n";
        //4.6-Hashed Payload
        hashedCanonicalRequest += contentHash;
        stringToSign += doHex(hashedCanonicalRequest);
        ///endregion

        ///region 重新生成簽名
        //計算簽名的key
        byte[] kSecret = ("AWS4" + secretAccessKey).getBytes("UTF8");
        byte[] kDate = doHmacSHA256(kSecret, date);
        byte[] kRegion = doHmacSHA256(kDate, region);
        byte[] kService = doHmacSHA256(kRegion, service);
        byte[] signatureKey = doHmacSHA256(kService, aws4Request);
        //計算簽名
        byte[] authSignature = doHmacSHA256(signatureKey, stringToSign);
        //對簽名編碼處理
        String strHexSignature = doBytesToHex(authSignature);
        ///endregion

        if (signature.equals(strHexSignature)) {
            return true;
        }
        return false;
    }

    private String doHex(String data) {
        MessageDigest messageDigest;
        try {
            messageDigest = MessageDigest.getInstance("SHA-256");
            messageDigest.update(data.getBytes("UTF-8"));
            byte[] digest = messageDigest.digest();
            return String.format("%064x", new java.math.BigInteger(1, digest));
        } catch (NoSuchAlgorithmException | UnsupportedEncodingException e) {
            e.printStackTrace();
        }
        return null;
    }

    private byte[] doHmacSHA256(byte[] key, String data) throws Exception {
        String algorithm = "HmacSHA256";
        Mac mac = Mac.getInstance(algorithm);
        mac.init(new SecretKeySpec(key, algorithm));
        return mac.doFinal(data.getBytes("UTF8"));
    }

    final protected static char[] hexArray = "0123456789ABCDEF".toCharArray();

    private String doBytesToHex(byte[] bytes) {
        char[] hexChars = new char[bytes.length * 2];
        for (int j = 0; j < bytes.length; j++) {
            int v = bytes[j] & 0xFF;
            hexChars[j * 2] = hexArray[v >>> 4];
            hexChars[j * 2 + 1] = hexArray[v & 0x0F];
        }
        return new String(hexChars).toLowerCase();
    }

    public static Map<String, String> parseQueryParams(String queryString) {
        Map<String, String> queryParams = new HashMap<>();
        try {
            if (queryString != null && !queryString.isEmpty()) {
                String[] queryParamsArray = queryString.split("\\&");

                for (String param : queryParamsArray) {
                    String[] keyValue = param.split("\\=");
                    if (keyValue.length == 1) {
                        String key = keyValue[0];
                        String value = "";
                        queryParams.put(key, value);
                    }
                    else if (keyValue.length == 2) {
                        String key = keyValue[0];
                        String value = keyValue[1];
                        queryParams.put(key, value);
                    }
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return queryParams;
    }

}