Java实现AWS S3 V4 Authorization自定义验证

曾经终败给现在 2024-03-16 19:56 47阅读 0赞

## 前言 ##

最近在开发文件存储服务，需要符合s3的协议标准，可以直接接入aws-sdk，本文针对sdk发出请求的鉴权信息进行重新组合再签名验证有效性，sdk版本如下

<dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>s3</artifactId>
                <version>2.20.45</version>
            </dependency>

## 算法解析 ##

首先对V4版本签名算法的数据结构及签名流程进行拆解分析，以请求头签名为示例讲解

signature = doSign(waitSignString)

### 签名示例 ###

#### 请求头签名 ####

AWS4-HMAC-SHA256 Credential=admin/20230530/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=6f50628a101b46264c7783937be0366762683e0d319830b1844643e40b3b0ed

#### Url签名 ####

http://localhost:8001/s3/kkk/test.docx?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230531T024715Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=admin%2F20230531%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=038e2ea71073761aa0370215621599649e9228177c332a0a79f784b1a6d9ee39

### 数据结构 ###

waitSignString = doHex（【第一部分】+【第二部分】+【第三部分】+【第四部分】），每部分使用\\n换行符连接，第四部分不要加上换行符

#### 第一部分 ####

Algorithm – 用于创建规范请求的哈希的算法，对于 SHA-256，算法是 AWS4-HMAC-SHA256，则这部分的内容固定为

"AWS4-HMAC-SHA256" + "\n"

#### 第二部分 ####

RequestDateTime – 在凭证范围内使用的日期和时间，这个时间为请求发出的时间，直接从请求头获取x-amz-date即可，这部分内容为

request.getHeader("x-amz-date") + "\n"

#### 第三部分 ####

CredentialScope – 凭证范围，这会将生成的签名限制在指定的区域和服务范围内，该字符串采用以下格式：YYYYMMDD/region/service/aws4\_request

这部分由4个内容信息拼接组成

*  请求时间的YYYYMMDD格式
 *  存储区域
 *  存储服务
 *  请求头

这些信息我们都可以从请求头的Authorization凭证提取出Credential部分进行拆分重新组合

String[] parts = authorization.trim().split("\\,");
            String credential = parts[0].split("\\=")[1];
            String[] credentials = credential.split("\\/");
            String accessKey = credentials[0];
            if (!accessKeyId.equals(accessKey)) {
                return false;
            }
            String date = credentials[1];
            String region = credentials[2];
            String service = credentials[3];
            String aws4Request = credentials[4];

这部分内容为

date + "/" + region + "/" + service + "/" + aws4Request + "\n"

#### 第四部分 ####

HashedCanonicalRequest – 规范请求的哈希

这部分内容为

doHex(canonicalRequest)

canonicalRequest具体拆解又可以6小部分组成，每部分使用\\n换行符连接，最后不要加上换行符

<HTTPMethod>\n
    <CanonicalURI>\n
    <CanonicalQueryString>\n
    CanonicalHeaders>\n
    <SignedHeaders>\n
    <HashedPayload>

*  HTTPMethod
    
    代表请求的HTTP方法，例如GET，POST，DELETE，PUT等，直接从request获取即可
    
    这部分内容为
    
        String HTTPMethod = request.getMethod() + "\n"
 *  CanonicalURI
    
    代表请求的路由部分，例如完成请求为http://localhost:8001/s3/aaaa/ccc.txt，则该部分为/s3/aaaa/ccc.txt
    
    需要进行encode操作，我这里直接获取则省略了这部分
    
    这部分内容为
    
        String CanonicalURI = request.getRequestURI().split("\\?")[0] + "\n";
 *  CanonicalQueryString
    
    代表请求参数的拼接成字符串key1=value1&key2=value2这种形式，拼接的key需要按照字母排序
    
    value需要进行encode操作，我这里直接获取则省略了这部分
    
        String queryString = ConvertOp.convert2String(request.getQueryString());
                if(!StringUtil.isEmpty(queryString)){
                    Map<String, String> queryStringMap =  parseQueryParams(queryString);
                    List<String> keyList = new ArrayList<>(queryStringMap.keySet());
                    Collections.sort(keyList);
                    StringBuilder queryStringBuilder = new StringBuilder("");
                    for (String key:keyList) {
                        queryStringBuilder.append(key).append("=").append(queryStringMap.get(key)).append("&");
                    }
                    queryStringBuilder.deleteCharAt(queryStringBuilder.lastIndexOf("&"));
                }
        
            public static Map<String, String> parseQueryParams(String queryString) {
                Map<String, String> queryParams = new HashMap<>();
                try {
                    if (queryString != null && !queryString.isEmpty()) {
                        String[] queryParamsArray = queryString.split("\\&");
        
                        for (String param : queryParamsArray) {
                            String[] keyValue = param.split("\\=");
                            if (keyValue.length == 1) {
                                String key = keyValue[0];
                                String value = "";
                                queryParams.put(key, value);
                            }
                            else if (keyValue.length == 2) {
                                String key = keyValue[0];
                                String value = keyValue[1];
                                queryParams.put(key, value);
                            }
                        }
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
                return queryParams;
            }
    
    这部分内容为
    
        String CanonicalQueryString = queryStringBuilder.toString() + "\n"
 *  CanonicalHeaders
    
    代表请求头拼接成字符串key:value的形式，每个head部分使用\\n换行符连接，拼接的key需要按照字母排序
    
    签名的请求头从Authorization解析获取
    
        String signedHeader = parts[1].split("\\=")[1];
                String[] signedHeaders = signedHeader.split("\\;");
    
        String headString = "";
                for (String name : signedHeaders) {
                    headString += name + ":" + request.getHeader(name) + "\n";
                }
    
    这部分内容为
    
        String CanonicalHeaders = headString + "\n"
 *  SignedHeaders
    
    代表请求头的key部分，使用;隔开
    
    这部分内容为从Authorization解析中获取
    
    这部分内容为
    
        String SignedHeaders = signedHeader + "\n"
 *  HashedPayload
    
    代表请求body部分的签名，直接从requet的head提取x-amz-content-sha256内容
    
    这部分内容为
    
        String HashedPayload = Stringrequest.getHeader("x-amz-content-sha256")

#### doHex ####

本部分只是一个字符串转16进制的一个操作

private String doHex(String data) {
            MessageDigest messageDigest;
            try {
                messageDigest = MessageDigest.getInstance("SHA-256");
                messageDigest.update(data.getBytes("UTF-8"));
                byte[] digest = messageDigest.digest();
                return String.format("%064x", new java.math.BigInteger(1, digest));
            } catch (NoSuchAlgorithmException | UnsupportedEncodingException e) {
                e.printStackTrace();
            }
            return null;
        }

### 签名流程 ###

doSign 的流程为doBytesToHex（doHmacSHA256（signatureKey，waitSignString ））

doBytesToHex为byte转16进制操作

private String doBytesToHex(byte[] bytes) {
            char[] hexChars = new char[bytes.length * 2];
            for (int j = 0; j < bytes.length; j++) {
                int v = bytes[j] & 0xFF;
                hexChars[j * 2] = hexArray[v >>> 4];
                hexChars[j * 2 + 1] = hexArray[v & 0x0F];
            }
            return new String(hexChars).toLowerCase();
        }

doHmacSHA256为签名算法

private byte[] doHmacSHA256(byte[] key, String data) throws Exception {
            String algorithm = "HmacSHA256";
            Mac mac = Mac.getInstance(algorithm);
            mac.init(new SecretKeySpec(key, algorithm));
            return mac.doFinal(data.getBytes("UTF8"));
        }

signatureKey签名密钥由secretAccessKey，请求时间，存储区域，存储服务，请求头这5个要素进行叠加签名生成

byte[] kSecret = ("AWS4" + secretAccessKey).getBytes("UTF8");
            byte[] kDate = doHmacSHA256(kSecret, date);
            byte[] kRegion = doHmacSHA256(kDate, region);
            byte[] kService = doHmacSHA256(kRegion, service);
            byte[] signatureKey = doHmacSHA256(kService, aws4Request);

将最终生成的再签名与Authorization中解析出的Signature进行比较，一致则鉴权成功

### 调试位置 ###

调试过程中需要验证每部分的签名是否拼接编码正确，我们需要和sdk生成的内容进行比对找出问题

调试software.amazon.awssdk.auth.signer.internal包下AbstractAws4Signer类的doSign类，获取stringToSign与你待签名字符串比对差异，源码如下

protected Builder doSign(SdkHttpFullRequest request, Aws4SignerRequestParams requestParams, T signingParams, ContentChecksum contentChecksum) {
            Builder mutableRequest = request.toBuilder();
            AwsCredentials sanitizedCredentials = this.sanitizeCredentials(signingParams.awsCredentials());
            if (sanitizedCredentials instanceof AwsSessionCredentials) {
                this.addSessionCredentials(mutableRequest, (AwsSessionCredentials)sanitizedCredentials);
            }
    
            this.addHostHeader(mutableRequest);
            this.addDateHeader(mutableRequest, requestParams.getFormattedRequestSigningDateTime());
            mutableRequest.firstMatchingHeader("x-amz-content-sha256").filter((h) -> {
                return h.equals("required");
            }).ifPresent((h) -> {
                mutableRequest.putHeader("x-amz-content-sha256", contentChecksum.contentHash());
            });
            this.putChecksumHeader(signingParams.checksumParams(), contentChecksum.contentFlexibleChecksum(), mutableRequest, contentChecksum.contentHash());
            AbstractAws4Signer.CanonicalRequest canonicalRequest = this.createCanonicalRequest(request, mutableRequest, contentChecksum.contentHash(), signingParams.doubleUrlEncode(), signingParams.normalizePath());
            String canonicalRequestString = canonicalRequest.string();
            String stringToSign = this.createStringToSign(canonicalRequestString, requestParams);
            byte[] signingKey = this.deriveSigningKey(sanitizedCredentials, requestParams);
            byte[] signature = this.computeSignature(stringToSign, signingKey);
            mutableRequest.putHeader("Authorization", this.buildAuthorizationHeader(signature, sanitizedCredentials, requestParams, canonicalRequest));
            this.processRequestPayload(mutableRequest, signature, signingKey, requestParams, signingParams, contentChecksum.contentFlexibleChecksum());
            return mutableRequest;
        }

## 代码示例 ##

通过拦截器进行验证的过程，完整代码如下，兼容了普通请求的头部验证和文件下载url的签名验证

@Component
    public class S3Intecept implements HandlerInterceptor {
        @Autowired
        private SystemConfig systemConfig;
    
        @Override
        public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
            boolean flag = false;
            String authorization = request.getHeader("Authorization");
            if(!StringUtil.isEmpty(authorization)){
                flag = validAuthorizationHead(request, systemConfig.getUsername(), systemConfig.getPassword());
            }else{
                authorization = request.getParameter("X-Amz-Credential");
                if(!StringUtil.isEmpty(authorization)){
                    flag = validAuthorizationUrl(request, systemConfig.getUsername(), systemConfig.getPassword());
                }
            }
            if(!flag){
                response.setStatus(HttpStatus.UNAUTHORIZED.value());
            }
            return flag;
        }
    
        public boolean validAuthorizationHead(HttpServletRequest request, String accessKeyId, String secretAccessKey) throws Exception {
            String authorization = request.getHeader("Authorization");
            String requestDate = request.getHeader("x-amz-date");
            String contentHash = request.getHeader("x-amz-content-sha256");
            String httpMethod = request.getMethod();
            String uri = request.getRequestURI().split("\\?")[0];
            String queryString = ConvertOp.convert2String(request.getQueryString());
            //示例
            //AWS4-HMAC-SHA256 Credential=admin/20230530/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=6f50628a101b46264c7783937be0366762683e0d319830b1844643e40b3b0ed
    
            ///region authorization拆分
            String[] parts = authorization.trim().split("\\,");
            //第一部分-凭证范围
            String credential = parts[0].split("\\=")[1];
            String[] credentials = credential.split("\\/");
            String accessKey = credentials[0];
            if (!accessKeyId.equals(accessKey)) {
                return false;
            }
            String date = credentials[1];
            String region = credentials[2];
            String service = credentials[3];
            String aws4Request = credentials[4];
            //第二部分-签名头中包含哪些字段
            String signedHeader = parts[1].split("\\=")[1];
            String[] signedHeaders = signedHeader.split("\\;");
            //第三部分-生成的签名
            String signature = parts[2].split("\\=")[1];
            ///endregion
    
            ///region 待签名字符串
            String stringToSign = "";
            //签名由4部分组成
            //1-Algorithm – 用于创建规范请求的哈希的算法。对于 SHA-256，算法是 AWS4-HMAC-SHA256。
            stringToSign += "AWS4-HMAC-SHA256" + "\n";
            //2-RequestDateTime – 在凭证范围内使用的日期和时间。
            stringToSign += requestDate + "\n";
            //3-CredentialScope – 凭证范围。这会将生成的签名限制在指定的区域和服务范围内。该字符串采用以下格式：YYYYMMDD/region/service/aws4_request
            stringToSign += date + "/" + region + "/" + service + "/" + aws4Request + "\n";
            //4-HashedCanonicalRequest – 规范请求的哈希。
            //<HTTPMethod>\n
            //<CanonicalURI>\n
            //<CanonicalQueryString>\n
            //<CanonicalHeaders>\n
            //<SignedHeaders>\n
            //<HashedPayload>
            String hashedCanonicalRequest = "";
            //4.1-HTTP Method
            hashedCanonicalRequest += httpMethod + "\n";
            //4.2-Canonical URI
            hashedCanonicalRequest += uri + "\n";
            //4.3-Canonical Query String
            if(!StringUtil.isEmpty(queryString)){
                Map<String, String> queryStringMap =  parseQueryParams(queryString);
                List<String> keyList = new ArrayList<>(queryStringMap.keySet());
                Collections.sort(keyList);
                StringBuilder queryStringBuilder = new StringBuilder("");
                for (String key:keyList) {
                    queryStringBuilder.append(key).append("=").append(queryStringMap.get(key)).append("&");
                }
                queryStringBuilder.deleteCharAt(queryStringBuilder.lastIndexOf("&"));
    
                hashedCanonicalRequest += queryStringBuilder.toString() + "\n";
            }else{
                hashedCanonicalRequest += queryString + "\n";
            }
            //4.4-Canonical Headers
            for (String name : signedHeaders) {
                hashedCanonicalRequest += name + ":" + request.getHeader(name) + "\n";
            }
            hashedCanonicalRequest += "\n";
            //4.5-Signed Headers
            hashedCanonicalRequest += signedHeader + "\n";
            //4.6-Hashed Payload
            hashedCanonicalRequest += contentHash;
            stringToSign += doHex(hashedCanonicalRequest);
            ///endregion
    
            ///region 重新生成签名
            //计算签名的key
            byte[] kSecret = ("AWS4" + secretAccessKey).getBytes("UTF8");
            byte[] kDate = doHmacSHA256(kSecret, date);
            byte[] kRegion = doHmacSHA256(kDate, region);
            byte[] kService = doHmacSHA256(kRegion, service);
            byte[] signatureKey = doHmacSHA256(kService, aws4Request);
            //计算签名
            byte[] authSignature = doHmacSHA256(signatureKey, stringToSign);
            //对签名编码处理
            String strHexSignature = doBytesToHex(authSignature);
            ///endregion
    
            if (signature.equals(strHexSignature)) {
                return true;
            }
            return false;
        }
    
        public boolean validAuthorizationUrl(HttpServletRequest request, String accessKeyId, String secretAccessKey) throws Exception {
            String requestDate = request.getParameter("X-Amz-Date");
            String contentHash = "UNSIGNED-PAYLOAD";
            String httpMethod = request.getMethod();
            String uri = request.getRequestURI().split("\\?")[0];
            String queryString = ConvertOp.convert2String(request.getQueryString());
            //示例
            //"http://localhost:8001/s3/kkk/%E6%B1%9F%E5%AE%81%E8%B4%A2%E6%94%BF%E5%B1%80%E9%A1%B9%E7%9B%AE%E5%AF%B9%E6%8E%A5%E6%96%87%E6%A1%A3.docx?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230531T024715Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=admin%2F20230531%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=038e2ea71073761aa0370215621599649e9228177c332a0a79f784b1a6d9ee39
    
            ///region 参数准备
            //第一部分-凭证范围
            String credential =request.getParameter("X-Amz-Credential");
            String[] credentials = credential.split("\\/");
            String accessKey = credentials[0];
            if (!accessKeyId.equals(accessKey)) {
                return false;
            }
            String date = credentials[1];
            String region = credentials[2];
            String service = credentials[3];
            String aws4Request = credentials[4];
            //第二部分-签名头中包含哪些字段
            String signedHeader = request.getParameter("X-Amz-SignedHeaders");
            String[] signedHeaders = signedHeader.split("\\;");
            //第三部分-生成的签名
            String signature = request.getParameter("X-Amz-Signature");
            ///endregion
    
            ///region 验证expire
            String expires = request.getParameter("X-Amz-Expires");
            DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'");
            LocalDateTime startDate = LocalDateTime.parse(requestDate,formatter);
            ZoneId zoneId = ZoneId.systemDefault();
            ZonedDateTime localDateTime = startDate.atZone(ZoneId.of("UTC")).withZoneSameInstant(zoneId);
            startDate = localDateTime.toLocalDateTime();
            LocalDateTime endDate = startDate.plusSeconds(ConvertOp.convert2Int(expires));
            if(endDate.isBefore(LocalDateTime.now())){
                return false;
            }
            ///endregion
    
            ///region 待签名字符串
            String stringToSign = "";
            //签名由4部分组成
            //1-Algorithm – 用于创建规范请求的哈希的算法。对于 SHA-256，算法是 AWS4-HMAC-SHA256。
            stringToSign += "AWS4-HMAC-SHA256" + "\n";
            //2-RequestDateTime – 在凭证范围内使用的日期和时间。
            stringToSign += requestDate + "\n";
            //3-CredentialScope – 凭证范围。这会将生成的签名限制在指定的区域和服务范围内。该字符串采用以下格式：YYYYMMDD/region/service/aws4_request
            stringToSign += date + "/" + region + "/" + service + "/" + aws4Request + "\n";
            //4-HashedCanonicalRequest – 规范请求的哈希。
            //<HTTPMethod>\n
            //<CanonicalURI>\n
            //<CanonicalQueryString>\n
            //<CanonicalHeaders>\n
            //<SignedHeaders>\n
            //<HashedPayload>
            String hashedCanonicalRequest = "";
            //4.1-HTTP Method
            hashedCanonicalRequest += httpMethod + "\n";
            //4.2-Canonical URI
            hashedCanonicalRequest += uri + "\n";
            //4.3-Canonical Query String
            if(!StringUtil.isEmpty(queryString)){
                Map<String, String> queryStringMap =  parseQueryParams(queryString);
                List<String> keyList = new ArrayList<>(queryStringMap.keySet());
                Collections.sort(keyList);
                StringBuilder queryStringBuilder = new StringBuilder("");
                for (String key:keyList) {
                    if(!key.equals("X-Amz-Signature")){
                        queryStringBuilder.append(key).append("=").append(queryStringMap.get(key)).append("&");
                    }
                }
                queryStringBuilder.deleteCharAt(queryStringBuilder.lastIndexOf("&"));
    
                hashedCanonicalRequest += queryStringBuilder.toString() + "\n";
            }else{
                hashedCanonicalRequest += queryString + "\n";
            }
            //4.4-Canonical Headers
            for (String name : signedHeaders) {
                hashedCanonicalRequest += name + ":" + request.getHeader(name) + "\n";
            }
            hashedCanonicalRequest += "\n";
            //4.5-Signed Headers
            hashedCanonicalRequest += signedHeader + "\n";
            //4.6-Hashed Payload
            hashedCanonicalRequest += contentHash;
            stringToSign += doHex(hashedCanonicalRequest);
            ///endregion
    
            ///region 重新生成签名
            //计算签名的key
            byte[] kSecret = ("AWS4" + secretAccessKey).getBytes("UTF8");
            byte[] kDate = doHmacSHA256(kSecret, date);
            byte[] kRegion = doHmacSHA256(kDate, region);
            byte[] kService = doHmacSHA256(kRegion, service);
            byte[] signatureKey = doHmacSHA256(kService, aws4Request);
            //计算签名
            byte[] authSignature = doHmacSHA256(signatureKey, stringToSign);
            //对签名编码处理
            String strHexSignature = doBytesToHex(authSignature);
            ///endregion
    
            if (signature.equals(strHexSignature)) {
                return true;
            }
            return false;
        }
    
        private String doHex(String data) {
            MessageDigest messageDigest;
            try {
                messageDigest = MessageDigest.getInstance("SHA-256");
                messageDigest.update(data.getBytes("UTF-8"));
                byte[] digest = messageDigest.digest();
                return String.format("%064x", new java.math.BigInteger(1, digest));
            } catch (NoSuchAlgorithmException | UnsupportedEncodingException e) {
                e.printStackTrace();
            }
            return null;
        }
    
        private byte[] doHmacSHA256(byte[] key, String data) throws Exception {
            String algorithm = "HmacSHA256";
            Mac mac = Mac.getInstance(algorithm);
            mac.init(new SecretKeySpec(key, algorithm));
            return mac.doFinal(data.getBytes("UTF8"));
        }
    
        final protected static char[] hexArray = "0123456789ABCDEF".toCharArray();
    
        private String doBytesToHex(byte[] bytes) {
            char[] hexChars = new char[bytes.length * 2];
            for (int j = 0; j < bytes.length; j++) {
                int v = bytes[j] & 0xFF;
                hexChars[j * 2] = hexArray[v >>> 4];
                hexChars[j * 2 + 1] = hexArray[v & 0x0F];
            }
            return new String(hexChars).toLowerCase();
        }
    
        public static Map<String, String> parseQueryParams(String queryString) {
            Map<String, String> queryParams = new HashMap<>();
            try {
                if (queryString != null && !queryString.isEmpty()) {
                    String[] queryParamsArray = queryString.split("\\&");
    
                    for (String param : queryParamsArray) {
                        String[] keyValue = param.split("\\=");
                        if (keyValue.length == 1) {
                            String key = keyValue[0];
                            String value = "";
                            queryParams.put(key, value);
                        }
                        else if (keyValue.length == 2) {
                            String key = keyValue[0];
                            String value = keyValue[1];
                            queryParams.put(key, value);
                        }
                    }
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
            return queryParams;
        }