I want to use an object's MD5 checksum value to verify the integrity of the object when I upload it to an Amazon S3 bucket. I also want to store the MD5 checksum value in a custom HTTP header so that the object's integrity can be verified when it is downloaded from S3.
First, calculate the MD5 checksum of the object that you are uploading to S3. When you upload the object by using the AWS Command Line Interface (CLI), you specify certain parameters to verify the integrity of the object and store the object's MD5 checksum value in an HTTP header.
Follow these steps to calculate the MD5 checksum of an object, verify the integrity of the object when you upload it to Amazon S3, and store the MD5 checksum value in a custom HTTP header. The MD5 checksum value can be used to verify the integrity of the object when it is downloaded from S3.
1. Obtain the base64 MD5 checksum of the file to be uploaded.
Using Windows - Download the File Checksum Integrity Verifier (FCIV) utility and extract the contents to a folder. Then add the location of the folder to the Windows system path by running the following command from an elevated (run as Administrator) command prompt, replacing c:\fciv with the folder that contains the extracted the FCIV utility files:
C:>set path=%path%;c:\fciv
Note that when you modify the Windows system path from a command prompt, the change does not persist when Windows is restarted. If you want to modify the Windows system path environment variable permanently, check the Windows documentation or search the Web for "Change Windows X path variable", substituting your version of Windows for X.
After installing the FCIV utility and updating the %path% environment variable with the location of the extracted FCIV utility files, run this command to return the hexadecimal MD5 checksum of the file to be uploaded to S3. Replace c:\S3\testfile with the location of the file you are uploading to S3:
fciv.exe c:\s3\testfile
Note: If the path to the file contains spaces, enclose the path in quotation marks (").
The value returned will be similar to the following value returned when calculating the MD5 checksum of the file C:\Windows\explorer.exe:
fciv C:\Windows\explorer.exe
//
// File Checksum Integrity Verifier version 2.05.
//
e1b0af69bfb6cbde9b53c55e4bf91992 c:\windows\explorer.exe
Important: The MD5 checksum returned by the FCIV utility is hexadecimal, and must be converted to base64 before it can be used as a checksum value for uploading messages to S3. If you use a hexadecimal MD5 checksum, you will receive the error message "The Content-MD5 you specified is invalid". There are several hexadecimal to base64 string decoders available on the Web; if you prefer, you can download script code such as the HexToBase64 script available at http://www.rlmueller.net/Programs/HexToBase64.txt. The base64 encoded equivalent of the hexadecimal value returned by the FCIV utility in the example is WZOTosUmxoARnYQVXZDx5Q==.
Using Linux - Linux natively provides the ability to calculate the base64 MD5 checksum of a file with the openssl command. To determine the base64 MD5 checksum for a file in Linux, run the following command from a Linux shell:
openssl md5 -binary PATH/TO/FILE | base64
The value returned will be similar to the following value returned when retrieving the base64 MD5 checksum of the file /bin/bash:
user@example:/home$ openssl md5 -binary /bin/bash | base64
+e9lnJtCrdoKwYqg9wlFwA==
2. Upload the object to Amazon S3.
Start the AWS CLI and run the aws s3api put-object command to upload the object to an S3 bucket. To ensure that S3 verifies the integrity of the object and stores the MD5 checksum in a custom HTTP header you must use the --content-md5 and --metadata arguments with the appropriate parameters.
In the following example, the aws s3api put-object command is used to upload the local file c:\S3\localfile.txt to the S3 bucket md5testbucket as fileupload.txt. This command also verifies the integrity of the file with the specified MD5 checksum value while saving the MD5 checksum value to the custom HTTP header x-amz-meta-md5chksum.
C:\S3\>aws s3api put-object --bucket md5testbucket --key fileupload.txt --body localfile.txt --metadata md5chksum=WZOTosUmxoARnYQVXZDx5Q== --content-md5 WZOTosUmxoARnYQVXZDx5Q==
{
"ETag": "\"599393a2c526c680119d84155d90f1e5\""
}
Note: If no ETag value is returned, an error similar to the following is returned by the AWS CLI:
A client error (InvalidDigest) occurred when calling the PutObject operation: The Content-MD5 you specified was invalid.
This error indicates a mismatch between the specified MD5 checksum value and the object that you are uploading to S3. The integrity of the object being uploaded cannot be verified until this mismatch is corrected.
To verify that the object was uploaded and that the MD5 checksum value was written to the custom HTTP header, run the following command from the AWS CLI:
C:\S3>aws s3api head-object --bucket md5testbucket --key fileupload.txt
{
"AcceptRanges": "bytes",
"ContentType": "binary/octet-stream",
"LastModified": "Thu, 31 Mar 2016 16:37:18 GMT",
"ContentLength": 605,
"ETag": "\"599393a2c526c680119d84155d90f1e5\"",
"Metadata": {
"md5chksum": "WZOTosUmxoARnYQVXZDx5Q=="
}
}
The value returned for md5chksum should match the value specified when you uploaded the file to S3. This value can be used to verify the integrity of the object when it is downloaded from S3.
Amazon S3, verify file integrity, upload, MD5 checksum, download, ETag, aws s3api, put-object, head-object, metadata, AWS CLI
Did this page help you? Yes | No
Back to the AWS Support Knowledge Center
Need help? Visit the AWS Support Center
Published: 2016-04-01
