Skip to content

Instantly share code, notes, and snippets.

@seanbamforth
Last active September 21, 2023 13:48
Show Gist options
  • Star 6 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save seanbamforth/9388507 to your computer and use it in GitHub Desktop.
Save seanbamforth/9388507 to your computer and use it in GitHub Desktop.
Calculate Amazon eTag hash for named file. Powershell
#Calculates an eTag for a local file that should match the S3 eTag of the uploaded file.
$md5 = new-object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
$blocksize = (1024*1024*5)
$startblocks = (1024*1024*16)
function AmazonEtagHashForFile($filename) {
$lines = 0
[byte[]] $binHash = @()
$reader = [System.IO.File]::Open($filename,"OPEN","READ")
if ((Get-Item $filename).length -gt $startblocks) {
$buf = new-object byte[] $blocksize
while (($read_len = $reader.Read($buf,0,$buf.length)) -ne 0){
$lines += 1
$binHash += $md5.ComputeHash($buf,0,$read_len)
}
$binHash=$md5.ComputeHash( $binHash )
}
else {
$lines = 1
$binHash += $md5.ComputeHash($reader)
}
$reader.Close()
$hash = [System.BitConverter]::ToString( $binHash )
$hash = $hash.Replace("-","").ToLower()
if ($lines -gt 1) {
$hash = $hash + "-$lines"
}
return $hash
}
@speaktech-account
Copy link

speaktech-account commented Dec 10, 2017

Thank you for a great script.

The actual value of eTag is different from the calculated value of eTag when I tested this script in case that file size is just 16MB because the multi-part upload into S3 works when file size is greater than "or equl to" 16MB.
I think that in line 14
if ((Get-Item $filename).length -gt $startblocks) { should be if ((Get-Item $filename).length -ge $startblocks) { .

@grenotjg
Copy link

grenotjg commented Apr 29, 2020

Thank you. This helped a lot.
I'm using this code to calculate the content-MD5 header.

So I had to convert the md5 to base64 (and remove the $hash = $hash + "-$lines part)

$bytes = New-Object -TypeName byte[] -ArgumentList ($hash.Length / 2)
   for ($j = 0; $j -lt $hash.Length; $j += 2) {
        $bytes[$j / 2] = [System.Convert]::ToByte($hash.Substring($j, 2), 16)
    }

    $EncodedText =[System.Convert]::ToBase64String($bytes)

    return $EncodedText

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment