Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
A bash script which supports uploading blobs to Azure Storage: ./upload-to-azure.sh [filename]
#!/usr/bin/env bash
FILENAME=${1}
# expected to be defined in the environment
# - AZURE_STORAGE_ACCOUNT
# - AZURE_CONTAINER_NAME
# - AZURE_ACCESS_KEY
# inspired by
# https://stackoverflow.com/questions/20103258/accessing-azure-blob-storage-using-bash-curl
authorization="SharedKey"
HTTP_METHOD="PUT"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2015-02-21"
# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"
x_ms_blob_type_h="x-ms-blob-type:BlockBlob"
FILE_LENGTH=$(wc --bytes < ${FILENAME})
FILE_TYPE=$(file --mime-type -b ${FILENAME})
FILE_MD5=$(md5sum -b ${FILENAME} | awk '{ print $1 }')
# Build the signature string
canonicalized_headers="${x_ms_blob_type_h}\n${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${AZURE_STORAGE_ACCOUNT}/${AZURE_CONTAINER_NAME}/${FILE_MD5}"
#######
# From: https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
#
#StringToSign = VERB + "\n" +
# Content-Encoding + "\n" +
# Content-Language + "\n" +
# Content-Length + "\n" +
# Content-MD5 + "\n" +
# Content-Type + "\n" +
# Date + "\n" +
# If-Modified-Since + "\n" +
# If-Match + "\n" +
# If-None-Match + "\n" +
# If-Unmodified-Since + "\n" +
# Range + "\n" +
# CanonicalizedHeaders +
# CanonicalizedResource;
string_to_sign="${HTTP_METHOD}\n\n\n${FILE_LENGTH}\n\n${FILE_TYPE}\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}"
# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $AZURE_ACCESS_KEY | base64 -d -w0 | xxd -p -c256)"
# Create the HMAC signature for the Authorization header
signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)
authorization_header="Authorization: $authorization $AZURE_STORAGE_ACCOUNT:$signature"
OUTPUT_FILE="https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_CONTAINER_NAME}/${FILE_MD5}"
curl -X ${HTTP_METHOD} \
-T ${FILENAME} \
-H "$x_ms_date_h" \
-H "$x_ms_version_h" \
-H "$x_ms_blob_type_h" \
-H "$authorization_header" \
-H "Content-Type: ${FILE_TYPE}" \
${OUTPUT_FILE}
if [ $? -eq 0 ]; then
echo ${OUTPUT_FILE}
exit 0;
fi;
exit 1
@codegagan
Copy link

codegagan commented Jun 19, 2018

Getting error

<?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit.
RequestId:f93e3dbc-e01e-0045-098e-07d78b000000
Time:2018-06-19T05:28:40.8713951Z</Message><MaxLimit>67108864</MaxLimit>

@saikovvuri
Copy link

saikovvuri commented Aug 16, 2019

Hello @rtyler.
I am trying to create a file system on an adlsgen 2 account with hierarchical namespace enabled.
However I am getting a different error than @codegagan

curl: (6) Could not resolve host: PUT; Unknown error
{"error":{"code":"AuthenticationFailed","message":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:27d7af5c-e01f-008f-7f44-5406ff000000\nTime:2019-08-16T15:06:27.7274914Z"}}https://storageaccountname.dfs.core.windows.net/filesystem?resource=filesystem

Could you advise on what might be wrong?

thanks

@M4rian
Copy link

M4rian commented Sep 11, 2019

useless

@gunnypatel
Copy link

gunnypatel commented Mar 22, 2020

Exactly what I was looking for and works perfectly.

@bhasi10
Copy link

bhasi10 commented Apr 24, 2020

https://gist.github.com/rtyler/30e51dc72bed23718388c43f9c11da76#gistcomment-3000362

@saikovvuri I am facing the same error. Were you able to resolve the error.? If so, how.?

@bhasi10
Copy link

bhasi10 commented Apr 24, 2020

@gunnypatel should you open up any firewall ports before pushing the file.?

@gunnypatel
Copy link

gunnypatel commented Apr 25, 2020

You shouldn't have to do anything with the firewall as long as you can make HTTPS calls to Azure. If you're getting an authentication errors then there's something wrong with the header being sent to Azure. Check to make sure that the time on your system is correct. I think if it's off by a few minutes it could cause an issue with the authentication headers.

@ShepFc3
Copy link

ShepFc3 commented Feb 18, 2022

Thank you, this was very helpful! 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment