Skip to content

Instantly share code, notes, and snippets.

@daryltucker
Last active December 15, 2023 23:19
Show Gist options
  • Save daryltucker/8bfd0c263667393a844070993b136df6 to your computer and use it in GitHub Desktop.
Save daryltucker/8bfd0c263667393a844070993b136df6 to your computer and use it in GitHub Desktop.
Chunk Upload to Azure Storage with curl
#!/bin/bash
# AZ_SAS_TOKEN=''
# ArtifactLocation=''
# ArtifactBaseName=''
DATE_NOW=$(date -Ru | sed 's/\\+0000/GMT/')
AZ_VERSION="2022-11-02"
AZ_BLOB_URL="https://STORAGE_ACCOUNT_HERE.blob.core.windows.net"
AZ_BLOB_CONTAINER="${ApplicationName}"
AZ_BLOB_TARGET="${AZ_BLOB_URL}/${AZ_BLOB_CONTAINER}/"
ARTIFACT_VERSION=$(sed 's/ //g' ${ArtifactLocation}VERSION.txt)
ARTIFACT_FILE="${ArtifactLocation}${ArtifactBaseName}-${ARTIFACT_VERSION}.7z"
fname=$(basename ${ARTIFACT_FILE// /_})
fsize=$(stat -c %s "${ARTIFACT_FILE}")
csize=$((99 * 1024 * 1024))
chunks=$((($fsize + $csize - 1) / $csize))
xml="<?xml version='1.0' encoding='utf-8'?>
<BlockList>"
curl -i -v -X PUT -H "x-ms-version: ${AZ_VERSION}" -H "x-ms-blob-content-type: application/octet-stream" -H "x-ms-blob-type: BlockBlob" -d "" "${AZ_BLOB_TARGET}$fname${AZ_SAS_TOKEN}"
for ((i = 0; i < chunks; i++)); do
start=$((i * csize))
end=$(($start + $csize - 1))
if ((end >= fsize)); then
end=$((fsize - 1))
fi
range="$start-$end"
cfile="$(basename ${ARTIFACT_FILE})_CHUNK_$i"
dd if="${ARTIFACT_FILE}" of="$cfile" bs=$csize skip=$i count=1 2>/dev/null
bid=$(echo -n "${fname}_$(printf "%04d" $i)" | base64 -w 0)
# -H "x-ms-blob-content-type: application/octet-stream"
#HEADERS="-H \"x-ms-date: ${DATE_NOW}\" -H \"x-ms-version: ${AZ_VERSION}\""
curl -i -v -X PUT -H "x-ms-version: ${AZ_VERSION}" -H "Content-Length: $((end - start + 1))" -H "Content-Range: bytes $range/$fsize" --data-binary "@$cfile" "${AZ_BLOB_TARGET}$fname${AZ_SAS_TOKEN}&comp=block&blockid=$bid"
rm "$cfile"
xml+="
<Latest>$bid</Latest>"
done
xml+="
</BlockList>"
#echo $xml
curl -f -v -X PUT -H "x-ms-version: ${AZ_VERSION}" -H "x-ms-blob-content-type: application/octet-stream" -H "Content-Type: application/xml" -d "$xml" "${AZ_BLOB_TARGET}$fname${AZ_SAS_TOKEN}&comp=blocklist"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment