Skip to content

Instantly share code, notes, and snippets.

@ake-persson
Last active June 2, 2021 18:19
Show Gist options
  • Save ake-persson/1ca682f258d4ab43569b5b550bc0e66e to your computer and use it in GitHub Desktop.
Save ake-persson/1ca682f258d4ab43569b5b550bc0e66e to your computer and use it in GitHub Desktop.
Pull multiple docker image layers in parallel using curl
#!/bin/bash
set -eu
reg="registry.hub.docker.com"
repo="gliderlabs"
image="alpine"
name="${repo}/${image}"
tag="latest"
parallel=4
INFO="\033[1;32m"
WARN="\033[0;33m"
FATAL="\033[0;31m"
CLEAR='\033[0m'
info() {
if [ -n "${QUIET:-}" ]; then
return
fi
printf "* ${INFO}${1}${CLEAR}\n"
}
warn() {
printf "! ${WARN}${1}${CLEAR}\n"
}
fatal() {
printf "!! ${FATAL}${1}${CLEAR}\n"
exit 1
}
# Get auth token
token=$( curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${name}:pull" | jq -r .token )
# Get layers
resp=$(curl -s -H "Authorization: Bearer $token" "https://${reg}/v2/${name}/manifests/${tag}" | jq -r .fsLayers[].blobSum )
layers=( $( echo $resp | tr ' ' '\n' | sort -u ) )
prun() {
while (( "$#" )); do
for (( i=0; i<$parallel; i++ )); do
if [ -n "${1:-}" ]; then
if [ -f "${1}.tar.gz" ]; then
checksum=$( shasum -a 256 $1.tar.gz | awk '{ print $1 }' )
if [ "$checksum" != "${1##sha256:}" ]; then
warn "File exist checksum doesn't match, download again: ${1}.tar.gz"
curl -s -o $1.tar.gz -L -H "Authorization: Bearer $token" "https://${reg}/v2/${name}/blobs/${1}" &
else
info "Skip file: ${1}.tar.gz"
fi
else
info "Download: ${1}.tar.gz"
curl -s -o $1.tar.gz -L -H "Authorization: Bearer $token" "https://${reg}/v2/${name}/blobs/${1}" &
fi
shift
fi
done
wait
done
}
prun ${layers[@]}
# Run twice in-case of failure
QUIET="true"
prun ${layers[@]}
@patrickleboutillier
Copy link

How do you make sure the layers are reassembled in the correct order? If 2 layers modify the same file, they need to be reapplied in the correct order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment