Skip to content

Instantly share code, notes, and snippets.

@ake-persson
Last active June 2, 2021 18:19
Show Gist options
  • Save ake-persson/1ca682f258d4ab43569b5b550bc0e66e to your computer and use it in GitHub Desktop.
Save ake-persson/1ca682f258d4ab43569b5b550bc0e66e to your computer and use it in GitHub Desktop.
Pull multiple docker image layers in parallel using curl
#!/bin/bash
set -eu
reg="registry.hub.docker.com"
repo="gliderlabs"
image="alpine"
name="${repo}/${image}"
tag="latest"
parallel=4
INFO="\033[1;32m"
WARN="\033[0;33m"
FATAL="\033[0;31m"
CLEAR='\033[0m'
info() {
if [ -n "${QUIET:-}" ]; then
return
fi
printf "* ${INFO}${1}${CLEAR}\n"
}
warn() {
printf "! ${WARN}${1}${CLEAR}\n"
}
fatal() {
printf "!! ${FATAL}${1}${CLEAR}\n"
exit 1
}
# Get auth token
token=$( curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${name}:pull" | jq -r .token )
# Get layers
resp=$(curl -s -H "Authorization: Bearer $token" "https://${reg}/v2/${name}/manifests/${tag}" | jq -r .fsLayers[].blobSum )
layers=( $( echo $resp | tr ' ' '\n' | sort -u ) )
prun() {
while (( "$#" )); do
for (( i=0; i<$parallel; i++ )); do
if [ -n "${1:-}" ]; then
if [ -f "${1}.tar.gz" ]; then
checksum=$( shasum -a 256 $1.tar.gz | awk '{ print $1 }' )
if [ "$checksum" != "${1##sha256:}" ]; then
warn "File exist checksum doesn't match, download again: ${1}.tar.gz"
curl -s -o $1.tar.gz -L -H "Authorization: Bearer $token" "https://${reg}/v2/${name}/blobs/${1}" &
else
info "Skip file: ${1}.tar.gz"
fi
else
info "Download: ${1}.tar.gz"
curl -s -o $1.tar.gz -L -H "Authorization: Bearer $token" "https://${reg}/v2/${name}/blobs/${1}" &
fi
shift
fi
done
wait
done
}
prun ${layers[@]}
# Run twice in-case of failure
QUIET="true"
prun ${layers[@]}
@ake-persson
Copy link
Author

Something like this would work.

extract_image() {
	local source=$1 full
	shift

	while [ -n "$1" ]; do
		file="$source/${1}.tar.gz"
		tar --overwrite -xzf $file -h -C $ROOT \
			--exclude=./sys \
			--exclude=./dev \
			--exclude=./proc \
			--exclude=./etc/fstab
		shift
	done
}

@rohithmn3
Copy link

rohithmn3 commented Mar 15, 2018

@Mickep76 But how i am expecting is "at any point in time 4 downloads should run" - i mean suppose we sent 4 file-downloads to background and 2 among the 4 gets completed very soon because of very less size, i want another 2 files from the queue should be send for download.
Will that be possible..!?
help me here..!?

@patrickleboutillier
Copy link

How do you make sure the layers are reassembled in the correct order? If 2 layers modify the same file, they need to be reapplied in the correct order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment