Skip to content

Instantly share code, notes, and snippets.

@lossyrob
Last active August 20, 2021 16:40
Show Gist options
  • Save lossyrob/7e1a340f558e8c719778f159e0315b53 to your computer and use it in GitHub Desktop.
Save lossyrob/7e1a340f558e8c719778f159e0315b53 to your computer and use it in GitHub Desktop.
Forcing an Azure Batch node to pull from ACR without restarting the node

Forcing an Azure Batch node to pull from ACR

Sometimes you want to keep a node in an Azure Batch pool up and running as you're debugging tasks so that you don't have to pay the waiting time to spin up and down a node. You may want to push a new image to ACR with the same tag to run the job based on fixes based on the last run. However, Azure Batch nodes don't pull an updated image by default, and will run with the same image if a task is run again even if ACR has been updated.

The best solution I've found to speed things up is:

  • In Batch Explorer, go to the pool view, right click on the node, and "connect"
  • This will ssh into the node. Inside of the shell, run:
sudo docker image ls -q | xargs -I{} sudo docker tag {} old-{} && \
  sudo docker rmi $(sudo docker images | awk '$1 ~ /${DOCKER_IMAGE_STR}/ { print $1,$2 }' | sed -En "s/(.*) (.*)/\1:\2/p")

Where ${DOCKER_IMAGE_STR} is a string that's in the relevant docker image names, e.g. "pc" for the planetary computer images.

This will re-tag images with an old- prefix and remove the original image tags. Because the image has a different tag now, the batch node will re-pull the images. Because the images still exist on the machine, the pull will be fast because it's based on the existing retagged images.

@zstatmanweil
Copy link

Thanks, Rob! I missed your comment in the chat until the very end but I definitely am going to give this a shot as I do often just sit there twiddling my thumbs while the the node shuts down and restarts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment