Skip to content

Instantly share code, notes, and snippets.

Last active July 11, 2019 16:47
Show Gist options
  • Save mcg1969/cbb1760cea6b0671959d8cbf957c89bf to your computer and use it in GitHub Desktop.
Save mcg1969/cbb1760cea6b0671959d8cbf957c89bf to your computer and use it in GitHub Desktop.
Cached Miniconda in CircleCI
MINICONDA: "$HOME/miniconda"
# This ensures we don't accidentally pull in ~/.condarc
# if it happens to be there for some strange reason
CONDARC: "$MINICONDA/condarc.none"
- git fetch --depth 1000000
- git fetch --tags
- rm -rf virtualenvs venv .pyenv
- |
if [[ ! -f $MINICONDA/.finished ]]; then
wget $ANACONDA_REPO_URL/static/extras/
bash -b -p $MINICONDA
# Use --system so the settings are placed in $MINICONDA
# Add any other conda settings you like here
$CONDA config --system --add channels conda-forge
# If you'd like to make sure conda is up to date
$CONDA install conda conda-build anaconda-client --yes
touch $MINICONDA/.finished
# Install some packages in our _cache environment that we know we're going to
# need during the build process---say, your conda package's dependencies. Because
# CircleCI saves this cache _before_ the build process proper, there is no way
# to just let that build process populate the cache. So we install them into
# this environment---and then promptly remove them. I have some rather large
# packages in my particular use case so this helps me.
- $CONDA create -n cache ... --yes # add packages here
- $CONDA remove -n cache --all --yes
# Ideally, what I'd do is blow away the tarballs and leave the extracted packages.
# That would save the extraction cost during the build process. Unfortunately, the
# absence of a tarball is interpreted as a missing package, because there's no MD5
# signature available to verify. I understand why this is done (indeed I might
# have implemented it) but this is certainly a downside. So what I've chosen to
# do is clean out the extracted packages here and suffer the extraction cost,
# trading smaller cache tarball size for package extraction time. Experiments are
# needed to determine if it is worthwhile.
- $CONDA clean --packages --yes
# Likely unnecessary, but just in case.
- $CONDA build purge-all
- ~/miniconda
- $CONDA build conda-recipe
branch: master
# The anaconda-client configuration is not cached so we have to set the URL here
- $ANACONDA config --set url $ANACONDA_REPO_URL/api
- $ANACONDA -t $ANACONDA_REPO_TOKEN upload --user creditflow --label dev --summary "auto-uploaded by circleci" $MINICONDA/conda-bld/*/*.tar.bz2 --force
Copy link

mcg1969 commented May 8, 2017

An ongoing experiment in building a circle.yml that caches miniconda.

Copy link

msarahan commented Feb 4, 2018

Thanks @mcg1969, this came in handy for me today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment