How to debug conda-forge recipes

how-to-conda-forge.md

A Short Guide for Debugging conda-forge Recipes

Documentation for trouble-shooting broken conda-forge feedstocks exists in several places scattered throughout GitHub, the conda-build docu and the "maintainer" section of the conda-forge docs. However, I was not able to find a concise step by step guide explaining how and what to do when. So here we go:

1. The very first step should be to fork your recipe from conda-forge to your private account. Then clone the fork on your machine and create a new branch for the fixes, e.g.,

   git clone git@github.com:pathto/my-feedstock.git
   cd my-feedstock
   git checkout -b fix_recipe
   git push -u origin fix_recipe

2. As stated in the forge docs, first try to build your recipe locally:

   conda activate myenv
   conda install conda-build
   conda build -c conda-forge .

   (use -c to supply any other channels you obtain your deps from). This builds the package locally on your system. Chances are that if the conda-forge CI pipelines crashed, the local build fails too, but maybe with a clearer error message.

3. If the local conda build did not yield any illuminating insights, you might want to re-create the exact environment that is used by the conda-forge pipelines (it might be a good idea to run the CI pipelines locally before pushing upstream even in any case...). Note that this requires a working Docker installation:

   conda activate myenv
   python build-locally.py

   If the conda-forge CI pipelines failed, the above command should fail with the exact same error message as well.

4. Now re-create the conda CI environment(s) for local debugging:

   conda debug -c conda-forge .

   This should set up the respective environments for you and end in a pretty self-explaining message:

   ################################################################################
   Test environment created for debugging.  To enter a debugging environment:
   
   cd /path/to/conda/envs/my-feedstock/conda-bld/debug_1624284554410/work && source /path/to/conda/envs/my-feedstock/conda-bld/debug_1624284554410/work/build_env_setup.sh
   
   To run your tests, you might want to start with running the conda_test_runner.sh file.
   ################################################################################

5. Copy-paste the above command in your terminal and start exploring what's going wrong in the build and/or test scripts. Careful though, both scripts are written to exit on errors which might close your (carefully set up) shell, so better run the scripts in an explicit subshell:

   cd /path/to/conda/envs/my-feedstock/conda-bld/debug_1624284554410/work && source /path/to/conda/envs/my-feedstock/conda-bld/debug_1624284554410/work/build_env_setup.sh
   bash conda_build.sh
   bash conda_test_runner.sh

Being able to interactively explore the conda environments that are automatically generated for creating the feedstock proved to be really helpful for tracking down the reasons for the failed build.