Setting up custom conda feedstock

customfeedstock.md

Setting up a feedstock from scratch

This is a step-by-step tutorial for setting up your own conda feedstock from scratch. I started with the instructions here:

https://medium.com/@lgouarin/conda-smithy-an-easy-way-to-build-and-deploy-your-conda-packages-12216058648f

As a novice, I found that the above tutorial lacked sufficient detail. So, I decided to document the process that I used for my own future reference.

0. Prerequisites

I assume that you are familiar with the conda-build system and have a recipe to build your software. In my case, I started with the isce2 recipe from https://github.com/isce-framework/recipes-test

1. Creating your tokens

Setup accounts on circleci, travis, github, appveyor and azure. Save the tokens in your $HOME/.conda-smithy folder as individual tokens with ".token" prefix

anaconda.token

You can create a token here: https://anaconda.org/yourusername/settings/access

This will always be needed to automatically upload the most recent successful build to conda channels. This token should allow you to atleast read and write using the API.

github.token

You can create a token here: https://github.com/settings/tokens

This will always be needed as your recipe and feedstock will be a github repo. This token will allow conda-smithy to automatically create a new repository and register it under your username.

travis.token

You can create a token here: https://travis-ci.org/account/preferences

This will be needed if you decided to use travis-ci for your automated builds.

circle.token

You can create a token here: https://circleci.com/account/api

This will be needed if you decided to use circle-ci for your automated builds. This is the CI system I ended up using for my setup.

appveyor.token

You can create a token here: https://ci.appveyor.com/api-token

This will be needed if you decided to use appveyor for your automated builds.

azure.token

You can create a token here: https://dev.azure.com/yourusename/\_usersSettings/tokens

This will be needed if you decided to use azure for your automated builds. This is purely optional and not included in the original instructions in the link at top. Looks like conda-forge now uses azure pipelines extensively. I did not use this token.

2. Setup conda-smithy

We will build a separate environment to work with conda-smithy

> conda create -n smithyenv
> conda activate smithyenv
> conda install -c conda-forge conda-smithy

3. First time setup

We will now setup the github repo contents for the first time. Let me use the "isce2" recipes from "isce-framework/recipes-test" repo as an example.

> tree isce2
isce2/
├── build.sh
├── conda_build_config.yaml
├── meta.yaml
├── README.md
└── scripts
    ├── activate.sh
    └── deactivate.sh

a. Initialize the repository

I first copied the "isce2" subfolder from "recipes-test" to a work area and then ran

> conda-smithy init isce2

This will create a new folder called isce2-feedstock.

> ls isce2-feedstock
conda-forge.yml recipe

b. Register the repo automatically with github

> conda smithy register-github --user yourgithubusername ./isce2-feedstock

This should have setup a new blank github repository named isce2-feedstock under the user name that your provided. If you want to setup the repository under an organization, use --organization option instead.

c. Update conda-forge.yml

We will include the github and anaconda information in conda-forge.yml file.

> cat isce2-feedstock/conda-forge.yml
github:
    user_or_org: yourgithubusername
channels:
    targets:
        - ["yourcondachannelname", "main"]

There maybe other entries that could be useful here like max_py_ver and compiler_stack. Refer to feedstocks of popular channels like gdal, fftw or opencv to see if they are relevant for your repo.

Definitely add a section on providers in your conda-forge.yml. In my case, I only used circle CI and I was only interested in building linux binaries.

provider:
    linux: circle
    osx: None
    win: None
    linux_ppc64le: None
    linux_aarch64: None

d. Update conda_build_config.yaml

We will also need to include the anaconda target channel in the recipe sub-folder of the feedstock.

> cat isce2-feedstock/recipe/conda_build_config.yaml
channel_targets:
    - yourcondachannelname main

This is also a place to set software versions that you may want to use. For more details, see -https://medium.com/@MaheshSawaiker/conda-multi-variant-builds-8edc35c215d7

For my example, I decided to freeze the Pytho7

python:
    - 3.7
hdf5:
    - 1.10.4

e. Register with CI services

We will need to now register the repo with the CI system that we intend to use. In this example, we only intend to use circleCI and hence, we will explicitly provide flags on the command line to skip other systems.

> conda smithy register-ci --user yourgithubusername --feedstock_directory ./isce2-feedstock --without-travis --without-azure --without-appveyor

When you login to your circle CI page, you should now see an active project associated with the isce2-feedstock repo.

f. Auto-generate CI related files

conda smithy rerender --feedstock_directory ./isce2-feedstock

This will generate CI related files in the isce2-feedstock folder. Since, we are not really interested in travis/ appveyor or azure - we only look at .circleci and .ci_support folders.

• .circleci folder contains boiler plate code and would not need any changes.
• .ci_support/linux_.yaml should contain the settings for different package versions that will be used to build your conda package.
• This step created some issues, as it was trying to set up azure pipelines even though I had explicitly disabled it. I just commented out the section in configure_feedstock.py script in conda-smithy package.
• Don't delete the files for services that you are not going to use. Leave them alone. You may end up using them in the future.

g. Test locally with docker

You are now ready to test your recipe setup with docker.

> cd isce2-feedstock 
> ./.circleci/run_docker_build.sh

This essentially replicates the process used by CI systems to build your recipe and upload the package to anaconda.org. Fix any issues with the build.

h. Commit to github

Once you have tested everything locally, commit your repository contents to github. This should automatically trigger a circle CI job. Any subsequent commit to the repository will trigger a new build and upload to your anaconda channel.

4. Maintaining the feedstock

1. If you are rebuilding the package for the same version, bump up the build number in meta.yaml
2. For new versions, update the URL and SHA256 hash; and commit to rebuild the new binaries

Nice! Update to include conda smithy init <recipe folder> perhaps?