Sensible SOP for managing conda envs on MacOS

! Managing conda envs.md

Creating / Managing conda envs

Rules / SOPs:

1. Don't work in base environment
2. Use mamba as a solver
3. Create default environments with pinned packages, and clone / update the default envs as needed

• Creating / Managing conda envs
  • Create a new env
    • Step-by-step walkthrough
    • New env in one command chunk
  • Clone an environment

Create a new env

Step-by-step walkthrough

1. Create env

   CONDA_SUBDIR=osx-arm64 conda create -n <env_name> python=3.9

   Allowed CONDA_SUBDIR architectures include: linux-64, linux-aarch64, linux-ppc64le, osx-64, osx-arm64, win-32, win-64

2. Activate env

   conda activate <env_name>

3. Set architecture Allowed CONDA_SUBDIR architectures include: linux-64, linux-aarch64, linux-ppc64le, osx-64, osx-arm64, win-32, win-64

   conda env config vars set CONDA_SUBDIR=osx-arm64 --name <env_name>

4. Set channels

   # add to top of list
   conda config --prepend channels conda-forge
   # add to end of list
   conda config --append channels intel

5. Pin package versions (if known in advance)

   touch $CONDA_PREFIX/conda-meta/pinned
   cat << EOF >> $CONDA_PREFIX/conda-meta/pinned
   python 3.9.*
   numpy 1.24.*
   scikit-learn 1.0.*
   EOF

6. Install packages

   • manually

     mamba install -n <env_name> pandas scipy scikit-learn statsmodels jupyterlab

   • from environment.yaml

     mamba env update -n <env_name> -f /path/to/environment.yaml

   see sample environment.yaml below

New env in one command chunk

Create env function (see conda_env_create.sh below) and load:

zsh ./conda_env_create.sh

Assuming environment.yaml exists with the CONDA_SUBDIR set:

env_name=...
conda_env_create "$env_name" python=3.9
# env is created and activated

# pin dependencies
touch $CONDA_PREFIX/conda-meta/pinned
cat << EOF >> $CONDA_PREFIX/conda-meta/pinned
python 3.9.*
numpy 1.21.*
EOF

# install packages
mamba env update -n "$env_name" -f /path/to/environment.yaml

# install pip packages
pip install --upgrade-strategy only-if-needed -r /path/to/requirements.txt
pip install --upgrade-strategy only-if-needed --index-url=https://<private_pip_url> -r /path/to/private-requirements.txt

unset env_name

Clone an environment

conda create --name clone_env --clone source_env

conda_env_create.sh

#!/usr/bin/env zsh

conda_env_create () {

  show_help () {
    cat << EOF
usage: conda_env_create [-h] [-n ENVIRONMENT] [-f FILE | package1, package2, ...]

Create an environment based on an environment file or list of packages

Options:
required arguments:
  -n ENVIRONMENT, --name ENVIRONMENT
                        Name of environment.

positional arguments:
  packages to install

optional arguments:
  -h, --help            Show this help message and exit.
  -f FILE, --file FILE  environment definition file (default: environment.yml)
EOF
  }

  ### ref: http://mywiki.wooledge.org/BashFAQ/035
  # Initialize all the option variables.
  # This ensures we are not contaminated by variables from the environment.
  env_name=
  file=
  package_list=()

  # parse options
  while [[ $# -gt 0 ]]; do
    case $1 in
      -h|-\?|--help)
          # Display a usage synopsis.
          show_help
          return
          ;;
      -n|--name)
          # Require option argument has been specified.
          if [ "$2" ]; then
              env_name=$2
              shift
          else
              echo 'ERROR: "--name" requires a non-empty option argument.'
              return 1
          fi
          ;;
      --name=?*)
          # Delete everything up to & including "=" and assign the remainder.
          env_name=${1#*=}
          ;;
      --name=)
          # Handle the case of an empty --name=
          echo 'ERROR: "--name" requires a non-empty option argument.'
          return 1
          ;;
      -f|--file)
          # Require option argument has been specified.
          if [ "$2" ]; then
              file=$2
              shift
          else
              echo 'ERROR: "--file" requires a non-empty option argument.'
              return 1
          fi
          ;;
      --file=?*)
          # Delete everything up to & including "=" and assign the remainder.
          file=${1#*=}
          ;;
    echo "Running x86/Rosetta install"
    echo "Running x86/Rosetta install"
    conda_arch="osx-64"

    # ensure env starts with "x86"
    if [[ ! "$env_name" =~ "^x86_" ]]; then
      env_name="x86_$env_name"
      echo "--- NOTE: Prepending 'x86_' to environment name ---"
    fi
  fi

  echo "Creating new conda env: '$env_name' ..."
  # [[ -z $file ]] || echo "from file $file "
  # [[ -z $package_list ]] || echo "with specified packages: $package_list"

  # run conda commands for setup
  if [[ "$(uname -p)" = "i386" ]]; then
    CONDA_SUBDIR="$conda_arch" conda create --name "$env_name" --quiet --yes && conda activate "$env_name"
    conda env config vars set CONDA_SUBDIR="$conda_arch" --name "$env_name"
    unset conda_arch
  else
    conda create --name "$env_name" --quiet --yes
  fi

  conda deactivate && conda activate "$env_name"  # not technically needed, but can prevent env overwrites
  conda config --prepend channels conda-forge

  # update env using specified packages
  if [[ ${#package_list[@]} -gt 0 ]]; then
    echo "... with specified packages: $package_list"
    mamba install --force-reinstall --name "$env_name" "${package_list[@]}" --quiet --yes
  fi

  # update env using file
  if [[ ! -z $file ]]; then
    if [[ ! -f $file ]]; then
      echo "ERROR: $file not found."
      return 1
    else
      echo "... from file $file "
      mamba env update --name "$env_name" --file "$file" --quiet
    fi
  fi

  # conda config --prepend channels conda-forge > /dev/null 2>&1
  echo "Environment ${env_name} created for $(conda config --env --show | grep 'subdir:') architecture"
  unset env_name
  unset file
  unset package_list

  # return
}

environment.yaml

# name: env_name
channels:
    - conda-forge
dependencies:
    - python=3.9
    - pip
    - cython
    - ipykernel
    - ipython
    # major dependencies
    - numpy
    - pandas
    - scipy
    - scikit-learn
    - sklearn-pandas
    - statsmodels
    # supporting packages
    # - colorama  # or rich
    - jupyterlab
    - matplotlib
    - plotnine
    - seaborn
    - pyyaml
    # linting
    - black
    - flake8
    - flake8-bugbear
    - flake8-builtins
    - flake8-colors
    - flake8-comprehensions
    - flake8-isort
    - isort
    - pep8-naming