Bootstrap miniconda Jupyter development environment on Windows

conda_setup.md

Bootstrap a Miniconda/Anaconda Environment

Bootstrap a *conda development environment for jupyter lab development using multiple environments and kernels under windows 11.

This will allow the use of multiple kernels from a single Jupyter Lab session. Each notebook or project can run within a specific python virtual environment by using the kernel drop-down menu within the notebook. This helps avoid dependency conflicts between various libraries and allows multiple different python, ane even R versions symultaniously.

This is particularly useful if you:

• Need to run an older verison of Python for legacy projects
• Need to use conflicting versions of librarys such as PIL and pillow
• Want to avoid the XKCD Python Environment hell

Requirements

• Miniconda installer or Anaconda Installer
• Multiple Conda environments each with ipykernel installed in each

Initial Setup

1. Download and install *conda using the suggested settings
2. Locate and launch the PowerShell Anaconda Prompt shortcut in the start menu
   • For Minicoda: Install jupyterlab in the in the base, default environment: conda install -c conda-forge jupyterlab
   • This is not required if Anaconda was installed
3. Install the conda nb_conda_kernels to allow access to other environments and kernels: conda install -c conda-forge nb_conda_kernels

Creating Independent Conda Environments

From any existing (base) conda environment, run the following command do the following to to create a new project environment with ipykernel installed and to make a kernel for YOUR_PROJECT_NAME availabile from within Jupyter Lab.

1. Create a conda environment: conda create -n YOUR_PROJECT_NAME ipykernel
2. Create a working directory for code (optional, but recommended): mkdir YOUR_PROJECT_NAME

Installing Packages in an Environment

Each environment can have independent packages. This can help prevent conflicts and allow pinning against a particular library version as necessary.

1. Activate an environment: conda activate YOUR_ENV_NAME
   • TIP: list envornments with conda env list
2. Install packages with conda: conda install PACKAGE
3. Install packages with pip: pip install PACKAGE

Using a conda Environment/Kernel with Jupyter

It is possible to access multiple conda environments and kernels at the same time from within Jupyter. Jupyter's file manager will set its root to be the working directory where Jupyter Lab is launched.

To take advantage of this feature, it is best to launch Jupyter Lab from the directory one level above the project dirs. In the example below this is done from C:\Users\MPalin\Documents\src

C:\
└── 📁 Users
    └── 📁 MPalin
        └── 📁 Documents
            └── 📁 src ← **HERE**
                ├── 📁 FooProject
                ├── 📁 SpamHam
                └── 📁 Deep_Learning

1. Launch a conda shell (base)
2. Change to the appropriate working directory: cd C:\Users\MPalin\Documents\src
3. Launch Jupyter Lab: jupyter lab
4. From within the Jupyter Lab file manager, navigate to the project directory
   • This is optional; the kernels are available in all contexts, but it does keep notebooks more organized
5. Create a new notebook by clicking on the appropriate kernel
   • This can be changed later using the dropdown menu in the upper right corner of the notebook window

Using Conda Enviroments with VS Code & Jupyter

VS Code can natively run Jupyter Kernels and function as a Jupyter environment.

Requirements

1. *Conda environment configured as described above
2. VS Code
3. VS Code Jupyter extension

Getting Started

1. Create at least one Conda environment
2. Launch VS Code and navigate to a project directory
3. Open or create a new .ipynb file
4. In the upper right corner of the interface, look for Detecting Kernels or Select Kernel and click on the dialogue
5. From the pallet choose Python Environments...
6. Select the appropriate kernel/environment for this project

Automatically Convert Notebooks to Pure Python

One disadvantage of working in notebooks is that they are less convenient to run on the command line and it is more challenging to use them as relative imports within a project. Jupytext solves this problem by automagically converting .ipynb to .py and allowing .py files to be opened as if they were native Jupyter notebooks.

For more information see Jupytext.

Note 0: This has only been tested running Jupyter Lab from the command line, not from within the Anaconda launcher.

Note 1: Occasionally Jupytext will horribly corrupt both .py and .ipynb files. It is strongly recommended to use a version control system to avoid tragedy.

To enable this magic, follow the steps below.

1. Install Jupytext in the environment where jupyter lab is typically executed
2. Install Jupytext in the working environment: conda install -c conda-forge -n MY_ENV_NAME jupytext
3. Create a jupytext.toml in the root of the project directory containing the text below. This will link .ipynb:.py files such that any changes made through in one will reflect in the other. See Note 2 Below for existing files.
4. If Jupyter Lab is running, restart it.
5. Any edits made in existing .ipynb will be automatically convered into .py files. .py files will be opened as notebooks (in most cases).

# jupytext.toml minimal sample
formats = "ipynb,py:light"

Note 2: Existing .py and .ipynb files may need to be manually paired. This can be done by using the command pallet in Jupyter Lab (ctrl+shift+c) and searching for pair. In this example, it makes sens to pair with light script.