Python Versions + Virtual Environments Tips and Tricks

Python+VirtualEnvs.md

TOC

• PIP
  • installing from Git Repo
• PIPX
• Managing Different Python Version with pyenv
  • macOS: use with virtualenvwrapper plugin
  • Linux: use with virtualenv
• Cloning a Virtual Env
• How to fix Broken Python Reference in Virtualenvs
• Jupyter Notebooks
  • Authoring/ turning your notebook into blogs

Table of contents generated with markdown-toc

PIP

installing from Git Repo

in general the command is as follows but see this blog post for the most detailed explanation:

pip install git+https://github.com/user/repositor.git@branch#subdirectory=src

• note that the @branch is completely optional
• also #subdirectory=src is also optional (sometimes the project to install is not in the root of the repo, e.g. Streamlit)
• either options like --ignore-requires-python might be useful if you think the python version requirement is not neccessary

PIPX

pipx is a tool to help you install and run end-user applications written in Python. It's roughly similar to macOS's brew, JavaScript's npx, and Linux's apt.

so for things like awscli, black, pipreqs, light-the-torch, or any other cool utilities like this you might wanna install them with pipx

to install pipx:

brew install pipx
pipx ensurepath
sudo pipx ensurepath --global # optional to allow pipx actions with --global argument

Managing Different Python Version with pyenv

installing on OSX is simple as brew install pyenv and on linux this guide will help.

then update your ~/.bashrc accordingly so that pyenv manages your python paths for you:

$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bashrc

Here are some common commands:

pyenv versions        # shows available python versions on your machine
pyenv version         # shows python version being selected
pyenv install -l      # give list of all avail versions
pyenv install 3.10.4  # install Python 3.10.4
pyenv shell 3.10.4    # select just for the current shell
pyenv global 3.8.0    # select globally for your user account

lastly confirm that your python path is pointed to the right version installed by pyenv by doing:

$ which python
$ python -V
$ pip -V

macOS: use with virtualenvwrapper plugin

brew install pyenv-virtualenvwrapper

then update your ~/.bashrc per this post

anytime a new version of Python is installed, we need to install virtualenvwrapper; for example:

pyenv install 3.9.19
pyenv shell 3.9.19
pyenv virtualenvwrapper_lazy

then the ususal virtualenvwrapper commands like mkvirtualenv, workon, and deactivate will be available

Linux: use with virtualenv

you don't really need the pyenv-virtualenv plugin. Here's what to do instead:

pyenv shell 3.8.0
mkvirtualenv my_env_py38      # make the python version explicit in the virtualenv name
workon my_env_py38
pip install -r requirements.txt

And next time before activating my_env_py38, run pyenv shell 3.8.0 first to make sure you are on the right python version

if using with pyenv-virtualenv plugin

pyenv virtualenv 3.10.4 my_env_py310          # this create a virtualenv with Python 3.10.4
pyenv virtualenvs                             # shows list of available virtual environments
pyenv activate my_env_py310                   # to "workon" the previously created virtual env
pyenv deactivate                              # exits the current virtualenv
pyenv uninstall my_env_py310                  # remove the virtualenv

Cloning a Virtual Env

virrtualenvwrapper has cpvirtualenv but is is NOT recommended.

Account to the community, virtualenv-clone is the best:

pip install virtualenv-clone # into source or dummy virtualenv
virtualenv-clone source/ target/

To find your virtualenv's directory, activate or workon your virtualenv, then which python shall reveal!

How to fix Broken Python Reference in Virtualenvs

with some hints from here

When you upgrade Python using Homebrew and then run brew cleanup, the symlinks in the virtualenv point to paths that no longer exist (because Homebrew deleted them).

this creates broken symlinks in your virtualenv:

find ~/.virtualenvs/my_virtual_env/ -type l

this assumes that your have ~/.virtualenvs as your env $WORKON_HOME

the solution is then to remove the broken symlinks and recreate them again using virtualenv:

find ~/.virtualenvs/my_virtual_env/ -type l -delete
virtualenv -p python3 ~/.virtualenvs/my_virtual_env

paying close attention to the -p args as this determines which python version will be used for your environment

Jupyter Notebooks

taking your virtualenv into Jupyter can be another wrinkle for a new data science. If there are something you installed in a virtualenv that you are having trouble importing in your notebook you might wanna checkout this link or this one.

But the TLDR is... make sure you do the following in your virtualenv:

pip install jupyter environment-kernels

Authoring/ turning your notebook into blogs

Authoring from Jupyter Notebook into HTML, PDF, or other formats are super simple using Quarto

Installation is as simple as brew install --cask quarto (see here for more details)

Note that Quarto needs node v18 which depends on macOS 10.15 or higher (see this related stackoverflow post)