Mostly-BSD/Python Development on FreeBSD.MD

## Python Development on FreeBSD.MD

      
    Raw
  

              Python Development on FreeBSD.MD
            
          
    Pythonic way to setup Python Development Environment on FreeBSD

This gist lists steps needed to setup python development environment on FreeBSD using isolated virtual environments.
Why not use pkg or ports?


Pythonic way is to create virtual enviroments for projects and add libs to these venvs not system python.
Binary packages available only for one python version. Currently this is 3.8.
Although we can build python packages against another version using ports, it's still not the Pythonic way.

Overview


Install python 3.9 (latest stable) using pkg
Install pip using get-pip.py
Install pipx using pip
Install poetry using pipx
Create and Manage python virtual environments using poetry

NOTE:
Both pipx and poetry provide isolated environments for python projects with different objectives. pipx is used to install python-based CLI/TUI tools in isolated env, while poetry is used to created isolated python environments for coding/development/testing etc.
Here pipx is used to install poetry in an isolated env. and from there on we use poetry to create/manage our virtual enviromentns for coding/development etc. Instead you can simply install poetry directly using pip w/o installing pipx.
Important Locations


System Python binary - /usr/local/bin/python3.9
System Python libraries - /usr/local/lib/python3.9
User Python binaries - $HOME/.local/bin
User Python libraries - $HOME/.local/lib/python3.9/site-packages
pipx installed binaries - $HOME/.local/bin
pipx installed virtual environments - $HOME/.local/pipx/venvs
poetry virtual environments - $HOME/.cache/pypoetry/virtualenvs


## setup-FreeBSD-Python-Dev.sh
#!/usr/bin/env bash

# Step 1: Install python 3.9 plus other reqd packages
sudo pkg install cmake gcc10 openblas blas libzmq4 python39 py39-sqlite3 py39-cython py39-numpy openjpeg libxml2 libxslt curl
python3.9 -VV # Make sure it works

# Step 2: Setup Variables
 # Use $HOME/TMP coz /tmp is mounted with 'noexec' option
mkdir -p $HOME/TMP
export TMPDIR=$HOME/TMP
# By default /usr/local/include are not in compiler's and linker's search path
export CFLAGS="$CFLAGS -I/usr/local/include/openjpeg-2.4 -I/usr/local/include"
export CXXFLAGS="$CXXFLAGS -I/usr/local/include/openjpeg-2.4 -I/usr/local/include"
export LDFLAGS="$LDFLAGS -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lomp"
export ZMQ_HOME="/usr/local" # Force pyzmq to use system zeromq.

# Step 3: Install pip in user's home directory
curl -sSL https://bootstrap.pypa.io/get-pip.py -o $HOME/get-pip.py
python3.9 $HOME/get-pip.py --user

# Step 4: Install pipx using pip in user's home directory
pip install -U pipx
pipx --version
pipx ensurepath # Add pipx to your $PATH
pipx completions # Setup pipx shell completions if reqd.

# Step 5: Install poetry using pipx in user's home directory
pipx install poetry
poetry about # Make sure it works
poetry completions --help # Setup poetry shell completions if reqd.

# Create a new venv using poetry
cd $HOME
poetry new myvenv
cd myvenv

# Add development dependencies
poetry add -D mypy flake8 black
poetry add -D python-lsp-server -E all

# Add Jupyter Notebook
poetry add -D jupyter jupyterlab

# Add Jupyter Interactive Widgets if needed
poetry add -D ipywidgets
poetry run jupyter nbextension enable --py widgetsnbextension --sys-prefix

# Add Contrib Extensions if needed
poetry add -D jupyter-contrib-nbextensions jupyter_nbextensions_configurator
poetry run jupyter contrib nbextension install --sys-prefix
poetry run jupyter nbextensions_configurator enable --sys-prefix
poetry run jupyter nbextension enable execute_time/ExecuteTime --sys-prefix

# Core Data Science
poetry add numpy pandas numexpr bottleneck

# Visualization
poetry add matplotlib seaborn graphviz

# Stats/ML
poetry add scipy scikit-learn statsmodels
poetry add xgboost pygam lightgbm

# Probabilistic Programming (Bayessian)
poetry add pymc3
#poetry add pystan # Currently doesn't install

# Deep Learning
#poetry add tensorflow keras # Currenly doesn't install
#poetry add pytorch pyro numpyro # Currently doesn't install

poetry check # Final Check

poetry run jupyter-notebook
	#!/usr/bin/env bash

	# Step 1: Install python 3.9 plus other reqd packages
	sudo pkg install cmake gcc10 openblas blas libzmq4 python39 py39-sqlite3 py39-cython py39-numpy openjpeg libxml2 libxslt curl
	python3.9 -VV # Make sure it works

	# Step 2: Setup Variables
	# Use $HOME/TMP coz /tmp is mounted with 'noexec' option
	mkdir -p $HOME/TMP
	export TMPDIR=$HOME/TMP
	# By default /usr/local/include are not in compiler's and linker's search path
	export CFLAGS="$CFLAGS -I/usr/local/include/openjpeg-2.4 -I/usr/local/include"
	export CXXFLAGS="$CXXFLAGS -I/usr/local/include/openjpeg-2.4 -I/usr/local/include"
	export LDFLAGS="$LDFLAGS -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lomp"
	export ZMQ_HOME="/usr/local" # Force pyzmq to use system zeromq.

	# Step 3: Install pip in user's home directory
	curl -sSL https://bootstrap.pypa.io/get-pip.py -o $HOME/get-pip.py
	python3.9 $HOME/get-pip.py --user

	# Step 4: Install pipx using pip in user's home directory
	pip install -U pipx
	pipx --version
	pipx ensurepath # Add pipx to your $PATH
	pipx completions # Setup pipx shell completions if reqd.

	# Step 5: Install poetry using pipx in user's home directory
	pipx install poetry
	poetry about # Make sure it works
	poetry completions --help # Setup poetry shell completions if reqd.

	# Create a new venv using poetry
	cd $HOME
	poetry new myvenv
	cd myvenv

	# Add development dependencies
	poetry add -D mypy flake8 black
	poetry add -D python-lsp-server -E all

	# Add Jupyter Notebook
	poetry add -D jupyter jupyterlab

	# Add Jupyter Interactive Widgets if needed
	poetry add -D ipywidgets
	poetry run jupyter nbextension enable --py widgetsnbextension --sys-prefix

	# Add Contrib Extensions if needed
	poetry add -D jupyter-contrib-nbextensions jupyter_nbextensions_configurator
	poetry run jupyter contrib nbextension install --sys-prefix
	poetry run jupyter nbextensions_configurator enable --sys-prefix
	poetry run jupyter nbextension enable execute_time/ExecuteTime --sys-prefix

	# Core Data Science
	poetry add numpy pandas numexpr bottleneck

	# Visualization
	poetry add matplotlib seaborn graphviz

	# Stats/ML
	poetry add scipy scikit-learn statsmodels
	poetry add xgboost pygam lightgbm

	# Probabilistic Programming (Bayessian)
	poetry add pymc3
	#poetry add pystan # Currently doesn't install

	# Deep Learning
	#poetry add tensorflow keras # Currenly doesn't install
	#poetry add pytorch pyro numpyro # Currently doesn't install

	poetry check # Final Check

	poetry run jupyter-notebook