Skip to content

Instantly share code, notes, and snippets.

@pansapiens

pansapiens/custom_environments_in_jupyter.ipynb

Last active July 21, 2020 01:53
Show Gist options
  • Save pansapiens/113a89838ff3e251ae8c7ceb89d122dc to your computer and use it in GitHub Desktop.
Save pansapiens/113a89838ff3e251ae8c7ceb89d122dc to your computer and use it in GitHub Desktop.
Download ZIP
Using custom environments in Jupyter
Display the source blob
Display the rendered blob
Raw
custom_environments_in_jupyter.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Installing Python packages into your /home for Jupyter"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The awkward way\n",
"\n",
"Skip to the alternative way below .. this method is inferior."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"import os\n",
"import site"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.\n",
"Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.\n",
"To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.\n",
"Requirement already up-to-date: pip in /persistent/home/andrewperry/.local/lib/python3.5/site-packages (20.1)\n"
]
}
],
"source": [
"# The --user flag installs stuff in ~/.local so you don't need sudo\n",
"\n",
"!pip install --user --upgrade pip"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.\n",
"Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.\n",
"To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.\n",
"Requirement already satisfied: plotnine in /persistent/home/andrewperry/.local/lib/python3.5/site-packages (0.5.1)\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.5/dist-packages (0.24.2)\n",
"Requirement already satisfied: numpy in /usr/local/lib/python3.5/dist-packages (from plotnine) (1.16.3)\n",
"Requirement already satisfied: matplotlib>=3.0.0 in /usr/local/lib/python3.5/dist-packages (from plotnine) (3.0.3)\n",
"Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.5/dist-packages (from plotnine) (1.3.0)\n",
"Requirement already satisfied: statsmodels>=0.8.0 in /persistent/home/andrewperry/.local/lib/python3.5/site-packages (from plotnine) (0.11.1)\n",
"Requirement already satisfied: patsy>=0.4.1 in /persistent/home/andrewperry/.local/lib/python3.5/site-packages (from plotnine) (0.5.1)\n",
"Requirement already satisfied: descartes>=1.1.0 in /persistent/home/andrewperry/.local/lib/python3.5/site-packages (from plotnine) (1.1.0)\n",
"Requirement already satisfied: mizani>=0.5.2 in /persistent/home/andrewperry/.local/lib/python3.5/site-packages (from plotnine) (0.5.4)\n",
"Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.5/dist-packages (from pandas) (2019.1)\n",
"Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.5/dist-packages (from pandas) (2.8.0)\n",
"Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.5/dist-packages (from matplotlib>=3.0.0->plotnine) (2.4.0)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.5/dist-packages (from matplotlib>=3.0.0->plotnine) (1.1.0)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.5/dist-packages (from matplotlib>=3.0.0->plotnine) (0.10.0)\n",
"Requirement already satisfied: six in /usr/local/lib/python3.5/dist-packages (from patsy>=0.4.1->plotnine) (1.12.0)\n",
"Requirement already satisfied: palettable in /persistent/home/andrewperry/.local/lib/python3.5/site-packages (from mizani>=0.5.2->plotnine) (3.3.0)\n",
"Requirement already satisfied: setuptools in /usr/local/lib/python3.5/dist-packages (from kiwisolver>=1.0.1->matplotlib>=3.0.0->plotnine) (41.0.1)\n"
]
}
],
"source": [
"!pip install --user plotnine pandas"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"descartes\t\t patsy-0.5.1.dist-info\n",
"descartes-1.1.0.dist-info pip\n",
"mizani\t\t\t pip-20.1.dist-info\n",
"mizani-0.5.4.dist-info\t plotnine\n",
"palettable\t\t plotnine-0.5.1.dist-info\n",
"palettable-3.3.0.dist-info statsmodels\n",
"patsy\t\t\t statsmodels-0.11.1.dist-info\n"
]
}
],
"source": [
"!ls /home/andrewperry/.local/lib/python3.5/site-packages"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"local_pkg_path = os.path.join(site.USER_BASE, 'lib', 'python3.5', 'site-packages')\n",
"sys.path.insert(0, local_pkg_path)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"sys.path.append('/usr/local/lib/python3.5/dist-packages')"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"# For Python packages installed in a conda environment you might do\n",
"# sys.path.append(os.path.expanduser('~/miniconda3/envs/myenv/lib/python3.5/site-packages'))"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['/home/andrewperry/.local/lib/python3.5/site-packages',\n",
" '/mnt/software/apps/jupyter-1.1.3/lib/python35.zip',\n",
" '/mnt/software/apps/jupyter-1.1.3/lib/python3.5',\n",
" '/mnt/software/apps/jupyter-1.1.3/lib/python3.5/plat-x86_64-linux-gnu',\n",
" '/mnt/software/apps/jupyter-1.1.3/lib/python3.5/lib-dynload',\n",
" '/usr/lib/python3.5',\n",
" '/usr/lib/python3.5/plat-x86_64-linux-gnu',\n",
" '',\n",
" '/mnt/software/apps/jupyter-1.1.3/lib/python3.5/site-packages',\n",
" '/mnt/software/apps/jupyter-1.1.3/lib/python3.5/site-packages/IPython/extensions',\n",
" '/persistent/home/andrewperry/.ipython',\n",
" '/usr/local/lib/python3.5/dist-packages']"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sys.path"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"from plotnine import *"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Alternative method - register a Jupyter kernel inside your environment\n",
"\n",
"*Advantage:* You can keep any environment setup stuff out of your notebook.\n",
"\n",
"*Disadvantage:* The Python environment (eg packages) are available, but the system `PATH` for `!` and `%%bash` shell commands isn't automatically set to the `bin` in your environment.\n",
"\n",
"Do this in a terminal:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create a virtualenv or conda env\n",
"```\n",
"virtualenv -p python3 ~/.virtualenvs/my-custom-env\n",
"\n",
"# _or_\n",
"# conda create -n my-custom-env python=3\n",
"```\n",
"\n",
"### Activate it\n",
"```\n",
"source ~/.virtualenvs/my-custom-env/bin/activate\n",
"\n",
"# _or_\n",
"# conda activate my-custom-env\n",
"```\n",
"\n",
"### Install the ipykernel package (or any other Jupyter kernel), and register it with Jupyter\n",
"```\n",
"pip3 install ipykernel\n",
"python3 -m ipykernel install --name my-custom-python-env --display-name \"Custom Python environment test\" --user\n",
"```\n",
"\n",
"### Install whatever packages you want\n",
"```\n",
"pip3 install plotnine\n",
"```\n",
"\n",
"### In Jupyter, when you create a new notebook, select 'Custom Python environment test' or whatever you called in the 'Select Kernel' dropdown\n",
"### For completeness - if you ever want to deregister the kernel (eg you are deleting the conda env or virtualenv), run:\n",
"```\n",
"jupyter kernelspec uninstall my-custom-python-env\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you need to add environment variables to be set _within_ your running notebook (eg, `PATH`, pointing to the correct version of `R` so that `rpy2` magic `%%R` cells use the correct version),\n",
"you can edit `~/.local/share/jupyter/kernels/{my_custom_kernel}/kernel.json` and add an `env` attribute like:\n",
"\n",
"```json\n",
"{\n",
"...snip...\n",
"\n",
"\"env\": {\"PATH\": \"${HOME}/miniconda3/envs/my-custom-env/bin/:${PATH}\"}\n",
"\n",
"...snip...\n",
"}\n",
"```\n",
"\n",
"More on the format of the `kernel.json` 'kernelspec' here: https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Other alternative to investigate\n",
"\n",
"https://pypi.org/project/envkernel/\n",
"\n",
"https://github.com/Anaconda-Platform/nb_conda_kernels\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Irrespective of the method used, the _shell environment_ (inherited from the shell where Jupyter was launched) will be mismatched with the _python environment_. For a more detailed discussion, see: https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/ "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# How I usually do it\n",
"\n",
"- Create a conda environment.\n",
"- Activate it, `conda install jupyterlab` and other dependencies.\n",
"- Run `jupyter lab` from inside the conda env.\n",
"- SSH tunnel to the server where your copy of Jupyter is running.\n",
"\n",
"This means everything can be preserved in a conda environment (eg `conda env export`)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment