Skip to content

Instantly share code, notes, and snippets.

@imrehg
Last active August 11, 2020 13:27
Show Gist options
  • Save imrehg/5c1aacebde15a2cddca010554a7ebdc6 to your computer and use it in GitHub Desktop.
Save imrehg/5c1aacebde15a2cddca010554a7ebdc6 to your computer and use it in GitHub Desktop.
Custom Python environment

Setting up a custom Python environments with Conda on Faculty Platform

The following scripts provide a foundation to use a different Python version on the the Faculty Platform servers than what's shipped in the default Python2 and Python3 Conda environments (Python 2.7.16 and 3.6.10 respectively, at the time of writing). These scripts are intended to be a quick fix while the Platform implements support for newer Python versions, and thus there are quirks of behaviour to take into account, as explained in this documentation.

There are 3 scripts attached here:

  • 01_plain_environment.sh - this is a barebones environment, to be minimal and quick to start up
  • 02_default_packages.sh - an environment that installs the default packages found on the platform platform
  • 03_gpu_default_packages.sh - this environment is the same as the previous one, with the addition of installing the GPU-enabled versions of the Tensorflow library (versions the same as on the platform)

Environment configuration

Create a Faculty environment for your custom Python setup. In the environment's Script section, copy in the whole content of the relevant script that you have chosen.

To adjust the behaviour of the script, edit the variables present at the top of the script:

PYTHON_VERSION sets the version of Python to be installed. Not all Python libraries support all Python versions, and thus not all Python version / script combinations have been tested. Please do your own testing whether any given library functions properly with the Python version you've chosen!

CONDA_ENV is the name of the new Conda environment (not to confuse with Faculty environment), where the new Python version is set up. Faculty uses Conda to create isolated Python environments in the platform servers, which can differ in their Python version, their installed packages, etc. The Conda environment name is used to refer to these. By default the Platform currently ships with two environments, Python2 and Python3, while this script will create a third environment to use, called Custom-3.7 in its current form. You can change this value if you prefer to something more natural for your use case.

If you want to specify extra packages to be installed in the given environment, the scripts have marked sections where those modifications (specifying package list for conda install and pip install) can be made.

Note that with these scripts, you cannot use the Faculty environments' "Python" section to define your Python packages (with Pip and Conda), as they apply to the Faculty-provided Python environments only. You need to use the scripts' variables as described above to set your Python package dependencies!

Using the environments

For interactive servers, we recommend using the 02_default_packages.sh environment. It takes a bit longer time to start up because of the installation steps, but results in an server that closely follows the setup of the Faculty-provided Python environment. This script also automatically sets up automatic activation of this new environment in the terminal of interactive servers.

When using Jupyter, you have to manually select this new environment from the list of your kernels to use:

In jobs, you will have to manually activate the environment in your job script before being able to use. Thus you might have to wrap those Python scripts in a small shell script to activate the Conda environment before running the Python calls, for example:

#!/bin/bash

conda activate Custom-3.7

<... rest of the script ...>
#!/bin/bash
###
# Adjustable configuration
###
# The Python version to install (such as 3, or 3.8, or 3.8.1, for example)
PYTHON_VERSION=3.7
# The name of the new kernel in Jupyter
# Kernel names can only contain ASCII letters and numbers,
# and these separators: - . _ (hyphen, period, and underscore).
CONDA_ENV="Custom-${PYTHON_VERSION}"
###
# End of configuration
###
# Stop on any error
set -o errexit
/opt/anaconda/bin/conda create -n "${CONDA_ENV}" -y python=${PYTHON_VERSION}
# If installing any packages with Conda, uncomment the following line and
# replace PACKAGE1, ... with the package names
# /opt/anaconda/bin/conda install -n "${CONDA_ENV}" -y PACKAGE1 PACKAGE2
/opt/anaconda/envs/${CONDA_ENV}/bin/pip install --upgrade pip
# If installing any packages with pip, uncomment the following line and
# replace PACKAGEA, ... with the package names
# /opt/anaconda/envs/${CONDA_ENV}/bin/pip install PACKAGEA PACKAGEB
# Export relevant environment variables
# https://docs.faculty.ai/how_to/environment_variables.html#setting-environment-variables-through-faculty-environments
SHARED_ENV_SCRIPT="/etc/faculty_environment.d/conda.sh"
echo "export CONDA_ENV=${CONDA_ENV}" >> "${SHARED_ENV_SCRIPT}"
# Activate Virtualenv in interactive shells automatically
sudo tee /etc/profile.d/zz-path.sh <<'END'
if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ] && \
[ "$FACULTY_IS_INTERACTIVE" = "1" ]; then
conda activate "${CONDA_ENV}"
fi
END
conda activate ${CONDA_ENV}
echo "Python version:"
python -V
#!/bin/bash
###
# Adjustable configuration
###
# The Python version to install (such as 3, or 3.8, or 3.8.1, for example)
PYTHON_VERSION=3.7
# The name of the new kernel in Jupyter
# Kernel names can only contain ASCII letters and numbers,
# and these separators: - . _ (hyphen, period, and underscore).
CONDA_ENV="Custom-${PYTHON_VERSION}"
###
# End of configuration
###
# Stop on any error
set -o errexit
/opt/anaconda/bin/conda create -n "${CONDA_ENV}" -y python=${PYTHON_VERSION}
# Install pacakages that are installed by default in the Faculty user images
/opt/anaconda/bin/conda install -n "${CONDA_ENV}" -y \
anaconda \
psycopg2 \
plotly \
flask \
'gunicorn==19.9.0' \
gevent \
greenlet \
nb_conda_kernels==2.1.1
# Add extra Conda packages here, don't forget the \ at the end of the line for newlines
# pip-installed dependencies
/opt/anaconda/envs/${CONDA_ENV}/bin/pip install --upgrade pip
/opt/anaconda/envs/${CONDA_ENV}/bin/pip install \
ipykernel \
boto3 \
google-cloud-storage \
google-cloud-bigquery \
pandas-gbq \
sqlalchemy \
psycopg2 \
mysqlclient \
lens \
tensorflow~=1.14.0 \
graphviz \
nbdime \
sherlockml \
faculty \
mlflow==1.7.2 \
mlflow-faculty==0.5.0 \
faculty-models
# Add extra pip packages here, don't forget the \ at the end of the line for newlines
# Export relevant environment variables
# https://docs.faculty.ai/how_to/environment_variables.html#setting-environment-variables-through-faculty-environments
SHARED_ENV_SCRIPT="/etc/faculty_environment.d/conda.sh"
echo "export CONDA_ENV=${CONDA_ENV}" >> "${SHARED_ENV_SCRIPT}"
# Activate Virtualenv in interactive shells automatically
sudo tee /etc/profile.d/zz-path.sh <<'END'
if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ] && \
[ "$FACULTY_IS_INTERACTIVE" = "1" ]; then
conda activate "${CONDA_ENV}"
fi
END
conda activate ${CONDA_ENV}
echo "Python version:"
python -V
#!/bin/bash
###
# Adjustable configuration
###
# The Python version to install (such as 3, or 3.8, or 3.8.1, for example)
PYTHON_VERSION=3.7
# The name of the new kernel in Jupyter
# Kernel names can only contain ASCII letters and numbers,
# and these separators: - . _ (hyphen, period, and underscore).
CONDA_ENV="Custom-${PYTHON_VERSION}"
###
# End of configuration
###
# Stop on any error
set -o errexit
/opt/anaconda/bin/conda create -n "${CONDA_ENV}" -y python=${PYTHON_VERSION}
# Install pacakages that are installed by default in the Faculty user images
/opt/anaconda/bin/conda install -n "${CONDA_ENV}" -y \
anaconda \
psycopg2 \
plotly \
flask \
'gunicorn==19.9.0' \
gevent \
greenlet \
nb_conda_kernels==2.1.1 \
cudatoolkit
# Add extra Conda packages here, don't forget the \ at the end of the line for newlines
# pip-installed dependencies
/opt/anaconda/envs/${CONDA_ENV}/bin/pip install --upgrade pip
/opt/anaconda/envs/${CONDA_ENV}/bin/pip install \
ipykernel \
boto3 \
google-cloud-storage \
google-cloud-bigquery \
pandas-gbq \
sqlalchemy \
psycopg2 \
mysqlclient \
lens \
tensorflow-gpu~=1.14.0 \
graphviz \
nbdime \
sherlockml \
faculty \
mlflow==1.7.2 \
mlflow-faculty==0.5.0 \
faculty-models
# Add extra pip packages here, don't forget the \ at the end of the line for newlines
# Export relevant environment variables
# https://docs.faculty.ai/how_to/environment_variables.html#setting-environment-variables-through-faculty-environments
SHARED_ENV_SCRIPT="/etc/faculty_environment.d/conda.sh"
echo "export CONDA_ENV=${CONDA_ENV}" >> "${SHARED_ENV_SCRIPT}"
# Activate Virtualenv in interactive shells automatically
sudo tee /etc/profile.d/zz-path.sh <<'END'
if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ] && \
[ "$FACULTY_IS_INTERACTIVE" = "1" ]; then
conda activate "${CONDA_ENV}"
fi
END
conda activate ${CONDA_ENV}
echo "Python version:"
python -V
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment