Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save jmansour/c42c203f0596f6bed6138a440818516a to your computer and use it in GitHub Desktop.
Save jmansour/c42c203f0596f6bed6138a440818516a to your computer and use it in GitHub Desktop.
Jupyter kernel via containerised Python environment
  1. chmod docker-kernel.py to make it executable (typically chmod +x docker-kernel.py).
  2. Place docker-kernel.py somewhere in your PATH. Or alternatively, modify kernel.json to use an absolute path to docker-kernel.py. The latter (absolute path) option will be necessary in cases where Jupyter isn't exposed to your PATH environment variable.
  3. If you wish to hard code the docker image to be used, modify kernel.json such that the third parameter of argv (currently set to "IMAGE_FROM_ENV") points to the required image. The default setting ("IMAGE_FROM_ENV") is a special token which directs the script to instead use a user specified environment variable (JUPYTER_KERNEL_DOCKER_IMAGE) for image specification. Note that any image you wish to use must have a compatible ipykernel_launcher module installed.
  4. If you wish to hard code extra docker run parameters, modify kernel.json such that the fourth parameter of argv (currently set to "MAP_CWD") specifies the parameters (as a string). Alternatively, you can at any time add parameters by setting the JUPYTER_DOCKER_EXTRA_PARAMETERS environment variable. In particular, you will specify any volume mapping requirements via this option. The default setting ("MAP_CWD") is a special token which directs the script to map the current working directory (usually where Jupyter was launched from) to a directory within the container. Parameters defined by environment variable take precedence over other options.
  5. Move kernel.json into some new directory (for example some_kernel_directory). The name doesn't really matter as it is only used for Jupyter's internal purposes. If you wish to change the actual kernel name displayed within Jupyter, your will need to manually modify the display_name parameter within kernel.json.
  6. Execute jupyter kernelspec install --user some_kernel_directory to install the kernel.
  7. Launch jupyter notebook.
#!/usr/bin/env python
import sys
import subprocess
import json
import os
# map from connection file to ipykernel_launcher args
argmap = {
"stdin_port" : "--stdin",
"ip" : "--ip",
"control_port" : "--control",
"hb_port" : "--hb",
"signature_scheme" : "--Session.signature_scheme",
"key" : "--Session.key",
"shell_port" : "--shell",
"transport" : "--transport",
"iopub_port" : "--iopub"
}
container_vals = {
"stdin_port" : 6000,
"control_port" : 6001,
"hb_port" : 6002,
"shell_port" : 6003,
"iopub_port" : 6004,
"ip" : "0.0.0.0"
}
# get connection file
connection_file = sys.argv[1]
image_name = sys.argv[2]
if image_name == "IMAGE_FROM_ENV":
IMAGE_KEY = "JUPYTER_KERNEL_DOCKER_IMAGE"
if IMAGE_KEY in os.environ:
image_name = os.environ[IMAGE_KEY]
else:
raise RuntimeError("You must set the 'JUPYTER_KERNEL_DOCKER_IMAGE' environment variable to specify your image.")
extra_parameters = sys.argv[3]
if extra_parameters == "MAP_CWD":
tmpdir = "/tmp/kernelmapdir"
cwd = os.getcwd()
extra_parameters = "-w {} -v {}:{}".format(tmpdir, cwd, tmpdir)
PARAMS_KEY = "JUPYTER_DOCKER_EXTRA_PARAMETERS"
if PARAMS_KEY in os.environ:
extra_parameters = os.environ[PARAMS_KEY]
with open(connection_file, 'r') as fp:
kjson = json.load(fp)
docker_args = ['docker', 'run', '--rm',
'-p', '{}:{}'.format( kjson[ 'stdin_port'] , container_vals[ 'stdin_port'] ),
'-p', '{}:{}'.format( kjson['control_port'] , container_vals['control_port'] ),
'-p', '{}:{}'.format( kjson[ 'hb_port'] , container_vals[ 'hb_port'] ),
'-p', '{}:{}'.format( kjson[ 'shell_port'] , container_vals[ 'shell_port'] ),
'-p', '{}:{}'.format( kjson[ 'iopub_port'] , container_vals[ 'iopub_port'] ),
]
docker_args += [extra_parameters,image_name]
command_line = "python -m ipykernel_launcher".split()
# overwrite json with values required by container
kjson.update(container_vals)
kjson["key"] = "'" + 'b"' + kjson["key"] + '"' + "'" # this guy needs to be described as bytes for python 3 ipykernel_launcher
for key in argmap.keys():
command_line.append( "{}={}".format( argmap[key], kjson[key] ) )
strguy = " ".join(docker_args+command_line)
subprocess.check_call(strguy, shell=True)
{
"display_name": "Docker Kernel",
"language": "python",
"argv": [ "docker-kernel.py", "{connection_file}", "IMAGE_FROM_ENV", "MAP_CWD" ]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment