Skip to content

Instantly share code, notes, and snippets.

@mehalter
Last active July 30, 2020 19:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mehalter/15103495b460e3a030e8e74f9fbd4a5b to your computer and use it in GitHub Desktop.
Save mehalter/15103495b460e3a030e8e74f9fbd4a5b to your computer and use it in GitHub Desktop.
Small walkthrough of my Julia set up and demonstration of some helper command line utilities. Making and using sandboxed virtual environments, making and using custom system images, and creating virtual environment Jupyter kernels.
# === Julia Utilities: ===
## Set JULIA_HOME environment variable
export JULIA_HOME="${XDG_DATA_HOME}/julia"
## Call base julia installations or run virtual environment if activated
jl() {
if type julia >/dev/null; then
julia ${@}
else
$JULIA_HOME/julia-${1}/bin/julia ${@:2}
fi
}
jlp() {
if type jlpkg >/dev/null; then
jlpkg ${@}
else
$JULIA_DEPOT_PATH/bin/jlpkg-${1} ${@:2}
fi
}
## Activate a virtual environment by name (default="env")
jlactivate() {
[ ! -z "${VIRTUAL_ENV}" ] && deactivate
venv_dir="${1}"
[ -z "${1}" ] && venv_dir="env"
[ ! -e "${venv_dir}" ] && venv_dir="${JULIA_HOME}/environments/${1}"
. ${venv_dir}/bin/activate
}
## Create a new virtual environment by name (default="env")
jlenv() {
if type julia >/dev/null; then
ver=""
venv_dir="${1}"
opts="${2}"
else
ver="${1}"
venv_dir="${2}"
opts="${3}"
fi
[ -z "${venv_dir}" ] && venv_dir="env"
jl ${ver} -e "using VirtualEnv; venv(\"${venv_dir}\"; ${opts})"
}
## Install basic packages in new Julia virtual environment
jlsetup() { jl ${@} -e 'using Pkg; Pkg.add(["PackageCompiler", "IJulia", "LanguageServer", "SymbolServer", "StaticLint", "Revise"]); Pkg.precompile()' && jlpkginstall ${1} && jlkernclean }
## Install jlpkg into virtual environments, don't use for base julia installs
jlpkginstall() {
if type julia >/dev/null; then
ver=""
dest="${VIRTUAL_ENV}"
command="jlpkg"
else
ver="${1}"
dest="${JULIA_DEPOT_PATH}"
command="jlpkg-${ver}"
fi
jl ${ver} -e "using Pkg; Pkg.add(\"jlpkg\"); import jlpkg; jlpkg.install(destdir=\"${dest}/bin\",command=\"${command}\",force=true)"
}
## Create a new environment, activate it, and run the setup alias
jlenvsetup() { jlenv ${@} && jlactivate && jlsetup }
## Create a Jupyter kernel for the current environment
jlkern() {
if [ -z "${VIRTUAL_ENV}" ]; then
ver="${1}"
envname="-Base"
[ -n "${2}" ] && envname="-${2}"
name="Julia${envname}"
opts="${3}"
else
ver=""
env=$(basename ${VIRTUAL_ENV})
[[ ${env} == "env" ]] && env=$(basename $(dirname ${VIRTUAL_ENV}))
envname=""
[ -n "${1}" ] && envname="-${1}"
name="Julia-${env}${envname}"
opts="${2}"
fi
envopts=""
[ -n "${JULIA_DEPOT_PATH}" ] && envopts="env=Dict(\"JULIA_DEPOT_PATH\"=>\"${JULIA_DEPOT_PATH}\")"
jl ${ver} -e "using IJulia; installkernel(\"${name}\", \"--project\"${opts}; ${envopts});"
}
## Delete automatic kernels made by IJulia
jlkernclean() { find ${XDG_DATA_HOME}/jupyter/kernels -type d -name 'julia-[0-9]*\.[0-9]*' -exec rm -rf {} + }
## Create a Jupyter kernel for the current environment with a custom system image
jlsyskern() {
if [ -f "${VIRTUAL_ENV}/SysImage.so" ]; then
ver=""
name="Sys"
proj="${VIRTUAL_ENV}"
else
ver="${1}"
name="Base-Sys"
proj="${JULIA_DEPOT_PATH}/environments/v${ver}-SysImage"
[ ! -f "${proj}/SysImage.so" ] && return 1
fi
jlkern ${ver} ${name} ", \"-J${proj}/SysImage.so\""
}
## Start Julia using the virtual environment's custom system image
jlsys() {
if [ -f "${VIRTUAL_ENV}/SysImage.so" ]; then
ver=""
proj="${VIRTUAL_ENV}"
opts="${@}"
else
ver="${1}"
proj="${JULIA_DEPOT_PATH}/environments/v${ver}-SysImage"
[ ! -f "${proj}/SysImage.so" ] && return 1
opts="${@:2}"
fi
jl ${ver} -J${proj}/SysImage.so ${opts}
}
## Start Julia in the virtual environment project
jlsysproj() {
if [ -n "${VIRTUAL_ENV}" ]; then
ver=""
proj="${VIRTUAL_ENV}"
opts="${@}"
else
ver="${1}"
proj="${JULIA_DEPOT_PATH}/environments/v${ver}-SysImage"
opts="${@:2}"
fi
jl ${ver} --project=${proj} ${opts}
}
## Run jlpkg in the virtual environment project
jlsyspkg() { [ -n "${VIRTUAL_ENV}" ] && jlpkg --project=${VIRTUAL_ENV} ${@} }
## Build the system image for the current virtual environment
jlbuildsysimage() {
if [ -f "${VIRTUAL_ENV}/Project.toml" ]; then
ver=""
proj="${VIRTUAL_ENV}"
else
ver="${1}"
proj="${JULIA_DEPOT_PATH}/environments/v${ver}-SysImage"
[ ! -f "${proj}/Project.toml" ] && return 1
fi
jlsysproj ${ver} -e "using PackageCompiler, Pkg; create_sysimage([Symbol(x.name) for x in values(Pkg.dependencies()) if x.is_direct_dep]; sysimage_path=\"${proj}/SysImage.so\")"
}

Install Julia

  1. Decide on a Julia home (I use ${XDG_DATA_HOME}/julia)
  2. Set up an environment variable (export JULIA_HOME="${XDG_DATA_HOME}/julia")
  3. Download Julia and extract it to ${JULIA_HOME}/julia-X.X (eg. ${JULIA_HOME}/julia-1.4)
  4. Repeat previous step for all desired Julia versions

Set Up Julia Utilities

  1. Place julia-utilities.sh somewhere on your system (eg. ~/julia-utilties.sh)
  2. In your shell config, make sure to source the file (eg. source ~/julia-utilities.sh)

Using Julia

To access an installed julia version, simply use jl <version>

$ jl 1.4

Note: Visual Studio Code looks for a julia executable in the path to run Julia, so this will not work with VS Code as I have set it up. For me personally, I have decided to only use VS Code when in a sandboxed environment, and that functionality does work because it adds the virtual environment's julia binary to the path.

Creating Sandboxed Virtual Environments

Make sure to have VirtualEnv.jl installed in the base julia installations

  1. cd into the desired location of the virtual environment
  2. Run jlenv <version> <env_name> to create an empty environment: jlenv 1.4 env (If no <env_name> specified, env is the default)
  3. Call jlactivate <env_name> to activate the desired environment: jlactivate env (If no <env_name> specified, env is the default)
  4. Now running julia or simply jl will run the virtual environment's instance of Julia
  5. At any time run deactivate to deactivate the environment

If you have a lot of packages that are commonly used, the jlsetup function in julia-utilities.sh helps automate setting up a new environment, and the jlenvsetup function automates creating the environment, activating it, and calling setup.

jlpkginstall is a useful function for installing the jlpkg command line package manager for Julia into your Virtual Environments. Then, after activating a given environment, you can use the jlpkg command to manage packages without having to open a Julia REPL.

Using Jupyter With Virtual Environments

Setting Up Kernels for the Base Installations

  1. Install IJulia into the base julia installations (eg. jlp 1.4 add IJulia)
  2. Install the base Jupyter Kernel (eg. jlkern 1.4)
  3. This installs the kernel named Julia-Base-X.X
  4. Start Jupyer lab (eg. jupyter lab)

Note: IJulia automatically installs kernels to Julia-X.X upon installation, but because while using sandboxed virtual environments, you may have multiple installs of IJulia, it will keep overwriting the Julia-X.X kernel. This is default behavior that currently cannot be disabled.

Setting Up Kernels For Virtual Environments

  1. Create and activate a virtual environment (eg. jlenv && jlactivate)
  2. Make sure IJulia is installed in the environment (eg. jlpkg add IJulia)
  3. Install the Jupyter Kernel (eg. jlkern)
  4. This installs the kernel named Julia-ENVIRONMENT_NAME-X.X

Advanced Walkthrough

Example using Virtual Environments, System Images, and Jupyter Kernels

Make a test directory and cd into it

$ mkdir TEST && cd TEST

Creates a new julia virtual environment, activates it, and installs base packages

This is the same as running: jlenv 1.4 && jlactivate && jlsetup

$ jlenvsetup 1.4

Add packages to env project to be built into a system image using jlsyspkg alias

Here, jlsyspkg = jlpkg --project=${VIRTUAL_ENV} ...

$ jlsyspkg add Plots, DifferentialEquations

Build a system image for the current virtual environment

$ jlbuildsysimage

Start Julia using the system image for the current virtual environment

$ jlsys
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.2 (2020-05-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Plots

julia> using DifferentialEquations

julia>

Create a Jupyter kernel for the virtual environment

$ jlkern

Create a Jupyter kernel for the virtual environment that uses the system image

$ jlsyskern

Deactivate virtual environment

$ deactivate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment