JuliaRun Doc Snippets

JuliaRunPackageBundles.md

Julia packages go through a process called compilation on their first use, where the Julia code is compiled to llvm bytecode. The bytecode is cached, and re-used thus making subsequent use faster. Package bundles are special disk volumes that hold Julia packages. They help prepare packages once and use them uniformly across many many containers. They are pre-compiled by a special PkgBuilder job and are mounted as read only volumes to containers that use them. That ensures that when your code starts to use them they are alerady compiled. They are mounted read-only to ensure that they are not modified inadvertendly by some user code.

Preparing a package bundle involves the exact same steps as installaing and loading a package in the Julia REPL, essentially - Pkg.add(...), Pkg.build(...), and using .... Let's say you wish to build a package bundle with these packages: JSON and Distributions. And you wish to call your package bundle mypkg.

• Create a volume for it: ~/.julia/v0.5/JuliaRun/scripts/local/compute/create_volume.sh mypkg
• Let JuliaRun know about it. Edit config.json so that the storage and packages sections look like this:

  ...
  "storage": {
      "hostpath": [ 
          { "name": "juliarun", "path": "/mnt/juliarun", "readonly": false }, 
          { "name": "pkgdefault", "disk": "/mnt/pkgdefault", "readonly": true },
          { "name": "mypkg", "disk": "/mnt/mypkg", "readonly": true }
      ],
      "scratch": [
          { "name": "pkgevalempty", "readonly": false }
      ]
  },
  "packages": ["pkgdefault", "mypkg"],
  ...
  

• Create a folder and a script that builds the package bundle:

mkdir -p /mnt/juliarun/mypkgbuilder/
cat <<EOF > /mnt/juliarun/mypkgbuilder/mypkgbuilder.jl
PKGS=("JSON", "Distributions")

# add the packages
for pkg in PKGS
    Pkg.add(pkg)
end

# build the packages
Pkg.build()
for pkg in keys(Pkg.installed())
    if !isdefined(Symbol(pkg))
        info("Importing $(pkg)...")
        try
            run(`"$(Base.julia_cmd())" "-E import $(Symbol(pkg))"`)
        catch ex
            println("Import failed: ", pkg, " : ", ex)
        end
    end
end
EOF
cp ~/.julia/v0.5/JuliaRun/src/kubernetes/templates/julia/pkgbuilder.sh /mnt/juliarun/

• Run a PkgBuilder job to create the package bundle:

ENV["JRUN_CONFIG"] = "config.json"
using JuliaRun
cm = init()
job = PkgBuilder("mypkgbuilder", "/mnt/juliarun/mypkgbuilder/mypkgbuilder.jl", "juliarun", "mypkg"; cpu="1000m", memory="8Gi")
submit(cm, job)
tail(cm; job=job, follow=true)
delete!(cm, job)

• If the above runs sucessfully, you would have created your package bundle at /mnt/mypkg. Do not modify the files in /mnt/mypkg directly. To update your packages in future, rerun the above PkgBuilder job again.