Julia package workflow

julia_revise_workflow.md

A julia package workflow with Revise

You know your project will become a package -- don't deny it. Here's a hygienic workflow to start with, right at the outset.

The first thing is to install Revise and use it.

• Hit the ] key to enter pkg> mode. When you see the pkg> prompt:
• pkg> add Revise
• hit Bksp to get back to the julia> REPL, and once there:
• julia> using Revise

Then generate your empty package MyFoo within julia>. Navigate to your development directory:

• julia> cd(ENV["JULIA_PKG_DEVDIR"])
• see where you are with julia> pwd()
• should be something like /home/myself/.julia/dev
• hit the ] key to enter pkg> mode. When you see the pkg> prompt:
• pkg> generate MyFoo

This is the output you will see

Generating  project MyFoo:
MyFoo/Project.toml
MyFoo/src/MyFoo.jl

• Now register this MyFoo project for local development
• pkg> dev MyFoo

You should see output like:

Path `MyFoo` exists and looks like the correct package. Using existing path.
   Resolving package versions...
    Updating `/home/myself/.julia/environments/v1.6/Project.toml`
  [486a3583] + MyFoo v0.1.0 `../../../../myself/.julia/dev/MyFoo`
    Updating `/home/myself/.julia/environments/v1.6/Manifest.toml`
  [486a3583] + MyFoo v0.1.0 `../../../../myself/.julia/dev/MyFoo`

We could be done here ... all you have to do is modify src/MyFoo.jl within $JULIA_PKG_DEVDIR/MyFoo. Everytime you restart Julia and use module MyFoo, changes in MyFoo.jl will be recompiled. But that's not interactive or efficient. So we use Revise instead

Now use Revise

• Without closing the original julia REPL, in a separate window / terminal, in your favourite editor replace the barebones $JULIA_PKG_DEVDIR/MyFoo/src/MyFoo.jl with this code snippet and save it. If you're not sure where this JULIA_PKG_DEVDIR is, read a few lines above, around "see where you are with".

module MyFoo

# packages used in this module
using Random, LinearAlgebra

# define some struct Bar
mutable struct Bar
    n :: Int # Don't need :: Int but helps write type stable code
end

# some function with same name as struct Bar, this is multiple dispach in action
function Bar(;n=10) # default argument is 10
    # this will initialise the struct Bar
    Bar(n) # last line of a function is return value
end

function usebar(b::Bar)
    X = rand(b.n)
    @info "Yay!"
    return cholesky(X*X' + I)
end

end # module

Now go back to your original julia> REPL session, and try running the following code

julia> using MyFoo

We'll deal with the warnings later.

[ Info: Precompiling Foo [08477627-533e-46b7-bb0c-497c24a2cff2]
Warning: Package Foo does not have Random in its dependencies:
- If you have Foo checked out for development and have
  added Random as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with Foo
Loading Random into Foo from project dependency, future warnings for Foo are suppressed.

For now, let's try using the functions we defined in MyFoo

julia> b = MyFoo.Bar()
MyFoo.Bar(10)

And then use the struct b

julia> c = MyFoo.usebar(b)
[ Info: Yay!
LinearAlgebra.Cholesky{Float64, Matrix{Float64}}
U factor:
10×?10 LinearAlgebra.UpperTriangular{Float64, Matrix{Float64}}:
 1.33327  0.254364  0.414919  0.0737133   0.411594   0.414414   0.00581548
          1.04077   0.130408  0.0231678   0.129363   0.130249   0.00182779
                    1.09745   0.0330869   0.184748   0.186013   0.00261033
                              1.00267     0.0298275  0.0300319  0.000421439
                                          0.206802   0.208219   0.00292194
 .
 .

Modify src/MyFoo.jl and save it

For example, change @info "Yay" to your favourite text. And then do

julia> d = MyFoo.usebar(b)
[ Info: Pithy quote
LinearAlgebra.Cholesky{Float64, Matrix{Float64}}
U factor:
10×10 LinearAlgebra.UpperTriangular{Float64, Matrix{Float64}}:
 1.09249  0.307465  0.111539  0.175899  0.0297159   0.113394   0.302177 ...

And you get the idea. Now let's look at those pesky warnings, which you can ignore if you are the only one using this code. If you want to distribute this code eventually from GitHub or wherever, all modules you have used in your code (here LinearAlgebra, Random) need to be listed as dependencies. To fix this, simply navigate to the root development directory for the package like so and follow along:

• julia> cd(ENV["JULIA_PKG_DEVDIR"]*"/MyFoo") that's how you concatenate strings in Julia by the way, with a *
• then activate the environment in pkg> mode. Press ] to enter this mode and
• pkg> activate . You should see output like

 Activating environment at `/home/myself/.julia/dev/MyFoo/Project.toml`

(MyFoo) pkg>

Now simply add the packages you have been warned about!

(MyFoo) pkg> add LinearAlgebra, Random
Updating registry ...

If you look for installed package status in pkg mode you will see

(MyFoo) pkg> status
     Project MyFoo v0.1.0
      Status `/home/myself/.julia/dev/MyFoo/Project.toml`
  [37e2e46d] LinearAlgebra
  [9a3f8284] Random

You can exit Julia with Ctrl+D and restart it. Then if you do

julia> using Revise, MyFoo

there may yet be a warning, like this

[ Info: Precompiling MyFoo [81c9f0e8-0b48-40f4-9ea9-77faf37d81cc]
 Warning: Package MyFoo does not have Random in its dependencies:
 - If you have MyFoo checked out for development and have
   added Random as a dependency but haven't updated your primary
   environment's manifest file, try `Pkg.resolve()`.
 - Otherwise you may need to report an issue with MyFoo
 Loading Random into MyFoo from project dependency, future warnings for MyFoo are suppressed.

which is simply resolved as suggested by going to pkg> mode and doing as suggested in the warning

(v1.6) pkg> resolve
...

there will be no more warnings the next time you are using MyFoo.

That's it folks

From now on, whenever you start Julia, whether in the shell, VSCode or Atom/Juno, make sure you do:

julia> using Revise # keeps track of changes in dev packages from now on
julia> using MyFoo 

make changes to MyFoo.jl, save it, and you can keep using the same REPL session.

Thanks Anand as this is helpful. I saw that you can add this to ~/.julia/config/startup.jl to automaticaly use Revise when Julia starts:

# Setup OhMyREPL 
import Pkg
let
    pkgs = ["OhMyREPL", "Revise"]
    for pkg in pkgs
    if Base.find_package(pkg) === nothing
        Pkg.add(pkg)
    end
    end
end

using OhMyREPL
using Revise

Yes indeed -- but not sure I want Revise loaded all the time ... especially for parallel code, I'm not entirely sure how Revise works in parallel (it does, I just don't know how) -- but 90% of the time you do want Revise loaded. Also, Revise doesn't work in the initial stages of development when you update the definition of a `mutable struct` :(

…

On Sat, 10 Jul 2021 at 04:59, Kerry Key ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Thanks Anand as this is helpful. I saw that you can add this to ~/.julia/config/startup.jl to automaticaly use Revise when Julia starts: # Setup OhMyREPL import Pkg let pkgs = ["OhMyREPL", "Revise"] for pkg in pkgs if Base.find_package(pkg) === nothing Pkg.add(pkg) end end end using OhMyREPL using Revise — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_e593751b24e45f8160ba8041fb811680-23gistcomment-2D3807816&d=DwMCaQ&c=-35OiAkTchMrZOngvJPOeA&r=2J0TEt1AUBEcd88gHSdo3g&m=MBZH1bWKCYTJsoB7Re8tNR8qaDeK0d810E5e3d_vmt0&s=7oWkMdZJwJiHiGwgVI-BUDpj8vbWEbddxA0-usUDXjQ&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMRVKW5AT6T3HCNKF2ZZOHDTW5BKTANCNFSM5ADI766Q&d=DwMCaQ&c=-35OiAkTchMrZOngvJPOeA&r=2J0TEt1AUBEcd88gHSdo3g&m=MBZH1bWKCYTJsoB7Re8tNR8qaDeK0d810E5e3d_vmt0&s=i88FhXlNKM4ZSM2_A5VAEeDCS6OrGvDJ841KagWtjLE&e=> .