Installation requires many many packages:
devtools::install_github("ropensci/RedisAPI")
# devtools::install_github("richfitz/RedisHeartbeat") # optional, requires libhiredis to be found
devtools::install_github("gaborcsardi/crayon")
devtools::install_github("gaborcsardi/progress")
devtools::install_github("traitecoevo/rrqueue")
You'll need redis server running!
In one R session, work through the main.R file. It'll block once you get to the last line.
In another R session in the same directory start a worker instance with:
rrqueue::worker("carl")
which will start processing jobs and you'll see the progress bar move. You can start other workers in the same directory with the same command. There is support for spawning them out of the current session but I get some weirdnesses with R's interrupt handling with that. There's also a shell script you can install to make it easy to launch them (rrqueue:::install_scripts
).
The plus side is that the worker pool is dynamic - you can add and remove workers at will (though recovering when a worker dies part way through a task requires some care).
The downside is that you currently have to worry about a lot more things at the moment. My aim is make this eventually as smooth as mclapply. The other downside is overhead. For your case, the overhead is a significant fraction of the computing time.
But it's great to have this as an example. I'll probably try to keep using this as an example of something of someone else's workflow I should try to make smooth.