SQLite as a Background Job Queue.md

• Features

  • Schedule jobs
  • Schedule jobs in the future
  • Explicit failure: An exception was thrown
  • Implicit failure: A job was picked up and not finished in a certain amount of time
  • Retry jobs that failed (explicitly or implicitly)
  • After a certain number of retries, fail the job
  • Have a way to force the worker to run immediately
  • On graceful termination finish existing jobs

• Questions

  • Where do we implement this?
    • node
      • Con: Needs to be aware of sqlite and server
        • Circular dependency, because server already depends on node
    • sqlite
      • Con: Needs to bring node in, which has its own signal handler stuff
    • Separate package
      • Con: A lot of fragmentation
  • Separate database (better performance) or same database (ease of management)?
  • How do we run the migration to create the schema?
    • On migrate()
    • On backgroundJob()
  • How to distribute work within a backgroundJob process
    • Different backgroundJob() per type
      • Pro: Simpler to implement
      • Con: Too many jobs of different kinds may lead to thrashing
      • Pro: If a certain type of job is too heavy, it doesn’t overwhelm a single process
    • Poll of workers
      • Pro: May work on multiple jobs of the same type, increasing throughput
      • Con: You’d have to poll too often

• Future

  • Force a worker to run immediately
    • Watch for filesystem changes
    • ZeroMQ to communicate between workers
    • Redis or something like that

• References

  • https://github.com/collectiveidea/delayed_job
  • https://github.com/betterment/delayed
  • https://github.com/bensheldon/good_job
  • https://github.com/litements/litequeue
  • https://github.com/diamondio/better-queue-sqlite
  • https://webapp.io/blog/postgres-is-the-answer/