So I guess in Rails and Laravel world queues and background jobs are really all front and center but I don't hear that mentioned even moderately in Node. What will be the equivalent?
Mike McNeil sent the following messages at 8:21 PM
The equivalent is like what's in the sendTemplateEmail() helper in default new sails apps, at the bottom
I know of a couple of valid reasons to use queues in Node.js:
- You need durability. i.e. if calling out to Sendgrid might not work, you can always add code to handle that. But if it's REALLY important to you that the email gets sent even in an extremely rare edge case where the server shuts down (e.g. you redeploy the app) at just the wrong moment, then you can use a queue to make the email sending task "durable". i.e. if the server shuts down without hearing a confirmed acknowledgment, then the task to send that email is still queued up.
- You have some kind of task that's REALLY CPU intensive, or intensive in some other way, and you want it to run on a separate server from any of your servers that are handling requests. I don't think this is a very good reason, but it's conceivably possible.
So I don't think there's much of a reason to do this. Definitely not in an MVP. It is, of course, possible to do this with Node.js, and valid sometimes. I would think of queuing like database transactions. It's hard to predict where you'll need it, it only matters at quite high scale, and you may never need it at all.
The thing is, people are accustomed to doing it from Rails, Java, etc, even for an MVP. (Because those servers are blocking, so if you start sending an email in your backend code, it's blocking all other requests from being handled by that server. But Node.js is non-blocking.)
In my experience, queuing jobs or tasks makes sense in certain cases where the processing can be deferred and you don't need the answer back to the requester:
You need to ingest requests at a speed higher that you can process in "real time", for example, downloading surveillance videos from a connection-limited edge device, where connection can be limited or even broken, like through cellular routers, to be stored on S3.
The process can take considerable time and you can fire a process to be run "whenever possible": video conversion, third party API calls to augment the information (AI APIs, etc).
For that, you can use different queueing systems, provide by your favorite cloud (AWS SQS, for example), other known third party queue systems, or Node modules like 'bull', which uses Redis or 'agenda', which uses MongoDB (simpler to view queues statuses, as you can query the collections).
One last aspect: if you have multiple node processes processing queues on limited resources, sometimes you need to use Semaphores to limit the use of those resources.