-
-
Save stevej/3868225 to your computer and use it in GitHub Desktop.
Queue Anti-patterns (an argument in DMs)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I can make a case that queues only do what people want if you don’t consider failure cases. | |
Qs are empty (normal) or full (fail). Normally things are processed quickly. Failure case processing time is unbounded (or “too long”). | |
Solution is always “dump the Q”. Which means you do care about how long it takes to process items. So you want the queue to always be empty | |
Which means you only want a non-failing Q. | |
So why not admit it, use in-proc buffers, run enough servers to handle load? Reject work up front instead of dropping oldest items w/… | |
“flush the queues!” | |
My broader point is: it's really silly to say that you can't come up with some rules of thumb for the use of any particular technique. Every technology has its use; it's just not useful for every case.
Rules of thumb are a poor substitute for understanding your system and the trade offs you are making. Rules of thumb are important, no doubt, when you are dealing with very complex systems that cannot be comprehended easily (your super scalar processor and its memory hierarchy!), but I don't think this holds for design questions involving distributed systems and queues.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is pretty simplistic; it's silly to say outright that queues are bad (and just as silly to say that they are good).
Without even arguing about durability or failure, queues can serve many purposes.
For example, they form a natural load balancer: if you have a large number of low-velocity producers (which cannot use a least-loaded balancer since they don't have enough load), inverting the relationship between producer and consumer effectively gives you load balancing.
Queues allow you to write truly stateless consumers. This can be used to simplify architecture.
Queues can serve as buffers to smooth out spikes in load.
...
Anyway, my broad point is: it's really silly to say that it's outright good or bad. Queues have their use.