gistfile1.txt

This is a note stemming from a Twitter conversation with Alexis Richardson (@monadic). I was
complaining that the message queueing software I'd looked at solved a performance problem I
didn't have, and ignored an admin problem I did have. He challenged me to write up my
particular use case for the RabbitMQ engineers to think about.

This is very much a wish-list. Ideally I'd like a queueing solution to massage my back, and
make me hot chocolate too. The practical solution, however, probably lies somewhere between
this document and current message queueing systems.

# Message Queueing Meets SMS

At RedTxt, we're working out how to improve our architecture for sending SMS. We've been
looking at open source message queueing products, but none of them have hit the sweet spot for
us.

We'd like to push all our outbound messages into a queue, and have some processes that watch
the queue, pull off the messages, and pass them on to our external SMS gateways. Something
like RabbitMQ would seem an obvious choice.

We don't need particularly high throughput. We send most of our SMSs in big batches, and each
batch can be a single message in the message queue.

What we do need is good admin tools. I'd love to be able to do many of the things I can do
with my email server (which, after all, is just a specialised message queueing server.)

In particular, we want to pause and resume queues, inspect the contents of a running queue,
and inspect, move, or delete individual messages. It'd also be nice to be able to suspend
individual messages, although this could be accomplished by moving them to another queue.

## Use Case: Double-Checking a Big Send

When we're using new code to generate larges batches of SMS messages, we're not always 100%
confident that we'll get it right in production.

We'd like to pause the queue, push all the messages into it, and manually inspect them to make
sure they're correct. If they are, we just resume the queue. If they're not, we delete them,
fix our code, and try again.

## Use Case: Emergency Stop

A couple of times we've stuffed up badly and starting sending a big batch of messages to the
wrong people. We need a big red emergency stop button that immediately pauses the queue. From
there we can look at what's left in the queue, kill any remaining bad messages, and resend as
necessary.

## Use Case: Handling Gateway Failures

Occasionally, we have a connection to an SMS gateway time out half way through sending a batch
of SMSs. In this case we can't be sure whether or not the batch was sent, except by manual
inspection.

We'd like to leave the message for the batch in the queue, but suspended, while other messages
are still processed. We can then raise an alert, manually check it, and either delete or
resume the suspended message.

(We could also implement this by pushing any message that fails onto a separate queue.)


Pete Yandell
pete@notahat.com
@notahat