- Queues are distributed
- Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
- Queues may have any number of clients sending/receiving messages at once
- Queues can be created with a built in delay on message visibility
- Queues can be created just for dumping messages whose processing has failed
- Messages are "best-effort ordered" given their distributed nature
- Use the Visibility timeout on queues and messages to ensure that a message is only available to a
ReceiveMessageonce or until the timeout expires - Adjust the visibility timeout in-flight to extend or cancel the timeout
- Messages should be manually deleted from the queue after processing
- Messages have unique IDs, but the receipt handle is also required in order to delete them
- Messages have a
bodyfor data - Message queues define their own line protocol (JSON is a simple human readbale choice)
- Messages may have typed and validated attribues (best used for metadata, not actual data)
- Messages may define timers
- Messages can be batched 10 at a time
- 256KB limit total message size or batch message size
- Even with the 256KB limits, buffering and batching client side may reduce costs
- Long Polling can reduce the number of
ReceiveMessagecalls - Using SNS to "subscribe" to queue may be the best approach for message processors
- Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue
- Message processors should be idempotent since each message may be delivered/processed more than once
- Python - SimpleQ (+1 on this one for python code)
- Python - Kombu supports SQS
- Python - boto.sqs boto.sqs tutorial
- Python - aws-sdk
- Python - mock sqs for testing
- Python - async SQS on tornado
- Python - twisted worker daemon for SQS
- Ruby - aws-sdk
- Ruby - sidekiq-sqs
- Super Simple Node and Ruby Example
- Making the most of SQS
- Python - Celery SQS transport (I dislike celery generally but include for completeness)
- Scaling SQS
- Benchmarking SQS (nice article)
- SQS from the shell (could be useful for scripting/testing)
- Neato way to backup a queue from the folks at Medium (looks very useful as a daemon too)
- Queue abstraction on top of SQS and Rabbit (this is a great idea)
- Sidekiq alternative based on SQS
- Another abstraction, this time SQS or Redis
- AMQP <--> SQS Facade
- CloudAMQP (As an alternative I like this since it could be self-hosted at any time)
- Dotcloud suggested their hosted RabbitMQ customers to migrate to CloudAMQP
- IronMQ
- Beanstalk vs IronMQ vs SQS