Skip to content

Instantly share code, notes, and snippets.

@BillBarnhill

BillBarnhill/queues.io.md

Forked from gtors/queues.io.md
Created January 18, 2023 21:41
Show Gist options
  • Save BillBarnhill/b7478200a8e241f91ebede25f2b441b4 to your computer and use it in GitHub Desktop.
Save BillBarnhill/b7478200a8e241f91ebede25f2b441b4 to your computer and use it in GitHub Desktop.
Download ZIP
queues.io snapshot
Raw
queues.io.md

Queues

Job queues, message queues and other queues. Almost all of them in one place.

About

There are many queueing systems out there. Each one of them is different and was created for solving certain problems. This page tries to collect the libraries that are widely popular and have a successful record of running on (big) production systems.

The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.

The whole site is open source and is updated quite frequently. If you find any mistakes or see something missing, feel free to post an issue or send a pull request. All contributions are welcome! Thanks.

Deadbeef.news: Fresh news about tech

Popular Projects

ActiveMQ

http://activemq.apache.org/

  • java

Apache ActiveMQ ™ is the most popular and powerful open source messaging and Integration Patterns server.

Resources

Amazon MQ

http://aws.amazon.com/amazon-mq/

  • service

Amazon MQ is a managed message broker service for Apache ActiveMQ that makes it easy to set up and operate message brokers in the cloud.

Resources

Amazon Simple Queue Service (SQS)

http://aws.amazon.com/sqs/

  • service

Amazon Simple Queue Service (Amazon SQS) is a messaging queue service that handles message or work flows between other components in a system.

Resources

Apollo

http://activemq.apache.org/apollo/

  • java
  • scala
  • amqp

ActiveMQ's next generation of messaging.

Resources

Beanstalkd

http://kr.github.io/beanstalkd/

  • c

Beanstalkd is a simple, fast work queue.

Resources

Beanstalkg

https://github.com/vimukthi-git/beanstalkg/pulls

  • go

Beanstalkg is a reimplementation of beanstalkd in Go.

Resources

Celery

http://www.celeryproject.org/

  • python

Distributed Task Queue

Resources

Darner

https://github.com/wavii/darner

  • cpp

simple, lightweight message queue

Resources

Delayed::Job

https://github.com/collectiveidea/delayed_job

  • ruby
  • mysql

Database based asynchronously priority queue system

Resources

Disque

https://github.com/antirez/disque

  • redis
  • disque

An in-memory, distributed job queue.

Resources

Faktory

https://github.com/contribsys/faktory

  • go
  • ruby
  • rocksdb

Language-agnostic, persistent background job system.

Resources

Gearman

http://gearman.org

  • c

Gearman Job Server

Resources

HornetQ

http://www.jboss.org/hornetq

  • java
  • amqp
  • jms

An open source project to build a multi-protocol, embeddable, very high performance, clustered, asynchronous messaging system (Donated to Apache ActiveMQ community).

Resources

huey

https://huey.readthedocs.org/en/latest/

  • python
  • redis
  • django

a little task queue

Resources

IronMQ

http://www.iron.io/mq

  • go
  • service

IronMQ is an easy-to-use highly available message queuing service. It is available as a cloud service on Amazon and Rackspace as well as on-premise with Iron.io's enterprise offering. Features include a nice dashboard to manage queues, easy to create webhooks, unicast and multicast Push Queues, autoscaling alerts for worker processes, and error queues.

Resources

Apache Kafka

http://kafka.apache.org/

  • scala

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Resources

Kue

https://github.com/Automattic/kue

  • node.js
  • priority
  • redis

A distributed priority job queue backed by redis, built for node.js

  • Delayed jobs
  • Optional retries with backoff
  • Distribution of parallel work load
  • Job TTL
  • Job event and progress pubsub
  • Rich integrated UI
  • RESTful JSON API
  • Graceful shutdown
  • Powered by Redis

Resources

Mappedbus

http://mappedbus.io/

  • java
  • high-throughput
  • low-latency
  • message-passing
  • ipc

A Java based high throughput, low latency message bus, using either a memory mapped file or shared memory as transport.

Resources

Message Bus

https://github.com/groupon/Message-Bus

  • hornetq
  • java
  • ruby

Message bus is a distributed messaging platform built with HornetQ, used extensively at Groupon.

Resources

nanomsg

http://nanomsg.org/

  • c
  • zeromq

nanomsg is a socket library that provides several common communication patterns

Resources

NATS

https://nats.io

  • go
  • .NET
  • node
  • nginx
  • java
  • ruby
  • python
  • scala

NATS is an open-source, high-performance, lightweight cloud messaging system.

Resources

NSQ

https://github.com/bitly/nsq

  • go

realtime distributed message processing at scale

Resources

QDB

http://qdb.io/

  • java
  • persistent
  • replay
  • backup

QDB is a Persistent Message Queue With Replay support

Resources

Apache Qpid

http://qpid.apache.org/

  • java
  • amqp
  • cpp

Apache Qpid™ makes messaging tools that speak AMQP and support many languages and platforms (in C++, Python, Java JMS and .NET)

Resources

queue_classic

https://github.com/ryandotsmith/queue_classic

  • ruby
  • postgres

Simple, efficient worker queue for Ruby & PostgreSQL.

Resources

RabbitMQ

http://www.rabbitmq.com/

  • erlang
  • amqp

Robust messaging for applications

Resources

Resque

https://github.com/resque/resque

  • ruby
  • redis

A rock-solid job queue, written in Ruby, backed by Redis.

Resources

RestMQ

http://restmq.com/

  • python
  • redis

RestMQ is a message queue which uses HTTP as transport, JSON to format a minimalist protocol and is organized as REST resources. It stands on the shoulder of giants, built over Python, Twisted, Cyclone (a Tornado implementation over twisted) and Redis.

Resources

RQ

http://python-rq.org/

  • python
  • redis

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers

Resources

Siberite

http://siberite.org/

  • go

Simple, lightweight, leveldb backed message queue

  • Darner rewritten in Go with additional features
  • Single topic can be consumed multiple times using durable cursors
  • Uses Kestrel (memcached) protocol
  • Keeps all messages out of process
  • Small amount of in-resident memory regardless of queue size
  • Two-phase reliable fetch

Resources

Sidekiq

http://sidekiq.org/

  • ruby
  • crystal
  • redis

Simple, efficient background processing for Ruby.

  • Based on Redis
  • Multithreaded, employs Celluloid under the hood
  • Web UI
  • Resque Compatibility
  • Sidekiq Pro - paid version ($750) with support and additional features (batches, notifications, reliability, metrics)
  • Fast growing community

Resources

Zaqar (ex Marconi)

https://wiki.openstack.org/wiki/Zaqar

  • openstack
  • python
  • mongodb
  • sqlite
  • durable

Zaqar is a queuing and notification service made by and for OpenStack, but not only for it

Resources

ZeroMQ

http://www.zeromq.org/

  • cpp
  • java

The Intelligent Transport Layer

Resources

by @lukaszx0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment