Redis vs AMQP vs JMS vs Kafka

amqp_vs_jms.md

AMQP vs JMS

JMS: only java, it is a specification
AMPQ: universal, it is a protocol, it is open standard for messaging

JMS doesn't define a protocol.

JMS is an API and AMQP is a protocol.

AMQP supports 4 message models: Direct, Fanout, Topic, Headers
JMS supports 2 message models: P2P (Point to Point), Publish/Subscribe

AMQP supports security i.e.SASL, TLS
JMS specification does not support security and hence implementation is left with JMS provider.

Message Format Type:
JMS: BytesMessage, MapMessage, ObjectMessage, StreamMessage and text message.
AMQP: binary.

AMQP: Supports transactions (across message queues) and distributed transactions (XA, X/Open, MS DTC)

The main difference with JMS (which is a Java EE specification and API) is that AMQP is just a binary wire protocol which was designed for interoperability between different vendors and platforms. As long it's AMQP complaint, changes at the broker level are not needed.
In other words, you can use a pure Java JMS client to communicate with the AMQP protocol if the messaging server supports it.

Why AMQP is better than JMS

While there are implementations such as ActiveMQ, HornetQ and SonicMQ, that offer some level of cross products interoperability, that is being done thru proprietary protocols, APIs, and client libraries. You can't just replace one broker with anther without go to your code, as it was the case for Java applications. We're deep coupled here to a specified broker that can't easily replaced, if that's possible at all.

AMQP is around a binary wire protocol which was designed for interoperability between different vendors and platforms. As long it's AMQP complaint, changes at the broker level are not needed.

The biggest benefit over JMS is not being locked into one language. AMQP doesn't care if your front end is in Ruby and your back end is in Java. It really makes it much easier to integrate diverse applications into a single stack. JMS is a lowest common denominator because it was designed as an abstraction layer over brokers that already existed. As a result the native APIs always were going to have a richer feature set.

By contrast AMQP, was not designed to layer over existing brokers, but to start with a fresh sheet of paper to provide the desired features for messaging using an open protocol. As a result, AMQP is not constrained by what existing brokers do or do not offer, instead it's design goals have been dictated by what do messaging users want? AMQP is a modern intepretation of what a messaging protocol should offer...both in it's 0.9.1 and 1.0 variations. That's why AMQP is very much not in the same "lowest common denominator" boat as JMS.

(...also crucially different...AMQP provides mechanisms for broker-specific extensions that don't break the overall protocol. So you get the best of both worlds, a common protocol with optional extensions you are free to utilize or not. Which is unlike JMS, where you HAVE to use a completely proprietary native API if you want those extensions. With AMQP you use AMQP, no matter what subset of it's features you want.)

Another key differentiator from AMQP, is that JMS clients are not cross-compatible even within Java. With rare exception you can't use vendor A's JMS client with vendor B's broker. And their adherence to the JMS API interfaces is even loose enough to the point that it's not always plug and play to change clients in your app either.

kafka_vs_rabbitmq.md

Kafka

The uniqueness of Kafka lies in the fact that it handles each topic partition as a log
(that is, an ordered set of messages), and that every message within a given partition is assigned a unique, one-of-a-kind offset.

Kafka doesn’t try to track which message was actually read by what consumer and just hold on to unread messages.
Instead, it holds all of the messages for a pre-specified amount of time, and consumers are charged with tracking their location within each log.

So, Kafka is able to support a huge quantity of consumers and hold tremendous amounts of data without incurring much at all in the way of overhead.

Unlike most messaging systems, the message queue in Kafka is persistent

The message stays in the queue until the retention period/size limit is exceeded, meaning the message is not removed once it’s consumed.
Instead, it can be replayed or consumed multiple times, which is a setting that can be adjusted.

Kafka can be seen as a durable message broker where applications can process and re-process streamed data on disk

Kafka maintains an offset for each message in a partition. The committed position is the last offset that has been saved. Should the process fail and restart, is this the offset that it will recover to. A consumer in Kafka can either automatically commit offsets periodically, or it can choose to control this committed position manually.

How Kafka keeps track of what's been consumed and what has not differs in different versions of Apache Kafka. In earlier versions, the consumer kept track of the offset.

RabbitMQ

Chief among RabbitMQ’s benefits is the ability to flexibly route messages.
Direct or regular expression-based routing allows messages to reach specific queues without additional code
RabbitMQ has four different routing options: direct, topic, fanout, and header exchanges.

RabbitMQ is useful when you have Complex Rounting

Kafka does not support routing; Kafka topics are divided into partitions which contain messages in an unchangeable sequence. You can make use of consumer groups and persistent topics as a substitute for the routing in RabbitMQ, where you send all messages to one topic, but let your consumer groups subscribe from different offsets.

RabbitMQ supports something called priority queues
A message cannot be sent with a priority level, nor be delivered in priority order, in Kafka. All messages in Kafka are stored and delivered in the order in which they are received regardless of how busy the consumer side is.

Both Kafka and RabbitMQ have support for producer acknowledgments (publisher confirms in RabbitMQ) to make sure published messages have safely reached the broker.

RabbitMQ's queues are fastest when they're empty, while Kafka is designed for holding and distributing large volumes of messages. Kafka retains large amounts of data with very little overhead.

RabbitMQ has lazy queues. Lazy queues are queues where the messages are automatically stored to disk, thereby minimizing the RAM usage, but extending the throughput time.

If you publish quicker then you can consume in RabbitMQ, your queue will start to grow and might end up with millions of messages, finally causing RabbitMQ to run out of memory.

Explanations

RabbitMQ: if the requirements are simple enough to deal with system communication through channels/queues, retention and streaming is not a requirement. For e.g. When the manufacture system built the asset it does notify the agreement system to configure the contracts and so on.

Kafka: Event sourcing requirement mainly, when you may need to deal with streams (sometimes infinite), huge amount of data at once properly balanced, replay offsets in order to ensure a given state and so on. Keep in mind that this architecture brings more complexity as well, since it does include concepts such as topics/partitions/brokers/tombstone messages, etc. as a first class importance.

While RabbitMQ is used for traditional messaging, Apache Kafka is used as streaming platform (messaging + distributed storage + processing of data).

You can use Kafka for "traditional messaging", but not use MQ for Kafka-specific scenarios.

Kafka is not competitive but complementary to integration and messaging solutions (including RabbitMQ).

Common Uses Cases:

For RabbitMQ
There are two main situations where I would choose RabbitMQ; For long-running tasks, when I need to run reliable background jobs. And for communication and integration within, and between applications, i.e as middleman between microservices, where a system simply needs to notify another part of the system to start to work on a task, like ordering handling in a webshop (order placed, update order status, send order, payment, etc.).

For Kafka
In general, if you want a framework for storing, reading (re-reading), and analyzing streaming data, use Apache Kafka. It’s ideal for systems that are audited or those that need to store messages permanently.
These can also be broken down into two main use cases for analyzing data (tracking, ingestion, logging, security etc.) or real-time processing.

https://stackoverflow.com/questions/42151544/is-there-any-reason-to-use-rabbitmq-over-kafka/42154452#42154452

https://www.cloudamqp.com/blog/2019-12-12-when-to-use-rabbitmq-or-apache-kafka.html

redis.md

Redis is an in memory database that can persists on disk.
It is an advanced key-value store, a data structure server with: strings, hashes, lists, sets, sorted sets.

Performance is the key factor in Redis.

Redis is not a plain key-value store, it is actually a data structures server, supporting different kinds of values.
What this means is that, while in traditional key-value stores you associated string keys to string values,
in Redis the value is not limited to a simple string, but can also hold more complex data structures.

Use cases for Redis:

One of the most apparent use cases for Redis is using it as a session cache.

Full Page Cache (FPC)
Outside of your basic session tokens, Redis provides a very easy FPC platform to operate in.
Going back to consistency, even across restarts of Redis instances,
with disk persistence your users won’t see a decrease in speed for their page loads —
a drastic change from something like PHP native FPC.

Short-lived messages
Session Caching
Data Caching
Shopping Carts
Voting Systems
Counting Lists
Sets of things
Hybrid Persistence

Redis vs Memcached
The advantages of using Redis over other session stores, such as Memcached, is that Redis offers persistence.

Redis vs AMQP
With Redis the client knows when queue is empty, AMQP does not provide a good way for clients to detect when a finite-length work queue is empty.