Skip to content

Instantly share code, notes, and snippets.

@michaellihs

michaellihs/devops-days-kiel-messaging.md

Last active November 26, 2022 16:57
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save michaellihs/18760df2d7b4000bd889fa393aee91b8 to your computer and use it in GitHub Desktop.
Save michaellihs/18760df2d7b4000bd889fa393aee91b8 to your computer and use it in GitHub Desktop.
Download ZIP
DevOps Days Kiel: Message Queues as an Integration Approach for Micro Services
Raw
devops-days-kiel-messaging.md

DevOps Days Kiel: Message Queues as an Integration Approach for Micro Services

Micro Services seem to be a trendy topic in software development nowadays and often they go hand in hand with message queues as an integration approach. So the question arises, whether message queues become the replacement for relational databases as an integration scenario and whether it is a better approach or not.

So here are some questions:

Are you using message queues in your (micro service) projects?

  • Yes, for emails --> RabbitMQ, 10,000 requests / seconds

Which vendor(s) do you use

  • RabbitMQ
  • Kafka
  • See ELK Stack for high load scenarios --> distribution to multiple queues

What are your use cases (event driven application, asynchronous requests...)

  • distribute work on several workers
  • moved from ActiceMQ to kafka

Are you planning for an outage of the queue(s)

  • DB (e.g. Redis) as a cache before the queue

What form of message persistence do you use

  • doesn't matter if messages are lost
  • RabbitMQ provides persistence to disk
  • persistence brings a performance penalty

Are you using your queues in a distributed setup / what about reliability / througput / scalability

  • Kafka --> seems to perform very well
  • Kafka as a replacement for ActiveMQ and Redis (in a Elkstack scenario)

If you are using message queues in a cloud environment: are you using shared or dedicated services - what is your experience with either

  • Amazon Queue SQS --> not reliable but cheap

Do you somehow implement transactionality across multiple operations

  • Using an error queue --> retrying
  • Saga pattern...
  • Think about your architecture --> bounded context --> "self contained system" Stefan Tilkow, Eberhard Wolf
  • Domain Driven Design by Eric Evans

Do you have any metrics for scaling up and down services based on the load of the message queue

  • Measure delays between read and write in the queue
  • Track time when message is written and read --> scale based upon this information

What message formats do you use (JSON, binary, ...)

  • JSON
  • Kafka can compress messages

Apache Kafka vs. RabbitMQ / Message Queues

Some facts from various sources (no guarantees)

Feature RabbitMQ Kafka
Messages/s 20k/s 200k/s write, 3M/s read
Routing capabilities Advanced, see https://www.rabbitmq.com/getstarted.html topic-like exchange only
Acknowledgemts message acknowledgement to assure delivery state consumer needs to remember delivery state (e.g. via Zookeeper)

RabbitMQ vs. Kafka table 1 RabbitMQ vs. Kafka table 2

Any recommendatations for books, source code, resources...

Monotlith vs. Micro services

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment