THEORY: Distributed Transactions and why you should avoid them (2 Phase Commit , Saga Pattern, TCC, Idempotency etc)

avoid-distributed-transactions.md

Distributed Transactions and why you should avoid them

1. Modern technologies won't support it (RabbitMQ, Kafka, etc.);
2. This is a form of using Inter-Process Communication in a synchronized way and this reduces availability;
3. All participants of the distributed transaction need to be avaiable for a distributed commit, again: reduces availability.

Implementing business transactions that span multiple services is not straightforward. Distributed transactions are best avoided because of the CAP theorem. Moreover, many modern (NoSQL) databases don’t support them. The best solution is to use the Saga Pattern.

[...]

One of the most well-known patterns for distributed transactions is called Saga. The first paper about it was published back in 1987 and has it been a popular solution since then.

There are a couple of different ways to implement a saga transaction, but the two most popular are:

• Events/Choreography: When there is no central coordination, each service produces and listen to other service’s events and decides if an action should be taken or not;
• Command/Orchestration: when a coordinator service is responsible for centralizing the saga’s decision making and sequencing business logic;

https://microservices.io/patterns/data/database-per-service.html / https://microservices.io/patterns/data/shared-database.html
https://microservices.io/patterns/data/saga.html
https://microservices.io/patterns/data/application-events.html
https://microservices.io/patterns/data/polling-publisher.html
https://microservices.io/patterns/data/transaction-log-tailing.html

• Compensating Transaction pattern
• Data Consistency Primer

Sagas - old but good post written in 2012

The Saga Pattern in Apache Camel

• Implementing Saga pattern via Events/Choreography
• Implementing Saga pattern via Command/Orchestration

RedHat: Saga: The new era of transactions in a
microservices architecture (está no meu dropbox pessoal também)

Eventual Data Consistency Solution in ServiceComb - part 1
Eventual Data Consistency Solution in ServiceComb - part 2
Eventual Data Consistency Solution in ServiceComb - part 3

TCC - Try, Confirm and Cancel Pattern

Some articles about this simple way of implementing distributed transactions:

• Presentation in InfoQ 2012: Transactions for the REST of Us
  • Slides: Transactions for the REST of Us or this better version of the slides from 2014
  • Article: Transactions for the REST of Us
  • Transactions for the REST of us
  • Transaction management API for REST: TCC or Download the e-book from here
  • Paper: Atomic Distributed Transactions: a RESTful Design or download the PDF directly from the site
  • Chapter in the book REST: from Research to Practice - Towards Distributed Atomic Transactions over RESTful Services
• Distributed Transaction Management with REST and Try Confirm/Cancel Pattern
• Talk no QconSP: Transações Compensatórias usando REST (2011) - by Luca Bastos
  • curioso que estive nessa talk no QconSP, muito boa por sinal! Luca Bastos deu um show de conhecimento e carisma;
  • mais curioso ainda é que um ano antes, em 2010, implementei exatamente esse pattern (mas desconhecia que era um padrão difundido) numa consultoria para integrar um PDV em Delphi com nosso sistema web em Java e VRaptor3;
  • pasmem, aprendi esse pattern e conceito com devs da velha guarda do Delphi;
• About the difference between TCC, Saga and 2-Phase-Commit
  • Saga Pattern vs Two Phase Commit vs TCC
  • 2PC, Saga, and TCC

Very good thread on Twitter by Mike Kowalski (@mikemybytes) about DIY Saga implementation vs BMP workflow-based one

Compensation is one of the trickiest parts of Saga or any other transaction-like processing in the #distributedsystems - here is why I think so... (thread)

• Camunda: Saga - How to implement complex business transactions without two phase commit.
• Example of Saga Pattern with async compensation using Camunda

InfoQ: Saga Orchestration for Microservices Using the Outbox Pattern

Existem diversos desafios de implementar sistemas distribuídos sem as facilidades de transações (ACID) - também conhecidos como "transações distribuídas". Alguns pontos importantes sobre lidar com sistemas distribuídos:

• como não há transação entre serviços distribuídos precisamos investir em retries, at-least once delivery e idempotency para cada chamada remota;
• ao investir nos itens acima significa que precisamos abraçar consistência eventual em algum nível, afinal os serviços vão estar inconsistentes em algum momento (geralmente entre os erros e retries);
• vale salientar que implementar reliable retries requer manter estado em algum lugar, ou seja, stateful retries;
• junto a isso, em sistemas mais complexos se faz necessário ainda compensação (rotinas de cancelamento, informar usuário sobre problemas, oferecer cupons ou descontos, agendar outro voo etc), que muitas vezes é delegada para o negócio ou operacional (intervenção humano);
• a idéia principal aqui é que não há espaço para "achismo": um serviço A, ao se integrar com outro serviço B, deve ter certeza que sua lógica de negócio foi executada de fato, ele precisa encerrar o ciclo do workflow (seja com sucesso ou erro - codigos HTTP validos por exemplo);

E aqui alguns artigos interessantes:

• Camunda: Achieving consistency without transaction managers
• Camunda: Lost in transaction?
  Strategies to manage consistency in distributed systems
• Paper: Life beyond Distributed Transactions:
  an Apostate’s Opinion
• Camunda: 3 common pitfalls in microservice integration — and how to avoid them

Ainda sobre transações distribuídas e garantias de entrega:

What You Want Is What You Don’t: Understanding Trade-Offs in Distributed Messaging

If you’re distributed, forget about ordering and start thinking about commutativity. Forget about guaranteed delivery and start thinking about idempotence.

Outros artigos:

• Nobody Needs Reliable Messaging
• What is commutativity and why is it so useful in distributed systems?

  • If you can make the order not matter, you often can eliminate a lot of coordination code.

• You Cannot Have Exactly-Once Delivery Redux
• Exactly-once message delivery (or Exactly-once processing)

Mensageria e At-Least Once Delivery

Trabalhar com mensageria com garantias At-Least Once significa abraçar seus tradeoffs:

Apart from being duplicated, in-flight messages can get re-ordered. There are many reasons for this to happen, one of the most obvious being message re-delivery mechanism. If a delivery fails, a message is available for reprocessing only after some back-off period. Any other in-flight message can be processed during that time causing the respective order of those messages to change.

When combined, duplication and re-ordering can result, at the receiver side, in many different processing sequences. The only guarantee is that the resulting sequence contains at least one copy of each message sent.

Message Ordering e sua complexidade

Message Ordering eh das coisas mais escorregadias de se fazer em Mensageira.

Não importa seu broker, seja um #RabbitMQ, #ActiveMQ ou #Kafka.

Mesmo quando se tem 1-único-Producer e 1-único-Consumer ainda assim existem edge-cases que podem levar a mensagens fora de ordem e consequentemente inconsistência dos dados 🤡

No fim, devemos abraçar um príncipio chamado de “The End-to-End Principle” (E2E Principle).

Quando falo de E2E Principle, estou falando de trazer alguma “inteligência” para as pontas do seu sistema, para seu producer e consumer, e se nortear por:

1. abrace At-Least Once Delivery;

2. não confie no seu Broker Delivery Order;

3. comutatividade em vez ordenação;

4. idempotência e/ou deduplication;

1) abrace At-Least Once Delivery

É praticamente impossível implementar a semântica Exactly-Once Delivery, não acredite cegamente no que os vendors te falam!

O que podemos fazer eh fingir sua execução implementando mecanismos que a façam acontecer em cima da semântica At Least-Once Delivery, como o uso de idempotência, deduplication, transações etc.

2) não confie no seu Broker Delivery Order

A ordem que as mensagens entram e saem no seu broker tem pouca ou nenhuma importância, confiar nela eh a melhor forma de se dar mal.

O que importa de fato eh a ordem de negócio das mensagens (Business event order), pois eh com base nela que vc vai desenhar e implementar sua solução.

3) comutatividade em vez ordenação

Dado que a ordem de mensagens não pode ser confiada nem garantida pelo broker, aceite que as mensagens podem (e eventualmente irão) chegar fora de ordem.

Portanto, programe seu código em consumers para isso, não espere o pior acontecer.

4) idempotência e deduplication

Embora ambos os conceitos sejam fáceis de entender, eles não necessariamente são fáceis de implementar.

Especialmente se você lida com sistemas externos, ou seja, sua lógica causa efeitos colaterais em outros serviços, como RDBMS, cache ou um processo qualquer.

Concluindo

Enfim, sem E2E Principle, a tal “inteligência” nas bordas, eventualmente você terá problemas com ordem de mensagens e inconsistência de dados.

Isso pode acontecer com você agora ou em 10 anos, quem sabe 🤷‍♀️

Você precisa se preocupar? Diria que sim, mas implementar algo para resolver eh outra estória. Desenhar e implementar para tolerância a falhas tem custo e sua criticidade depende do seu contexto.

Links:

• @rponte: thread no twitter
• Messaging Reliability and Ordering
• The end-to-end principle in distributed systems
• Wiki: End to End Principle
• StackOverflow: Difference between idempotence and exactly-once in Kafka Stream
• Zalando REST API Guide: 212 - SHOULD design for idempotent out-of-order processing
• Safely repeating failed calls
• An Introduction to Distributed Systems: Distributed Queues

Artigos bacanas sobre Outbox e Inbox Pattern, garantias de entrega e uso de Logical Replication do Postgres em vez de um Job em background:

• Outbox, Inbox patterns and delivery guarantees explained
• Push-based Outbox Pattern with Postgres Logical Replication
• How Postgres sequences issues can impact your messaging guarantees
• Panayiotis Kritiotis: Outbox pattern - Why, How and Implementation Challenges
• Panayiotis Kritiotis: Implementing the Outbox pattern in go

Outros artigos:

• Transactional outbox - Como acabar com a escrita dupla

IBM: Extend Saga to the cloud for distributed transactions

https://twitter.com/martinkl/status/1349099872897658880?s=46&t=YBcOZUyNB0BvjxHI_HTrKQ

• A Use Case for Transactions: Adapting to Transactional Outbox Pattern Semantics in Spring Cloud Stream Kafka Binder

But as you said, saga is a distribution transaction pattern also. So... what is "avoid distribution transaction"?

• Parallel, Back-pressured Kafka Consumer
  

This article has a very didact explanation of how to implement the different types of delivery semantics, as we can see below:

Offset Manager

Each message in Kafka is associated with an offset - an integer number denoting its position in the current partition. By storing this number, we essentially provide a checkpoint for our consumer. If it fails and comes back, it knows from where to continue. As such, it is vital for implementing various processing guarantees in Kafka:

• For at-most-once, we need to save $offset + 1 before processing $offset. If our consumer fails before successfully process $offset and restarts, it will continue from $offset + 1 and not reprocess $offset.
• For at-least-once, we need to successfully process $offset before saving $offset + 1. If our consumer fails before saving $offset + 1 and restarts, it will continue from and reprocess $offset.
• For exactly-once using an external transactional storage - we need to process $offset and save $offset + 1 within one transaction and roll back if anything goes wrong.

AWS Prescriptive Guidance

• AWS Prescriptive Guidance: Transactional outbox pattern
• AWS Prescriptive Guidance: Saga patterns
  • Saga choreography pattern
  • Saga orchestration pattern

Monkey Exchange - DUAL WRITE: MANTENDO A CONSISTÊNCIA

• @rponte's Gist: Designing fault-tolerant and idempotent APIs with HTTP requests mapped to database's transactions at 1:1 model
• LinkedIn: "Como vocês tem implementado idempotência nas APIs REST das tuas aplicações?"

• Twitter: pergunta do Tomaz sobre impactos do dual-write
  • Twitter: minha resposta
  • Twitter: quando caiu a ficha

Meu RT quando caiu a ficha:

excelente pergunta do @tomazfernandes_ 👏🏻👏🏻

uma solução robusta, tolerante a falhas e barata de implementar que seja retriable, idempotente e abrace consistência eventual eh justamente Outbox Pattern 🤤🤤

viva o ACID dos RDBMS + aquele job em background que roda a cada 1min 🥳

CodeOpinion blog: Listen to yourself pattern: Is it an alternative to the Outbox Pattern?

Tweet by @DominikTornow

The ingredients of failure tolerance & transparency are more basic than you would think:

1⃣ Guarantee retries via persisting the invocation
2⃣ Guarantee correctness of retries via idempotent steps

Outlined in Fault Tolerance via Idempotence