There are three types of semantic guarantees Apache Kafka® provides between the broker, producers and consumers. Semantic guarantee refers to how the broker, producer and consumer agree to share messages.
-
At most once: Messages are delivered once, and if there is a system failure, messages may be lost and are not redelivered.
-
At least once: Messages are delivered one or more times. If there is a system failure, messages are never lost, but they may be delivered more than once.
-
Exactly once: This is the preferred behavior in that each message is delivered once and only once. Messages are never lost or read twice even if some part of the system fails.
See Message Delivery Guarantees for more details.
See how Spring for Apache Kafka supports transactions and implements Exactly Once Semantics.
Notes.
-
Transactions are enabled by providing the DefaultKafkaProducerFactory with a transactionIdPrefix.
-
For applications running with multiple instances, the transactionIdPrefix must be unique per instance.
-
With Spring Boot, it is only necessary to set the spring.kafka.producer.transaction-id-prefix property - Boot will automatically configure a KafkaTransactionManager bean and wire it into the listener container.
-
-
For a read → process → write sequence, it is guaranteed that the sequence is completed exactly once. (The read and process have at least once semantics).
-
Starting with versions 2.5.17, 2.6.12, 2.7.9 and 2.8.0, if the commit fails on the synchronized transaction (after the primary transaction has committed), the exception will be thrown to the caller. Previously, this was silently ignored (logged at debug). Applications should take remedial action, if necessary, to compensate for the committed primary transaction.
See RetryTopicConfigurer and @RetryableTopic for the details on how to accomplish a distributed retry / DLT pattern in a non-blocking fashion (at the expense of ordering guarantees). See DefaultErrorHandler for blocking retry implementation. Some examples are available at https://www.baeldung.com/spring-retry-kafka-consumer.
The following configuration worked well.
@RetryableTopic(autoCreateTopics = "false")Setting autoCreateTopics to false enables either use manually created topics or auto create topics by the broker using default partition number. It's important for the retry/dlt topics partition number to match the source topic configuration.
The default value of attempts is set to 3 which is reasonable and results in the 3 addtional topics used:
- {sorce-topic}-retry-0
- {sorce-topic}-retry-1
- {sorce-topic}-dlt
While it would have been more convenient to provide a configuration function reference in the annotation, overwriting the topic names is not too difficult. Spring Boot 3 example:
@Bean
public RetryTopicComponentFactory customRetryTopicComponentFactory() {
return new RetryTopicComponentFactory() {
@Override
public RetryTopicNamesProviderFactory retryTopicNamesProviderFactory() {
return new CustomRetryTopicNamesProviderFactory();
}
};
}
static class CustomRetryTopicNamesProviderFactory implements RetryTopicNamesProviderFactory {
@Override
public RetryTopicNamesProvider createRetryTopicNamesProvider(
DestinationTopic.Properties properties) {
if(properties.isMainEndpoint()) {
return new SuffixingRetryTopicNamesProvider(properties);
}
else {
return new SuffixingRetryTopicNamesProvider(properties) {
@Override
public String getTopicName(String topic) {
return "my-prefix-" + super.getTopicName(topic);
}
};
}
}
}If a consumer group is changed (to re-process messages), not only messages from the source topic are re-processed but also those from the re-try topics.
With attempts set to 3 and the 3 addtional topics used:
- {sorce-topic}-retry-0
- {sorce-topic}-retry-1
- {sorce-topic}-dlt
not only all of the messages from the source topic are processed but also all of messages previsously re-tried from {sorce-topic}-retry-0 and from {sorce-topic}-retry-1.