At-least-once delivery (the transactional outbox from Day 30) means messages arrive; it says nothing about what happens when one can't be processed. Every consumer in OrderHub up to now used the same blunt policy: catch the error, log it, return normally — swallow and log. It never crashed, but a genuinely broken ("poison") message was silently dropped with no way to recover it. And the only alternative the consumers had — let the exception escape — would loop the partition forever. Neither is acceptable. Day 31 installs a real policy in one place. Here's the whole thing.
The problem with both old options
@KafkaListener(topics = "order-placed")
public void onOrderPlaced(OrderPlacedEvent event) {
try {
inventoryService.reserve(event.item(), event.quantity());
} catch (Exception e) {
log.warn("failed: {}", e.toString()); // swallow: logged, then GONE
} // offset advances; no retry, no trace
}
// let it ESCAPE instead? container redelivers the SAME record forever -> partition WEDGED
Swallow-and-log loses the record silently and gives a transient fault zero retries. Escaping wedges the partition so nothing behind the poison record moves. Day 31 replaces both.
One error handler, in one place
Spring Kafka's DefaultErrorHandler is the container-level hook that decides what happens when a listener throws. Configure it once and attach it to the listener-container factory, and every listener inherits the exact same retry-then-recover behaviour. It takes two collaborators: a backoff (how many retries, how far apart) and a recoverer (what to do once retries are exhausted).
@Bean
public DefaultErrorHandler kafkaErrorHandler(KafkaTemplate<Object,Object> dltTemplate) {
var recoverer = new DeadLetterPublishingRecoverer(dltTemplate); // where exhausted records go
var backOff = new FixedBackOff(props.retryBackoffMs(), props.retryAttempts());
return new DefaultErrorHandler(recoverer, backOff);
}
FixedBackOff(interval, maxRetries) is the retry budget: maxRetries is the number of extra deliveries after the first, so total attempts = 1 + retryAttempts, each separated by interval ms. A poison message that always fails is tried a bounded number of times and no more. The backoff is what makes retries useful for transient faults — a dependency that's briefly down often recovers by attempt 2 or 3, so the message succeeds without ever reaching the DLT.
Route the exhausted record to <topic>.DLT
The recoverer is called once, after retries are spent. DeadLetterPublishingRecoverer republishes the failed record to a dead-letter topic — by default the original name with a .DLT suffix, so order-placed becomes order-placed.DLT — preserving the key and payload and adding headers describing why it failed (exception, original topic/partition/offset). It's an ordinary Kafka topic: a parking lot where poison messages stay durable and inspectable while the main partition's offset advances past them, so healthy traffic is never blocked.
factory.setCommonErrorHandler(kafkaErrorHandler); // retry + DLT for every listener on this factory
A business decline is NOT an error
This is the single most important rule. Only technical failures should retry and dead-letter; expected business outcomes must not. "Insufficient stock", "unknown SKU" and "payment declined" are normal results of the domain — the system already handles them by recording the outcome and emitting a fact the saga compensates off. If they were thrown, they'd be pointlessly retried, parked on the DLT as if they were bugs, and the saga would never hear about them.
try {
reservation = inventoryService.reserve(item, qty); // may throw
} catch (InsufficientStockException e) { // BUSINESS: record + emit,
reservation = Reservation.failed(..., "INSUFFICIENT_STOCK"); // never thrown, never dead-lettered
}
// anything else (a real technical fault) propagates -> error handler -> retry -> DLT
The idempotency trap: unclaim on a technical failure
There's a subtle interaction with the idempotency guard that will silently defeat the DLT if you miss it. The listener claims the order id up front (putIfAbsent) so a duplicate is skipped. But a retry is also a re-delivery of the same record — on attempt 2 the claim finds the id already taken, takes the duplicate-skip branch, and returns normally, so the record looks "handled" and never reaches the DLT. The fix: release the claim on a technical failure, then rethrow so the handler still sees the error.
if (!ledger.claim(orderId)) { log.info("duplicate — skip"); return; }
try {
... reserve + record + publish ... // business exceptions caught inside
} catch (RuntimeException ex) {
ledger.unclaim(orderId); // release so the RETRY genuinely re-processes
throw ex; // rethrow so DefaultErrorHandler retries with backoff, then DLTs
}
An @EmbeddedKafka test proves it end to end with no external broker: a poison record lands on order-placed.DLT after the retries (and the mock was invoked more than once, so it really retried), while a valid record processes exactly once and never appears on the DLT. The retry budget is externalized config (retry-attempts / retry-backoff-ms), so ops can retune per environment with no recompile. Retries + DLT are the natural partner of the outbox and of the idempotent consumers coming next.
Live walkthrough: https://dev48v.infy.uk/orderhub.php
Repo: https://github.com/dev48v/order-hub-from-zero