How a five-minute retry delay in an event-driven pipeline turned into a lesson on transaction boundaries — and a 78% drop in event reprocessing.
Before
pod A
pod B
row 402
Both pods write at once → unique-constraint violation → 5-minute retry.
After
pod A
pod B (waits)
row 402
Row lock serializes access — the second pod waits, then sees it's already claimed.
The setup
At Infosys, I work on an event-driven pipeline that reacts to user actions across services. Because we run multiple pods for resiliency, a Message Relay Service polls an event table every 10 seconds looking for freshly captured events to push onto AWS SQS for downstream consumers. Simple in theory — until you run more than one pod.
Where it broke
Two pods, same 10-second poll window, same unprocessed row. Both pick it up. Both try to mark it "in flight" and update its status. One update lands; the other collides with a unique constraint and throws a SQL exception. The event doesn't just retry immediately — it falls into a five-minute backoff, because that window was tuned for a completely different failure mode: transient network errors calling a downstream API. A benign race condition was paying the same tax as a real outage.
Multiply that by the volume of events flowing through the table, and a meaningful slice of "processed" events were either being processed twice or sitting in an unnecessary five-minute queue.
Why the obvious fix doesn't work
The instinctive first move is an application-level check: read the status, see it's UNPROCESSED, then update it. But between the read and the write there's a window — and in a system with more than one pod, something else can slip into that window. It's a textbook time-of-check-to-time-of-use gap, and no amount of if statements inside a single JVM closes it, because the race isn't inside one JVM. It's across two.
What actually closed the gap
The fix wasn't cleverer application code — it was moving the guarantee down to the row itself, with a pessimistic lock:
EventRepository.java
public interface EventRepository extends JpaRepository {
@Lock(LockModeType.PESSIMISTIC_WRITE)
@Query("select e from EventEntity e where e.id = :id")
Optional<EventEntity> findForProcessing(@Param("id") Long id);
}
EventProcessingService.java
@Transactional
public void processEvent(Long eventId) {
EventEntity event = eventRepository.findForProcessing(eventId)
.orElseThrow();
if (event.getStatus() != EventStatus.UNPROCESSED) {
return; // another pod already claimed it
}
event.setStatus(EventStatus.PROCESSING);
eventRepository.save(event);
publishToQueue(event);
event.setStatus(EventStatus.PROCESSED);
eventRepository.save(event);
}
PESSIMISTIC_WRITE translates to a SELECT ... FOR UPDATE under the hood. Whichever pod's transaction gets there first holds the row lock until it commits or rolls back; the second pod's transaction simply waits — and when it finally gets the row, it sees the status has already moved to PROCESSING and quietly backs off instead of colliding with a constraint violation.
The lock scope matters here: it's held only for the width of that one row's processing logic, not for the whole batch, so throughput on unrelated rows is untouched.
What changed
Event reprocessing dropped 78% — most of it was this exact race, not real downstream failures.
The five-minute retry delay disappeared for this class of event entirely, since there was no longer an exception to trigger it.
On-call noise dropped too — fewer "why did this event process twice" questions to chase down.
When I'd reach for this again
Pessimistic locking isn't free — a held row lock is a held row lock, and under high contention it can turn into a queue of waiting transactions. It earns its place when contention is rare but the cost of a collision is real (duplicate side effects, constraint violations, silent data drift), and when the locked section stays small. If contention is frequent and collisions are cheap to detect and retry, optimistic locking with a version column is usually the better trade. Here, with a handful of pods racing over the same few rows for a few hundred milliseconds at a time, pessimistic locking was the simpler, more predictable fix.