Don t Lock Your API - Lock Your Scheduler Instead

How inverting responsibility eliminated distributed lock contention from a high-throughput settlement pipeline

When building the ingestion layer of a real-time interbank settlement pipeline, I faced a classic distributed systems problem: how do you guarantee that all orders belonging to the same settlement window are grouped together before being published to Kafka, without killing your API throughput?

The naive answer involves a distributed lock on the hot path. I went a different direction.

The Problem

The system receives settlement orders via a REST API built with Spring Boot 3.5 and Java 21. Orders must be grouped by settlement window (a time-based boundary, e.g., 17:00) before being published as a batch to Kafka. The consumer downstream expects a complete, immutable batch, not a partial one.

The constraint: orders can arrive at any rate. The window must close atomically. No order should be published before the window closes, and no order from a closed window should sneak into the next batch.

The Naive Approach And Why It Fails

The first instinct is to coordinate at ingestion time:

POST /orders → acquire distributed lock → check window → persist → release lock

This works at low volume. At scale, it becomes your bottleneck:

Every incoming request competes for the same lock
Lock contention grows with throughput
Latency spikes under load, exactly when you need stability most
Redis or ZooKeeper become single points of failure on the critical path

You've introduced coordination overhead into the one place that should be as fast as possible: the API entry point.

The Inversion

Instead of coordinating at ingestion, I moved all coordination out of the hot path entirely.

The architecture splits responsibility across two distinct phases:

Phase 1 — API: Persist and Forget

@PostMapping("/orders")
public ResponseEntity<Void> receiveOrder(@RequestBody SettlementOrderRequest request) {
    settlementOrderService.persist(request, OrderStatus.PENDING);
    return ResponseEntity.accepted().build();
}

The API does exactly one thing: persist the order with status PENDING. No window check. No lock. No coordination. Just a fast write via HikariCP and a 202 back to the client.

Phase 2 — Scheduler: Close, Batch, Publish

The scheduler entry point iterates over all participants. Before the loop, the JPA first-level cache is cleared to avoid stale reads:

@Scheduled(cron = "0 0 17 * * MON-FRI")
@ShedLock(name = "settlement-window-close", lockAtMostFor = "PT10M")
public void closeWindowAndPublish(String windowKey) {
    entityManager.clear(); // flush JPA first-level cache before processing

    List<Participant> participants = participantPort.findAll()
        .stream()
        .filter(p -> p.getType() != ParticipantType.BACEN)
        .toList();

    for (Participant participant : participants) {
        try {
            processor.process(window, today, participant);
        } catch (Exception e) {
            log.error("Critical error processing participant [{}]. Skipping.", participant.getIspb(), e);
        }
    }
}

Each participant is processed in complete isolation via REQUIRES_NEW, so a failure in one does not roll back the others:

@Transactional(propagation = Propagation.REQUIRES_NEW)
public void process(SettlementWindow window, LocalDate today, Participant participant) {

    if (!window.isOpen(clock)) {
        rejectPendingOrders(window, today, participant); // explicit cutoff rejection
        return;
    }

    if (batchPort.existsActiveBatch(window, today, participant.getId())) {
        log.warn("Batch already exists for window [{}] participant [{}]. Skipping.",
            window.getPartitioningKey(), participant.getIspb());
        return;
    }

    List<SettlementOrder> pending = orderPort.findPendingForWindow(window, today, participant.getIspb());
    if (pending.isEmpty()) return;

    List<SettlementOrder> batched = pending.stream().map(SettlementOrder::batch).toList();
    FileBatch savedBatch = batchPort.save(new FileBatch(window, today, batched, participant));
    orderPort.updateStatusBatch(batched.stream().map(o -> o.associateWithBatch(savedBatch.getId())).toList());

    // Kafka is fired only after the transaction commits successfully
    TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
        @Override
        public void afterCommit() {
            publisherPort.publish(savedBatch);
        }
    });
}

The scheduler owns the window boundary. It runs once per window, outside the request lifecycle, under zero user-facing load pressure. ShedLock with Redis (SET NX PX) ensures only one instance executes across the entire ECS/EKS cluster, but this lock fires once per window, not once per request.

Four Details That Matter

1. entityManager.clear() before the loop

Some operations in the batching flow rely on JPQL bulk updates. Since JPQL bulk updates bypass Hibernate's managed entities and go directly to the database, the persistence context may contain stale state after those updates, Hibernate still believes it holds the correct version of those entities in memory. Clearing the persistence context before processing ensures that subsequent reads are served from the database rather than from outdated managed entities.

2. REQUIRES_NEW per participant

Each participant gets its own isolated transaction. If processing fails for participant A, participants B through N are unaffected and their transactions commit independently. Without this, a single failure rolls back the entire batch cycle.

3. afterCommit() before publishing to Kafka

This is a correctness guarantee, not a performance optimization. Publishing inside the transaction would risk sending a Kafka message for a batch that never actually committed to the databas a phantom message that consumers would try to process against data that doesn't exist. afterCommit() ensures the database write is durable before Kafka ever sees the event.

4. Explicit rejectCutoff() for late orders

Orders that arrive after the window closes are not silently ignored, they are explicitly marked as REJECTED_CUTOFF. This makes the system auditable: at any point you can query exactly which orders missed which window and why.

Why This Works: The Key Insight

The distributed lock didn't disappear, it moved.

Concern	Naive Approach	Inverted Approach
Lock location	API hot path	Scheduler (cold path)
Lock frequency	Every request	Once per window
Lock contention	High under load	Zero
API latency	Unpredictable	Consistent
Throughput impact	Degrades with volume	None

ShedLock on a scheduler that fires once per settlement window has negligible throughput impact. ShedLock on an API endpoint receiving thousands of requests per minute does not.

Kafka Partitioning by Window

Once the batch reaches Kafka, ordering guarantees are maintained by partitioning on the window key:

kafkaTemplate.send("str.batch.emission", batch.windowKey(), batch);
// windowKey: "STR-D0-17h00", "STR-D1-17h00", etc.

Each settlement window maps to a dedicated partition. This guarantees temporal ordering without any additional locking mechanism. Kafka's own partition semantics do the work.

Status Flow

The order lifecycle makes the phase separation explicit:

PENDING  →  BATCHED  →  EMITTED
  (API)    (Scheduler)  (Consumer)
             ↓
       REJECTED_CUTOFF
      (window closed)

Each transition is unambiguous. At any point you can query exactly where in the pipeline an order is and why it got there.

What About Late Orders?

Orders that arrive after the scheduler closes the window are not silently dropped or deferred, they are explicitly rejected with status REJECTED_CUTOFF. This is intentional on two levels: the STR protocol requires complete, bounded batches, and a silent failure is always worse than an explicit one. The system knows exactly which orders missed which window, and so does the operator.

Takeaway

If you find yourself reaching for a distributed lock on an API hot path to coordinate time-based grouping, ask whether the coordination can be deferred to a scheduled process instead.

Moving the lock from the hot path to the cold path cost nothing in correctness and eliminated the primary throughput bottleneck. The API became a pure write endpoint. The scheduler became the sole owner of window semantics.

Lock the thing that controls the boundary. Not the thing that feeds it.

This article is part of a series on the STR-XML-Pipeline, a high-throughput interbank settlement system built with Spring Boot 3.5, Java 21, Apache Kafka, PostgreSQL 16, Redis 7, and AWS S3 & Fargate.