High latency with idle CPU is not a mystery — it's a diagnosis. A P1 post-mortem on connection pool starvation in Spring Boot.
A P1 post-mortem on connection pool starvation in Spring Boot, and the three unglamorous fixes that brought p95 latency from 4.2s back to single digits.
The alert nobody wants at peak hour
Traffic was normal. Around 2,000 requests per minute. No deploy had gone out that day.
Then the dashboards lit up:
- p95 latency: 180ms → 4,200ms
- HikariCP saturation alerts firing continuously (pool size: 10)
- CPU under 40% across every pod
- ~3% of responses returning orders with missing line items
- A mix of raw 500 stack traces and empty 200s going out to clients
That third bullet is the one that told us what kind of problem we had.
If the CPU is idle while latency explodes, nobody is computing anything. Threads are sitting still, blocked on I/O, waiting for something that isn't coming back fast enough.
So the question stopped being "what's slow" and became "what are we waiting on, and why are we holding a database connection while we wait."
Root cause: one endpoint, three anti-patterns
Everything traced back to a single controller method. Here is what was actually running in production:
@GetMapping("/orders/{userId}")
public List<OrderDTO> getOrdersByUser(@PathVariable Long userId) {
List<Order> orders = orderRepository.findByUserId(userId);
List<OrderDTO> result = new ArrayList<>();
for (Order order : orders) {
OrderDTO dto = new OrderDTO();
dto.setId(order.getId());
dto.setStatus(order.getStatus());
// Triggers N+1: lazy loading inside the loop
dto.setItems(order.getItems());
// A synchronous 350ms HTTP call. Inside the same loop.
dto.setPaymentStatus(paymentClient.getStatus(order.getId()));
result.add(dto);
}
return result;
}
It reads fine. That's the problem with this class of bug: it passes code review, it passes tests against a seeded database with three orders, and it falls apart the moment a real customer with real history hits it.
Issue 1: the N+1 query
findByUserId fetches the orders. Then order.getItems() inside the loop lazily initializes the items collection, one order at a time. For N orders, Hibernate issues N+1 queries.
At low traffic this is invisible. At 2,000 req/min it multiplies your query volume by roughly the average number of orders per user.
Issue 2: blocking I/O inside the loop
This was the real killer. paymentClient.getStatus(...) is a synchronous REST call to an external Payment Service, averaging 350ms round trip.
With a typical 15 orders per user:
15 × 350ms = 5,250ms
Five seconds of network wait, in a loop, per request.
Now combine the two. That entire loop was running inside an open transaction, which means each request held a HikariCP connection for the full five seconds while doing nothing but waiting on a socket.
With a pool of 10, the arithmetic is brutal. Ten concurrent requests consume the entire pool. Every request after that queues on connection acquisition and eventually times out.
This is why the CPU stayed flat: the service was not overloaded, it was blocked.
Little's Law makes it concrete:
concurrency = arrival rate × latency
When latency jumps from 180ms to 5s, in-flight requests grow by nearly 30×. No pool sized for the healthy case survives that.
Issue 3: no transactional boundaries, no centralized error handling
Because the lazy collections were sometimes touched outside an active transaction, some responses came back with uninitialized item lists. That's the 3% of orders missing line items.
And with no @ControllerAdvice, each endpoint handled its own failures. Some leaked stack traces, some swallowed the exception and returned an empty 200. Clients had no consistent way to tell success from failure.
The fix
Three changes. None of them clever.
1. Fetch the relation explicitly
Replace the derived query with a JPQL query using LEFT JOIN FETCH, so orders and their items come back in one round trip.
@Repository
public interface OrderRepository extends JpaRepository<Order, Long> {
@Query("SELECT DISTINCT o FROM Order o LEFT JOIN FETCH o.items WHERE o.userId = :userId")
List<Order> findByUserIdWithItems(@Param("userId") Long userId);
}
An @EntityGraph works equally well here if you prefer to keep the query derived.
2. Batch the external call, get it out of the loop
The Payment Service already had the capability to accept multiple IDs. We just weren't using it. One call replaces fifteen.
@RestController
@RequestMapping("/orders")
public class OrderController {
private final OrderRepository orderRepository;
private final PaymentClient paymentClient;
@GetMapping("/{userId}")
@Transactional(readOnly = true)
public List<OrderDTO> getOrdersByUser(@PathVariable Long userId) {
List<Order> orders = orderRepository.findByUserIdWithItems(userId);
if (orders.isEmpty()) return Collections.emptyList();
List<Long> orderIds = orders.stream().map(Order::getId).toList();
Map<Long, String> paymentStatuses = paymentClient.getStatusesInBatch(orderIds);
return orders.stream().map(order -> {
OrderDTO dto = new OrderDTO();
dto.setId(order.getId());
dto.setStatus(order.getStatus());
dto.setItems(order.getItems());
dto.setPaymentStatus(paymentStatuses.getOrDefault(order.getId(), "UNKNOWN"));
return dto;
}).toList();
}
}
Two things worth calling out.
getOrDefault(..., "UNKNOWN") means a partial failure in the payment lookup degrades one field instead of failing the whole response. And @Transactional(readOnly = true) gives us a defined boundary, so lazy access happens inside a live session rather than by luck.
If you can't batch on the downstream side, the fallback is parallel calls on a bounded executor. The rule you cannot break is the one about holding a database connection while you wait on the network.
3. Centralize exception handling
@RestControllerAdvice
public class GlobalExceptionHandler {
@ExceptionHandler(Exception.class)
public ResponseEntity<Map<String, String>> handleAllExceptions(Exception ex) {
log.error("Unhandled exception", ex);
return ResponseEntity
.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body(Map.of("error", "INTERNAL_SERVER_ERROR",
"message", "An unexpected error occurred."));
}
}
Log the full detail on our side, return a stable shape to the client. No more stack traces on the wire, no more silent empty 200s.
Making it hold under burst: Redis
The code fix solved the incident. It didn't answer the next question, which was what happens when traffic goes past 2,000 req/min.
Order history is read-heavy and tolerant of being slightly stale, which makes it an easy caching target. We put @Cacheable on the read path with a 60-second TTL.
Sixty seconds was chosen deliberately. Long enough to absorb the repeated reads that dominate this endpoint, short enough that a status change surfaces well within what users notice.
Results
| Metric | Before | After |
|---|---|---|
| p95 latency | 4,200ms | < 5ms on cache hit |
| HikariCP pool usage | Saturated at 100% | Under 15% at peak |
| DB query pressure | Baseline | Down ~85% |
| Responses with missing items | ~3% | 0 |
| Error responses | Inconsistent 500s / empty 200s | Consistent structured errors |
What I'd tell any Spring developer after this
Never put network or database I/O inside a loop. Batch it, or make it non-blocking. This one rule would have prevented the entire incident.
A blocked thread holding a connection is worse than a slow one. Long I/O inside a transaction is how you exhaust a pool with modest traffic and idle CPUs. Do your external calls outside the transactional boundary.
Fetch explicitly when you're mapping to DTOs. JOIN FETCH or an entity graph. Never let lazy loading fire inside a mapping loop.
Low CPU with high latency is a diagnosis, not a mystery. It means blocking I/O. Go look at your pools and thread dumps, not your algorithms.
Centralize your error handling on day one. @ControllerAdvice costs ten minutes and saves you from leaking internals during the exact incident where you least want to be leaking internals.
Test with realistic data volumes. Every one of these bugs is invisible with three seeded rows and obvious with fifteen.
Have you hit connection pool starvation or thread exhaustion in production? I'm curious which anti-pattern caused it, because in my experience it's almost always this same shape wearing a different hat.