I maintain nopanic, a small resilience toolkit for Python (retries, circuit
breakers, timeouts, rate limits). Everything passed its tests, so I did what
I should have done earlier: I put it under real load. The result taught me a
lesson worth writing down.
The setup
A circuit breaker watches the failure rate of a dependency. If too many
calls fail inside a time window, it "opens" and fails fast instead of
hammering something that is already down. To know the failure rate, it has
to remember recent outcomes.
My first implementation remembered them the obvious way: a list of
(timestamp, ok/fail) records inside a sliding time window. On every failure
it counted how many entries in that list were failures, divided by the
total, and compared to the threshold.
Correct. All tests green. Shipped.
The load test
I wrote a stress test: 32 threads, 100,000 calls each, one shared breaker,
about a third of the calls failing (below the trip threshold, so the breaker
stays closed and keeps recording). 3.2 million calls total.
It took 2,413 seconds. Forty minutes. Single threaded, the breaker does
roughly 850,000 calls per second, so those calls should have finished in
about four seconds.
The diagnosis
Two problems, one root cause.
First, memory. The window kept one record per call. At high throughput that
list holds every call inside the window. I measured it: one million calls in
the window was about 88 MB, for a single breaker. It grows with traffic,
without bound.
Second, and worse, time. On every failure the code recomputed the failure
count by scanning the entire list. As the list grew, each scan grew with it.
N failures, each doing O(N) work, is O(N squared). Under sustained partial
failure the whole thing collapses into a quadratic crawl, and because the
scan happens inside the breaker's lock, all the threads pile up behind it.
That is the forty minutes.
The realization
The fix was not a faster scan. It was noticing that a circuit breaker's
memory is not a log. It only needs to answer one question: what is the
failure rate right now. It never needs the individual records. I was storing
a detailed history to compute a single ratio.
Detailed per-request history is a real need, but it belongs in an
observability stream that the user forwards to their own logging, not in the
breaker's hot path. Keeping millions of records in RAM to compute one
percentage was the actual bug.
The fix
Replace the per-call list with a fixed set of time buckets, each holding two
integers: how many calls, how many failures. Keep running totals updated as
buckets rotate out of the window. This is how Hystrix and resilience4j do
it, and now I understand why.
- Recording an outcome: O(1).
- Reading the failure rate: O(1), from the running totals.
- Memory: constant, no matter the traffic. Ten buckets of two integers.
The same 3.2 million call load now finishes in about 2.4 seconds instead of
40 minutes. One million in-window calls use 0.001 MB instead of 88. The
per-call cost on the success path even dropped slightly.
The trade-off: outcomes now expire in bucket-sized steps instead of at an
exact per-call age. For deciding whether to trip a breaker, that precision
was never worth its price.
Two lessons
First, tests that check correctness do not check behavior under load. My
suite was green the entire time the library was quadratic. A partially
failing dependency at high traffic, the exact situation a circuit breaker
exists for, was the situation that broke it.
Second, a bonus one I tripped over while fixing this: I was testing with an
editable install, and a stale copy of the package in site-packages was
shadowing my source. My tests were passing against old code. Now the CI also
builds the wheel, installs it into a clean environment, and runs the suite
against the installed artifact. Never fully trust pip install -e alone.
The library is nopanic (pip install nopanic) if you want to see the code