This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
The most expensive bug I fixed this year was not in the code. It was in the documentation, and it had been shaping what everyone believed the code did.
The setup
HydraDNS is an open-source DNS security gateway I build in Go. Router points at it, it filters every DNS query on the network against a 92k-domain blocklist, blocks the bad ones, forwards the rest. Before putting it on anyone else's network I wanted a real number for what one box could take, so I sat down with dnspyre and a rule I had written for myself: every number becomes a sales claim or a fix ticket. No number, no claim.
Our feature sheet said the blocklist was backed by a Bloom filter, sub-millisecond lookups. Here is the uncomfortable part: at every load this system had ever run, that claim was indistinguishable from the truth. Normal-traffic latency sat at one or two milliseconds. There was nothing to doubt, because nothing observable disagreed.
The first ceiling
The redline test capped at about 500 queries per second. Odd, but fine, until I noticed the cap would not move. Blocked queries capped at ~500. Cached queries that never touch upstream also capped at ~500. Two paths doing completely different work, same wall, CPU sitting under 30% on a 22-core dev machine.
That combination is worth memorizing: when two very different code paths hit the same ceiling and the CPU is bored, the bottleneck is not in either path. It is in something they share.
Ours was the blocklist check. IsBlocked ran a SQL COUNT against the 92k-row table on every single query, because the check sits in front of the cache, so even cache hits paid for it. Every one of those reads was serialized through a single SQLite connection, MaxOpenConns=1, which was also absorbing the async write traffic from query logging. Engine self-latency under load: p50 of 50ms, p99 of five full seconds. For DNS.
And the Bloom filter? I went looking for it so I could tune it, and found a misattribution rather than a fabrication. The codebase does have a Bloom filter, a real one with tests, in the policy engine next door. Somewhere across three documents, "the policy engine uses a Bloom filter" drifted into "the blocklist uses a Bloom filter" and started reading like established fact. Every sanity check, mine included, recognized a component that genuinely exists and moved on.
Here is what makes this class of bug nasty: reading the blocklist code would not have flagged it either. The SQL path was correct code doing exactly what it claimed. Code review asks whether an implementation is right, and it was. No review asks whether an implementation matches an adjective in a feature sheet. The only reviewer that can ask that question is a measurement at redline, and that is precisely why the stress plan existed.
The fix, which is embarrassingly boring
I built the thing the docs had been imagining, minus the cleverness: MemoryChecker, an atomic in-memory domain set, loaded at startup, swapped wholesale on the existing 6-hour blocklist refresh. The parent-domain walk (block example.com, catch ads.example.com) reproduces the exact candidate set the SQL version checked, so behavior stayed identical. Only the cost changed.
Throughput: ~500 to ~9,500 QPS. Nineteen times, from taking one membership check off the database. Not a probabilistic data structure. A map.
The second ceiling was waiting behind the first
Re-ran the redline. The box got fast, then started eating memory: RSS climbed from 98MiB to a little over 2GB as load went to 10,000 QPS. It drained back when load stopped, so not a leak, a backlog. But the target hardware for this thing has 2 to 4GB of RAM total. That transient is an OOM kill in the middle of a traffic burst, which is the exact moment a DNS server must not die.
Query logging spawned a goroutine per query, each doing an INSERT plus a stats UPDATE through that same single connection. At 500 QPS the queue drained and nobody noticed. At 10,000, goroutines piled up faster than one connection could ever clear them. The first bottleneck had been rationing the second. Lift one ceiling and the next bomb arms itself.
The replacement: one writer goroutine, a bounded channel of 4,096 entries, batches flushed at 256 entries or 500ms, bulk INSERT plus one aggregated stats UPDATE per batch. Enqueue is non-blocking, and when the buffer is full the entry is dropped and the drop is counted, because losing a log line is acceptable for a DNS server and stalling resolution is not.
Before and after
Throughput ceiling: ~500 -> ~9,500 QPS (19x)
Engine p99: 5000ms -> 20ms
RSS at 10k QPS: 2,063MiB -> ~85MiB
CPU at 10k QPS: 56% -> 14%
Confirmed with a soak: 248,166 queries at 1,379 QPS, zero engine errors, memory a bounded sawtooth instead of a climb. All numbers from a 22-core dev machine with load generated in-container, and I label them that way everywhere, because the other lesson of this story is what happens to unverified claims.
What I learned
- Same ceiling on two different paths plus an idle CPU means stop optimizing the paths and find what they share.
- Fixing one bottleneck unmasks the next. The box's real limit is wherever you stop looking.
- Unbounded "async" is not async, it is an OOM with a delay on it. Bound every queue and count every drop.
- The one I keep thinking about: some claims are empirically true at every load you have run and false at the load you have not. They survive code review, doc review, and months of correct operation, because nothing observable disagrees with them. The only experiment that can falsify them is a redline test, which is what a stress plan is actually for. Not big numbers for a landing page. Falsification.
The longer write-up, with the full latency data and the test harness bug that almost let the soak pass on an idle server, is here: https://dev.to/lopster568/two-ceilings-taking-a-go-dns-server-from-500-to-9500-qps-2poj