I built a small training log in Go called go-gym. Two front doors over one backend: a JSON API and a server-rendered HTMX UI. Nothing exotic. It had been sitting there with a green CI for weeks: build passes, tests pass, lint passes.
Last week I added a /metrics endpoint, got curious about the numbers behind it, and pointed a load generator at the dashboard to see the p95.
It didn't get slow. It fell over.
The setup was hey, 150 concurrent connections, 15 seconds, against GET /app. The dashboard is the heaviest read in the app: a stats aggregate, a page of workouts, and the activity heatmap counts. I expected mediocre latency. Instead the worst request took 13.5 seconds and a big chunk of them came back as errors.
The server log said why:
FATAL: sorry, too many clients already (SQLSTATE 53300)
Postgres was refusing connections. 321 of them across the run.
The part I'd never thought about was the database setup, which was one line:
db, err := sql.Open("pgx", dsn)
database/sql hands you a connection pool for free, and I'd been quietly assuming it was sensible out of the box. It isn't. SetMaxOpenConns defaults to zero, and zero means unlimited. So under load, every query that can't grab an idle connection just opens a new one. At 150 concurrent requests each running a few queries, the pool opened connection after connection until it walked straight into Postgres's max_connections, which is 100 by default. Past that, Postgres says no, and the handler returns a 500.
The /metrics endpoint I'd added the day before turned out to be useful, not as the thing under test but as the instrument. I could watch the latencies climb while the error rate went up. The bottleneck wasn't the query. It wasn't a missing index, which is what I'd have guessed if you woke me up and asked. It was that I was opening hundreds of connections to a server that wanted a hundred.
The fix is four lines:
db.SetMaxOpenConns(25)
db.SetMaxIdleConns(25)
db.SetConnMaxIdleTime(5 * time.Minute)
db.SetConnMaxLifetime(time.Hour)
25 sits well under Postgres's 100, with room to spare for migrations, a background job, and anything else holding a connection. The first line is the one that matters. Once the pool is capped, extra requests don't open new connections. They wait for one to free up. The queue moves to the client side, where waiting is cheap, instead of the database side, where it's fatal.
You might be reaching for the reply: just use pgxpool. Fair. Its native pool defaults to a bounded size, so it wouldn't have blown up this way out of the box. But the store and the migrations (goose) are both built on database/sql, so switching isn't free, and you'd still have to size the pool for your database either way. The point isn't which pool you pick. It's that you have to know its default and set the bound. database/sql just happens to ship the dangerous default.
Same load test after the change:
GET /app, c=150, 15s |
before | after |
|---|---|---|
| failed requests | 321 | 0 |
| p95 latency | requests were erroring out | 32 ms |
| p99 latency | 92 ms | |
| throughput | ~8,400 req/s |
Before the fix, the worst request sat at 13.5 seconds before it gave up. After, the slowest 1% came back inside 92 milliseconds and nothing failed. One run, same machine, same dataset, so it's a clean comparison of the one thing I changed.
Let me be honest about what this is. It's hey on my laptop against a local Postgres with a small dataset, so the absolute numbers don't mean much. What's real is the failure mode. An unbounded pool will exhaust any connection-limited database the moment your concurrency is higher than the number of slots the database has, and it picks the worst possible time to do it, which is when you have traffic.
Two things stuck with me.
go build and my CI were green the entire time. Compiling tells you the types line up. It tells you nothing about what happens at 150 connections. The only reason I found this is that I ran the thing under load, which I did almost by accident.
And I would have guessed wrong. My reflex for "slow under load" is "add an index" or "there's an N+1 in there somewhere." The queries were fine. The pool was the problem, and I only knew that because the error said so, in plain words, in the log line up above. Measure the actual thing. Your instinct is a hypothesis, not an answer.
The four lines live in database.go, and there's a scripts/loadtest.sh in the repo if you want to reproduce the run. The whole project is at github.com/Aejkatappaja/go-gym. Set your pool limits. The default is a trap, and it stays quiet right up until it isn't.