Why Splitting Logs by Business Key Becomes a Resource-Management Problem

I wanted to make "what happened to player-12345?" easy to answer. It turned out that routing was the easy part.

1. The Problem

Some player, order, or tenant breaks, and I need to know what it was doing before it broke.

The database tells me its state right now. The logs should tell me how it got there. But the logs aren't organized by player, order, or tenant. It's spread across dozens of outputs by logger and package, mixed in with thousands of unrelated objects. I just want the timeline for player-12345 — and instead I'm grepping through a pile of files.

2. Traditional Logging Organizes Around a Different Dimension

There's nothing wrong with Logback's or Log4j2's logger hierarchy. It answers a specific question — "which class or package did this log line come from?" — and it does that well.

The problem is that wasn't the question I was asking. I was asking "which business object does this log line belong to?" Those are two different dimensions. The usual pipeline, logger → appender → output, is built around code structure. I needed a different one.

3. What If the Key Became the Routing Dimension?

What if each log event carried a business key, and that key decided where the event goes?

log event → key → routing → output

Same key, same destination. "One file per order" or "one directory per tenant" becomes a routing rule instead of something I have to hand-roll.

Log4KeyLogger logger = Log4KeyLoggerFactory.getLog4KeyLogger(Demo.class);
ILogKey key = DefaultLogKey.of("player-12345");

logger.info(key, "login success");
logger.info(key, "place order {}", "order-10086");

Run this and a player-12345.log shows up under logs/. To answer "what happened to player-12345," I open one file.

A quick note on MDC: MDC.put attaches a value to the event as metadata, for filtering later. A key here is routing information — it decides which output the event goes to. The difference isn't where the value is stored. It's what the value is used for.

4. The First Surprise: High Cardinality

At first this felt almost too obvious to write about. One key, one file. Just write it.

Then I kept increasing the number of keys:

100
1000
10000
...

The problem showed up fast. You can't leave every key holding an open file — a process can only have so many files open at once. With enough keys, you have to reuse: close the idle ones, open new ones. Once access becomes sufficiently scattered, opening and closing files can become a frequent part of the log-processing path.

At that point, "split logs by key" was no longer a routing problem. It was a problem of managing files, file descriptors, caches, and concurrency.

5. How I Ended Up Implementing It

It converged on a few decisions:

• The application threads only format, route, and shard events, then hands it off. Anything that touches a file — open, close, buffer, flush, evict — lives on the worker side, so business threads don't get dragged down by I/O.
• Events for the same key are routed deterministically to the same worker, so one key's logs are handled in arrival order, while different keys can run in parallel.
• maxOpenFiles caps how many files are open at once. When the active set exceeds that budget, an LRU policy reuses those resources — hot keys keep their handles, while idle ones give them up.
• Writes follow append → write (batched) → flush, collapsing a lot of small writes into fewer batched ones.

None of this changes what the disk can actually sustain. What it does is keep resource exhaustion contained within the logging layer, instead of letting it take down the app.

6. What the Benchmark Actually Shows

Environment (all from the JMH report): JDK 1.8.0_481, JMH 1.37, 20 cores, Windows 11. Absolute numbers don't matter here; the behavior change does.

Raising the key count:

At 200 keys there's no rejection. At 210, rejection shows up — mailbox full, events dropped — and write throughput falls off noticeably. The interesting part isn't the absolute throughput; it's the cliff. Everything is fine up to a certain working-set size, then rejection appears and throughput and actual I/O drop together. Based on the implementation, this appears consistent with the active key set exceeding the open-file budget.

Now push to 1000 keys and vary only the worker count:

Going from 1 to 4 workers barely moves the production rate. Actually-consumed events climb from 2.4K to 5.2K, but rejection stays near 16K. More parallelism improved consumption a little; it didn't remove the file-management and I/O pressure in this workload.

7. When This Model Makes Sense

The benchmark results come back to a plain business judgment: one key per file isn't the right call everywhere.

It probably fits when you're debugging one player, order, or tenant at a time; when that object's logs only make sense as a complete timeline; and when physically separating them helps you isolate, audit, or export.

It probably doesn't fit when your main workflow is centralized search; when keys are extremely high-cardinality and mostly cold, with each key logging only occasionally; or when a single key's file has little value. In those cases, physical splitting may just be the wrong answer.

8. Why I Built It

I started from a simple idea: if I could follow one business object through its logs, debugging would get a lot easier. Actually building it turned out to be much more complicated than I expected — the hard part wasn't the splitting, but managing files, descriptors, and I/O afterward.

That's what became Log4Key.

• GitHub: https://github.com/log4key/log4key
• Maven: com.log4key:log4key-all (use the latest version on Maven Central)
• The JMH code and results are in the repo — feel free to reproduce them or poke holes.

9. How Do You Solve This?

How do you usually debug the history of a single player, order, or tenant? Do you use MDC with centralized logging, separate files, custom routing, or something else? I'm genuinely curious how others handle this.