The vendor documents this bug. A 30k-star repo shipped it anyway.

This is part three of a series about pointing an append-only audit log at things that count tokens. Part one found that a single missing model line was half my AI agent overspend. Part two found two accounting bugs in splitrail, both of which undercounted — one hid 54% of my messages.

This one points the other way. The target is claude-code-templates — 30.1k stars, 3.3k forks, the largest Claude Code template repo — and its analytics over-counted my tokens by 2.35×.

The part that changed how I wanted to write this up: while drafting the report I went looking for whether the behaviour was documented anywhere. It is. Anthropic's own SDK documentation has a warning box about it, and ships a reference implementation of the fix. The bug was not hiding. Four separate mechanisms surrounded this code and none of them closed the loop, and the reason they didn't is the actual subject of this post.

The fix has now been open for ten days. Two AI reviewers approved it. No human has replied. More on that near the end, because the silence turned out to carry information too.

(Part four is the stated-vs-revealed routing analysis I promised at the end of part two. It got bumped. This was worth the detour.)

The bug

calculateRealTokenUsage() summed message.usage over every parsed record.

Claude Code writes a single assistant message as several JSONL records — one per content block (thinking / text / tool_use) — and every one of those records repeats the same message.id and a byte-identical usage object. So each message's tokens were counted once per block, and everything downstream inflated by the average block count: conversation.tokens, summary.totalTokens, the per-project rollups, and the Total Tokens figure on the analytics dashboard.

Measured against my frozen ~50-day corpus:

Three records per message is the mode — thinking, then text, then a tool call.
Two-thirds of my assistant messages were being counted more than once, and on
every single one of them the repeated usage was byte-identical, which is the
detail that makes the over-count a multiplication rather than a rounding error.

Then I drove the analyzer over a synthetic corpus with known-exact totals:

The line that turns this from a hunch into a diagnosis: before the fix, every field equalled the per-record sum exactly. Not approximately. Exactness is what pins the cause to record-level summation rather than to anything in parsing, and it is why the report could name the defect instead of reporting a discrepancy.

There is also a check that needs no corpus, no fixture and no golden file, which is the part I'd want you to take away: messagesWithUsage should equal the number of distinct message.ids. It equalled the number of records. One assertion, available from day one, would have failed loudly.

It was in the docs the whole time

Here is what I found when I went to check whether this was known. From Anthropic's Agent SDK cost-tracking guide, in a callout box:

Parallel tool calls produce multiple assistant messages whose nested BetaMessage shares the same id and identical usage. Always deduplicate by ID to get accurate per-step token counts.

And in the prose above it:

When Claude uses multiple tools in one turn, all messages in that turn share the same ID, so deduplicate by ID to avoid double-counting.

The page ships a working seenIds implementation. Its explanatory diagram is captioned, in part, "Step 1 has four assistant messages sharing the same ID and usage (count once)." There is a public issue on the Claude Code repo with the symptom in its title — #6805, "Token Usage Statistics Duplicated in stream-json Mode Causing Massive Cost Inflation". And codeburn, another tracker in this space, documents its own global seenMsgIds set doing precisely what the SDK page recommends.

So this is not an obscure edge case. The SDK that produces the data tells consumers, in a warning box, to deduplicate by ID. A widely used consumer of that data summed per record instead, for as long as the function existed.

I want to be exact about what my audit log did and did not contribute here, because the distinction is the whole point and I would rather make it myself than have it made at me: an append-only log was not necessary to discover this. Reading the documentation would have done it. What the log contributed was three things the documentation cannot give you — whether this particular codebase had the defect, what it cost (2.36×, on 8,123 real messages), and proof that the fix was exact rather than merely different.

Documentation describes hazards. Measurement finds instances. Those are different verbs and the gap between them is where this bug lived.

What the two AI reviewers said

The PR went through two commercial AI code reviewers. Both cleared it. Neither mentioned the over-count.

Greptile posted a summary and a verdict:

Confidence Score: 5/5 — The PR appears safe to merge. No blocking failure remains.

It reviewed both commits, and left the only 👍 the PR has received.

cubic ran twice. Its first pass reported "AI review completed with 2 reviews. Found 2 issues across 3 files" — a maintainability note about the function being duplicated across two files, and this:

P3: Distinct records can still be merged when an id equals a generated positional key or another record's uuid, causing token undercounting for that conversation. Namespace persisted identifiers and use a non-colliding generated key for id-less records.

That is a fair catch and it deserves credit. It is a real hazard: ConversationAnalyzer.js:297 sets id: item.message.id || item.uuid || null, so a uuid can already arrive in the id field, and my fallback chain put id, uuid and a positional key in a single namespace. On real Claude Code data the probability is approximately zero. The cost to eliminate it was four lines — namespace each source (id: / uuid: / idx:) — plus a test that goes red before the change and green after. cubic's second pass dropped to "1 issue found across 3 files."

Now the observation, which I want to state as flatly as I can.

Both tools read this function closely enough to reason about a hypothetical key collision in a line I had written minutes earlier. Neither remarked that the function's output had been wrong by a factor of two since the day it was written.

That is not a miss. By the standard each was applying, both answers were correct. A diff-scoped reviewer answers "is this change safe and self-consistent?" The pre-existing error was not in the diff, and nothing in a review context contains an independent measurement of what the right number is.

The asymmetry is worth naming precisely, because it generalizes past these two products: cubic's finding was a hypothesis about the shape of code. The bug was a discrepancy between a number and reality. Only the second kind of claim requires evidence from outside the repository. A reviewer holding the entire codebase in context still would not catch this one, because the code is internally consistent — it does exactly what it says it does, and what it says is wrong. You cannot read your way to that conclusion. Somebody has to measure.

The hole nobody is looking at

While I was in there: this repo runs no test suite on pull requests. The only check on #754 is the AI reviewer. "Eight unit tests pass" is a sentence I had to type into a comment, because nothing in the pipeline can demonstrate it.

With 135 open pull requests and 86 open issues, that is capacity, not negligence. But it is exactly the condition under which a silent numerical defect lives indefinitely, and it reorders the priorities: two AI reviewers and no test CI is a weaker correctness story than no AI reviewers and one test that asserts a total against a known input. Reviewers are additive. They are not a ground truth.

Ten days later

As of publication, #754 has been open ten days. Every check is green, the branch merges cleanly, and the two AI-reviewer verdicts haven't moved since day one. Human words on the thread: zero. No label, no assignee, no milestone. The one open review question — which of the two duplicated copies of the function should survive — can only be answered by a maintainer, and hasn't been. In those same ten days the repo gained about 200 stars and its open-PR queue grew from 129 to 135.

The same ten days, elsewhere in this series:

• splitrail, 216 stars. I filed #220, a streaming-snapshot over-count, with the inflation ratios predicted from my corpus before any fix existed. A fix matching the prediction on every field was merged within two hours. Third shipped fix in that repo for this series.
• tokscale, 4.6k stars. My drift report #994 was fixed and released in v4.9.0. From the maintainer's closing note: "The measurement in this report is what made it actionable — the before/after table landing exactly on the drifted prediction pinned it to persistence rather than parsing, and saved us looking in the wrong place."
• viberank. In an issue I filed about leaderboard drift, yoo-minho (he maintains the submission tool for a rival leaderboard, clauderank) confirmed the same class at a scale I can't generate: a month of roughly $15.5k list-price usage whose month-to-date total fell 11% between two submissions 16 hours apart. His words: "A cumulative month-to-date total went down, which shouldn't be possible." He then wrote the countermeasure up as usage-drift-log, a six-field append-only record, and traceguard now implements it verbatim. One page of spec, two independent implementations.

So the method is not slow, and maintainers are not indifferent. Response time tracked queue depth, nothing else: the 216-star repo answered in hours, the 4.6k-star repo in days, and the 30.1k-star repo has 135 pull requests ahead of mine.

Which corrected something I had wrong when I started this series. I treated the merge as the finish line. It isn't — a merge is a statement about maintainer bandwidth, and only the fixture is a statement about the code. It runs in a minute on anyone's machine, red on upstream main, green on the branch, and it means the same thing whether or not anyone at the repo ever reads it. If the Total Tokens figure on that dashboard matters to you, you don't need to trust me and you don't need to wait for the queue: ./run_check.sh settles it.

Counterweights I owe you

• The documentation already said it. Worth repeating, because it is the strongest argument against the framing I originally wanted for this post.
• Other trackers get this right. ccusage, opcode and ccseva key on messageId:requestId; codeburn keys on a global seenMsgIds set. On my corpus the two strategies are equivalent — distinct (message.id, requestId) pairs came out equal to distinct message.ids, so message.id alone was sufficient here. There is no "trackers are broken" story to sell.
• Both signs occur. splitrail undercounted. This one over-counted. No directional rule.
• A hypothesis of mine was false. After part two I expected the subagent-directory blind spot to be an industry-wide pattern. I read thirteen trackers from source. It wasn't — only one unmaintained project had it, and a third-party had already reported the same class of gap in tokscale eighteen days before I filed mine. Saying so is more useful than the story I was hoping for.
• In tokscale, my proposed fix lost the argument, and should have. I suggested the append-only store this series runs on. The maintainer turned it down: retention that never deletes would resurrect sessions a user deliberately removed, so he scoped retention to dedup keys that stay stable across files instead. I tested his rule — deleting a transcript dropped exactly that file's contents, 40 messages, 78,770 output tokens, nothing resurrected — and conceded in the thread. The audit finds the discrepancy. It has no special authority over the fix.
• cubic found something my method wouldn't have. A latent hazard in code that had not yet met hostile data leaves no trace in any measurement, because it hasn't happened yet. The two approaches cover different failure classes, and I'd run both.

What I'd generalize

Reading is not measuring. Documentation, stars, code review, and more code review are all mechanisms for reading. Stacking more of them does not asymptotically produce a measurement. If nothing in your pipeline compares an output to an independently derived number, that comparison is not happening, however many eyes are on the diff.

Assert an invariant, not a total. Totals need fixtures, corpora and maintenance. messagesWithUsage == distinct message.ids needs none of those and would have failed on the first commit. Nearly every accounting path has an invariant like this hiding in it; the work is noticing which one.

Exactness is the evidence. "Roughly double" is a shrug that invites a debate about whose numbers are right. "Exactly the per-record sum, to the digit, on every field" is a diagnosis that ends the debate before it starts.

Ship the mechanism with the report. Four upstream fixes across two projects so far, three in splitrail and one in tokscale, the fastest merged two hours after the report — all because a runnable red-green fixture arrived attached to the claim rather than after someone asked for one.

When the same hazard hits several tools, converge on a contract, not more reports. The rewrite-drift class now has a one-page spec with two independent implementations behind two different leaderboards. That does more than a third bug report would have.

When the vendor documents a hazard, treat it as a test case, not as trivia. Every warning box in an SDK's docs is a bug someone has already shipped. That is a free list of things to go measure.

Timeline

The layer underneath

All three parts of this series run on TraceGuard's routing_audit module — an append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store. It is a few hundred lines and a database file, and everything above is downstream of one design decision: never let the source rewrite history.

Since part two, that layer grew a public edge: after every scheduled ingest it appends the six-field usage-drift-log record (frozen at first sight, never recomputed), so the drift class yoo-minho measured shows up as a warning line instead of a silent change.

Stable totals are the substrate, not the product. The thing worth building on top is stated-vs-revealed routing analysis, priced per decision — which model you said you'd route to, which one actually ran, and what the difference cost. That's part four, and this time I mean it.

pip install traceguard — Apache-2.0. The reproduction harness for this post is self-contained, runs in about a minute, and touches no real ~/.claude data — it builds a synthetic $HOME and hands the upstream analyzer nothing but a directory: usage-tracker-audit/cct-dedup-check.