A penny I sent myself switched off the balance check. The coins went out to the network; nothing was debited.
It wasn't a typo and it wasn't a missing check. It was a sentence I believed: an idempotency key makes a retry safe. It reads like an axiom. It's false.
Ten sentences below, that one first. All of them about moving money: holds, deposits, fees, reserves, asset precision. I believed every one, and every one cost real money. Under each there is a commit and a test you can open. Some are covered in other people's CVEs and audits, and I'll point at them. For some I found no public write-up at all, and I did look.
01. An idempotency key makes a retry safe
The retry, yes. Which checks that retry is allowed to skip, the key says nothing about. Whoever wrote the branch decides.
Ours skipped the balance check. The reasoning was simple and looked sound: if this is a replay, the request already passed every check once, so why run them again. That is why the branch sat first, ahead of the type, account, amount and funds comparisons.
And we built the hold's key by concatenation: external_ref + ":hold". So the key wasn't invented by the server. It was invented by whoever turned up.
The rest is arithmetic. Send yourself a penny with external_ref set to X:hold. Then create a withdrawal with external_ref = X. The server builds the hold's key as X + :hold: exactly the one your penny is already filed under. The hold finds it, says "ah, a replay, all done", and reserves nothing. The coins go to the network. The account is not debited.
It worked end to end on TRC20 and TON. Each network has its own send path; we call them rails. On the Bitcoin and Ethereum rails the balance was checked a second time, down there, so it wouldn't have gone through. We learned that afterwards, taking it apart, not because anyone planned it.
Idempotency keys have been written about for years, and the happy path is well covered. The unhappy one has been covered recently too: "Idempotency is easy until the second request is different" has the canonical-command comparison, the ordering relative to authorisation, and mutable state such as balance. What I found nowhere, that post included, is a derived key the server synthesises from a field the client sent. Everyone writes about a key's scope. Nobody writes about what it was glued together from.
What catches it. A replay has to prove it is the same operation, not merely the same string: compare type, account and amount before you hand back a stored response. And build the key for a derived step on the server, out of the request itself, instead of pasting a suffix onto something the client supplied. Ours is TestHold_PlantedTransactionUnderSameRef_IsRefused.
02. A deposit monitor that's running loses nothing
The other way round. A broken one loses nothing; a healthy one loses everything.
The monitor walked the chain and moved its checkpoint after each block it scanned. A transaction that hadn't reached the confirmation depth it skipped: a line in the log saying "waiting for confirmations", and on to the next block. Reasonable enough: we'll pick it up on the following pass.
There was no following pass. By then the block was marked processed, and nothing will ever read it again.
Now the inversion. It scanned right up to the tip, and the interval on Ethereum is fifteen seconds. So it saw a fresh transaction a couple of blocks after it appeared, always before the confirmations had accrued. Every deposit that arrived while the monitor was running was lost. And if the monitor had been down for an hour and was catching up, it saw blocks that already had their confirmations and credited all of them correctly.
Crypto has no name for this. Streaming does: it's committing the offset before processing, and the Kafka documentation warns about it in as many words. The only difference is that streaming has no notion of confirmation depth, which is why the "healthy loses, lagging doesn't" inversion can't happen there.
What catches it. Scan to tip вИТ MinConfirmations, not to the tip. After that the "waiting for confirmations" branch is unreachable in normal operation. The confirmation count itself is an assumption rather than an invariant: on proof-of-stake the right boundary is consensus finality, not a block counter. Our test asserts the cursor's position, that it never went past tip вИТ MinConfirmations, and that is weaker than I'd like: the branch is still there in the code.
03. Checking an amount is cheap
Parsing {"amount":"1e1000000000"} genuinely is. Mantissa one, exponent a billion, twelve bytes on the wire. The number is not expanded at parse time. It is expanded at the first arithmetic operation.
The first arithmetic operation is our check. "Amount is positive." "Amount is within the limit." A comparison against zero looks like the cheapest operation in the world, and it aligns both operands to a common exponent.
I measured it rather than guess at the order of magnitude. Go 1.26, arm64, shopspring/decimal; the clock covers exactly one GreaterThan(Zero) after parsing:
1000.50 0.0 ms +0 MB
1e100000 0.6 ms +0 MB
1e1000000 20 ms +3 MB
1e5000000 251 ms +21 MB
Parsing is free in all four. The comparison pays.
A separate piece of nastiness: the bound was present in six methods, but the deposit path with a non-zero fee didn't go through a guarded one. So the check switched itself on and off according to a pricing setting.
In Java the class is known and closed: the Johnzon CVE is literally about 1e20000000, and the fix was a scale limit on BigDecimal. Meanwhile the industry default, Jackson's maxNumberLength=1000, bounds the literal's length: it catches a long mantissa and lets a short exponent through. Two projects, two different bounds, and neither is about the exponent. shopspring/decimal has zero advisories in OSV at all.
What catches it. Check the exponent and the number of significant digits. Exponent() and NumDigits() don't expand the mantissa, so the rejection costs nothing. A length bound won't catch this shape: our twelve bytes clear any such limit. You still want the length bound, because a long mantissa is exactly what it does catch, and it belongs first, before the number is built. And remember that the rejection has to be cheap: a validator that refuses a number at the cost of a gigabyte has closed nothing. In Go that matters more than elsewhere: running out of memory is fatal, it kills the process rather than the request, and no middleware will intercept it.
04. An incoming transfer to a user's address is a deposit
Usually. Except when the sender is you.
A sweep collects money from deposit addresses into the hot wallet, and on EVM it is impossible without gas sitting on the address itself. So the platform sends ETH to the customer's deposit address in order to take tokens off it afterwards. By address alone that transfer is indistinguishable from a real deposit: in both cases money moves from the hot wallet to a user's address.
The monitor saw it as an ordinary incoming transfer and credited it. The top-up default is 0.003 ETH, and that much accrued out of nowhere every cycle. Liabilities grew while the on-chain funds belonged to the platform throughout, and the reserve check drifted along with them.
Filtering by address cannot work here, in principle. The only source of truth is your own registry of your own transaction hashes, which the sweeper writes and the monitor reads.
Vitalik has written about exchanges shuffling collateral between each other to perform solvency. That is asset inflation, and it is deliberate. This is liability inflation, and it is accidental: caused by your own housekeeping traffic.
What catches it. A registry of internal transfers, and a monitor that checks against it. And while you're there, look at your units: the same commit fixed a muddle where the balance arrived in wei, the gas cost was subtracted in ETH, and the result was multiplied by 10¬євБЄ a second time. The sweeper tried to send a quintillion times the balance.
05. A send error means nothing went out
For us, any send error released the hold except a hot-wallet shortage. Unconditionally, on all four rails: it didn't send, so put the money back.
Alongside it, on the Bitcoin rail, lived a list called nonRetryablePatterns, commented "fundamental issues with the transaction", and among its entries sat txn-mempool-conflict. That list decided whether to stop retrying.
Both constructions rest on the same premise: the send returned an error, so nothing went out.
That doesn't follow from an error. Classify by stage, not by text. Building, encoding and signing all happen before the network is touched, so their failure proves the transaction does not exist. Anything that fails after the network call proves only that you don't know. There are three outcomes, not two: it went, it definitely didn't go, and unknown.
Card processors solved this with a header: at Stripe, the absence of Stripe-Should-Retry in a response means "we cannot determine whether this is safe to retry." Blockchains have no such header, so everyone matches strings, even though the nodes themselves classify by class rather than by wording. In go-ethereum, ErrAlreadyKnown is documented as "the transactions is already contained within the pool"; Bitcoin Core has TX_CONFLICT, commented "Tx already in mempool or conflicts with a tx in the chain".
And there you can see why strings are a poor source of truth. In Bitcoin Core, txn-mempool-conflict means a conflict with a transaction in the mempool: a double spend of the same input. It does not prove that yours will go through. We read it as "all good, it's already in there", which was a guess from a string like every other guess from a string.
What catches it. Release a hold only on proven absence. In every other case keep it and call a human. A stuck withdrawal is resolved by hand; a second payout in non-reversible coins is resolved by nothing.
06. A column of the right shape is a key
A batch of payouts arrives as a file, and that file usually has its own column of idempotency keys. There may be no header row at all, and if there is one it's in the customer's language, so we guess the column by its contents.
And a column is a key only if its values are distinct. Without that check the period column would have found its way into the idempotency keys, "2026-08" in every row, and the second line would have been refused as a duplicate: someone doesn't get paid because of our guess. That didn't happen: role detection and the uniqueness condition landed in the same commit. The other direction did happen.
Someone pasted an export with their own keys, the detector didn't recognise them, the keys quietly vanished, and the batch went out under keys we invented. A second upload of the same file produced different invented ones: that is, a second payment. Precisely where the key was supposed to stop it.
Column auto-detection looks like a convenience. What it actually does is decide whether the product's principal safeguard works at all, and it can fail in both directions.
What catches it. A uniqueness check on the values: without it, a "key" is just the first column of the right shape. Ours looks at the first twenty non-empty rows, and that's a compromise rather than a solution: a file whose repeats begin on the twenty-first row will pass. Plus a rule about guessing: if you recognised nothing, return the original order rather than guessing half of it. Half-guessed is worse than not guessed, because the operator reads it as "the system worked it out" and stops checking.
07. Address case is a property of the network
It's a property of the format.
bech32 may be written entirely in capitals: BIP-173 carries a test vector for it, and a decoder is required to accept it on equal terms with the lowercase form. The only thing forbidden there is mixed case. We compared against the sanctions list case-sensitively, with the rules split by network. The same address, submitted in capitals, didn't match. The check answered "clear".
Base58, meanwhile, is genuinely case-sensitive; there the case carries information. So the rule cannot be one per network: a single chain hosts formats with opposite requirements, and normalisation has to follow the format, not the rail.
NVD has zero entries for bech32. Zero for EIP-55. The class has no name, even though both specifications describe in plain words the ambiguity it grows out of: ERC-55 goes as far as listing compatibility with mixed-case-accepting parsers as an advantage.
What catches it. Normalisation by format, sitting next to normalisation by network, in the same comparator. One does not replace the other: the per-network rule still does real work where case genuinely is a property of the rail. And a test that an address in capitals is still matched by the same sanctions list: TestScreen_Bech32UppercaseStillMatches.
08. A signing policy limits what leaves
It limits the transfer amount. What leaves the account, it doesn't limit at all.
The fee is the hot wallet's second till, and on EVM the caller names it: gas √Ч gas_price, both fields his. A one-penny transfer with a fee equal to the entire remaining balance satisfied the rules: the amount ceiling was respected, and the wallet was empty.
On Bitcoin it's worse, and that part isn't our omission. There the fee equals inputs minus outputs, and for legacy inputs the transaction being signed does not carry the input amounts at all. An isolated signer, which by construction never reaches the chain, physically cannot compute it. Taking it as declared by the caller is pointless: the caller is precisely the party being distrusted.
This is the motivation for BIP-143, stated in the document in plain words: for an offline device, not knowing the input amount makes the fee impossible to compute. Except it says this about a cold wallet, an offline signing device. I found no formulation anywhere applying it to a custodial signing policy, even though the position is identical: a signer who must not trust its caller.
What catches it. Before segwit, a bound on the untrusted side, expressed as the ratio of fee to withdrawal amount. After it, the fee becomes verifiable at the signer itself, and that is where it must be verified.
Then two conditions without which a ceiling isn't a ceiling. First: a per-transaction limit bounds nothing while requests can arrive concurrently; the exposure equals the ceiling multiplied by the number of parallel signatures. Second: the output sum is computed with an overflow check. Four outputs of 2вБґ¬≤ each come to exactly zero; a fifth of a hundred satoshi makes the total small and positive, and the policy sees a hundred satoshi where the transaction pays eighteen quintillion.
09. A solvency circuit breaker protects money
It freezes it.
The reserve check compares liabilities against what sits on-chain at our addresses. The address enumerator filtered on is_active = TRUE, which seemed reasonable: we only ever deactivate addresses that were never used.
The argument was true of the application's invariant and false of the database. The code does deactivate only unused addresses; it checks. But migrations write rows that no code path can produce, and among them are rows with money on them. On the staging environment that dropped 1.522 ETH against liabilities of 1.5246: the check saw zero across the addresses it got from the database, i.e. a shortfall roughly the size of the entire liability. Under RECONCILIATION_ENFORCE that is a halt on ETH withdrawals, with the money sitting exactly where it should be.
The failure mode here is inverted. An ordinary circuit breaker is dangerous because it might not trip. This one is dangerous because it does, and it trips precisely when everything is fine.
And the cost of a false alarm is comparable to the cost of a miss: halted withdrawals at an exchange are not an inconvenience, they're an incident. Which means the breaker's inputs sit on the critical path, not in some auxiliary query.
The entire public conversation about proof of reserves, meanwhile, is about an exchange hiding addresses or borrowing funds. That is, about intent. About an enumerator losing its own addresses with no intent whatsoever, I found nothing.
What catches it. Separate two different questions that look like one: "which addresses are ours" and "where are we currently accepting deposits". The second is filtered by activity. The first, never.
10. An error in the customer's favour is harmless
It's harmful because it switches off the channel through which such errors are found.
An asset carries two numbers side by side. precision: how many places we count to. decimals: how many places it has on the rail. Only the second was ever reconciled against the chain.
And precision is one number per asset, while decimals differ across rails for one and the same asset. USDT was configured with a precision of 2 against six decimals on ETH, TRC20 and TON, and eighteen on BSC. USDC, the same 2 against six. ETH, 8 against eighteen.
The fee is rounded down by precision. So a fee below a cent became zero. A shortfall nobody knew about.
And nobody could have. A monetary rounding defect is normally found quickly, because somebody turns up and complains. Here there was nothing to complain about: the error ran in the customer's favour, and the feedback loop an engineer implicitly relies on had been switched off by the error's direction.
The second consequence faced outward. The same number was handed to customers through /assets as the input precision. A screen that built its input mask from it would refuse 0.000001 USDT from a human: an amount the network moves without a murmur.
What catches it. Derive precision as at least the number of places across the asset's rails, rather than maintaining it as a hand-kept list. And the general rule the whole item exists for: reconciliations and alerts have to be two-sided. A one-directional error will never raise a ticket.
What they have in common
Every one of the ten is true. In the case I had in mind while writing the code.
"An idempotency key makes a retry safe." True if the server invented the key rather than whoever turned up. "A send error means nothing went out." True up to the network call. "Address case is a property of the network." True while a chain hosts one format. None of them is a lie. Each simply has a domain, and past its boundary it stops working silently.
And I tested each of them on exactly the case I'd written it for.
That isn't a figure of speech. Here's an example that isn't in the list above: a query looked up incoming payments by a metadata key that only two rails out of four ever wrote. On the other two the screen showed "received 0, last received: never" for money that had arrived and been credited. I checked on TRC20, where it added up. The check was real: by hand, with real money. And that is exactly why it showed nothing.
What I haven't proved here
This is not statistics. Ten cases from one set of repositories is one sample, with one author and one set of habits. Some of the classes did turn up elsewhere afterwards: a case-sensitive comparison against an allowlist sits open in peatio, two lines away from where the same address is normalised; an unbounded fee at a gas payer produced five consecutive CVEs in a single payments library this year; parsing an amount with an arbitrary exponent is closed by CVEs in both Elixir and the Go standard library. But eight other people's repositories is not a sample you compute frequencies from either.
I have not verified that my fixes are correct. I verified that they catch exactly the case that broke. Those are different claims, and the second is the weaker one.
And the gate caught none of the ten.
Eight were caught by two things. Four by addition on a live staging environment and a run with real money. Four by a separate pass of a security scanner. The remaining two arrived differently: one from checking the code against our own written rules, the other I don't remember. Which is to say that finding a hole in the checks took a check that wasn't among them. What to do about that, I still don't know.
Originally published at jeffreyjorgensen.dev. The reconciliation checks this came out of are MIT-licensed: ledger-reconcile.