devLap

The Idempotency-Key tutorial has a hole, and M-Pesa found it for me

typescript dev.to

Every idempotency tutorial I have read ends the same way.

Take the Idempotency-Key header. Look it up in Redis. If it is there, replay the cached response. If not, run the handler and cache the result. Done.

I shipped exactly that in the first version of FyberPay, a billing platform for internet service providers in Kenya. It worked for a year. Then it nearly handed one ISP's payment response to another ISP.

This post is about that hole, the fix, and the second hole the fix does not cover.

The setup: eight payment rails, many tenants

FyberPay sits between M-Pesa (plus Paystack, KopoKopo, Tuma, Bill Manager and bank paybills) and the routers of dozens of small ISPs. Every ISP is a tenant. Every tenant's subscribers pay through the same webhook endpoints.

Daraja, Safaricom's M-Pesa API, retries callbacks aggressively. If your endpoint does not answer 200 within a few seconds, the same payment arrives again. Twice is normal. Three times happens weekly.

So idempotency is not optional. The question is what the cache key is made of.

What the textbook key actually is

In the textbook version the caller chooses the entire key:

Idempotency-Key: 7f3c1a2e-...-91ab
redis.get('idem:7f3c1a2e-...-91ab')
Enter fullscreen mode Exit fullscreen mode

Now imagine two tenants. Tenant A's integration and Tenant B's integration were both written from the same sample code. Both generate keys from the same seed, or both reuse a key after a crash, or one key simply leaks through a log line.

Tenant B sends key 7f3c.... Redis already has it. Your interceptor does what it was told: it replays Tenant A's cached response body to Tenant B.

The system is idempotent. It is also a data leak.

The fix: namespace before you trust

The interceptor now builds the key from the server's view of the request first. The client's header goes last, as a suffix, never as the address:

// idempotency.interceptor.ts (NestJS)
const userId = (request.user as { id?: string } | undefined)?.id;
const orgId = (request.org as { id?: string } | null | undefined)?.id ?? 'root';
const principal = userId ?? `anon:${request.ip ?? 'noip'}`;
const path = String(request.originalUrl ?? request.url ?? '').split('?')[0];

const bodyHash = createHash('sha256')
  .update(JSON.stringify(request.body ?? {}))
  .digest('hex')
  .slice(0, 32);

const redisKey =
  `${KEY_PREFIX}${principal}:${orgId}:${method}:${path}:${bodyHash}:${idempotencyKey}`;

// Atomic claim. One winner runs the handler; everyone else replays.
const claimed = await this.redis.set(redisKey, 'processing', 'EX', IDEMPOTENCY_TTL, 'NX');
Enter fullscreen mode Exit fullscreen mode

Read the key left to right:

idem : u_8812 : org_kisumu-fiber : POST : /payments/stk : 9e1c...f2a0 : 7f3c-...-91ab
       who      which tenant       verb   route           body hash       client's key
Enter fullscreen mode Exit fullscreen mode

Two things fall out of this.

Leakage becomes structurally impossible. Tenant B's principal and org are baked into the address before the lookup. There is no key Tenant B can send that resolves to Tenant A's slot.

Body changes never replay. A client that changes one byte gets a fresh execution. That is stricter than the spec, and in a payments path it is what you want. "Same key, different amount" should never silently return the old receipt.

The NX matters as much as the namespace. SET ... NX is the claim. Two concurrent retries race for one slot, exactly one wins, and the loser waits for the stored response instead of running the handler a second time.

The second hole: the request that never sees your interceptor

Here is a real Tuesday from the logs, one M-Pesa receipt, four arrivals:

10:41:02.114  webhook #1  RJK4T7  redis MISS  ->  claim  ->  INSERT ok  ->  200
10:41:02.980  webhook #2  RJK4T7  redis HIT   ->  replay 200
10:41:31.400  webhook #3  RJK4T7  redis HIT   ->  replay 200
11:15:07.000  hourly Daraja pull   RJK4T7      ->INSERT fails UNIQUE  ->  rollback
Enter fullscreen mode Exit fullscreen mode

Arrivals two and three are the interceptor doing its job. Arrival four is the one the tutorial never mentions.

FyberPay also pulls transactions from Daraja on a schedule, as a safety net for webhooks that never arrive at all. That pull runs in a worker. It is not an HTTP request. It never touches the interceptor. Redis cannot help it.

What stops the double credit is one line in a hand-written migration:

ALTER TABLE payments
  ADD CONSTRAINT payments_receipt_unique UNIQUE (gateway, receipt_number);
Enter fullscreen mode Exit fullscreen mode

The M-Pesa receipt number is unique on Safaricom's side, so it is unique on ours. The insert fails, the transaction rolls back, and the ledger never moves twice. The interceptor is the fast path. The constraint is the truth.

What to take from this

If you are building anything where the same payload can arrive from more than one direction (webhooks plus polling, webhooks plus manual replay, two regions), two rules:

  1. The client's idempotency key is a suffix, not an address. Prefix it with everything the server knows: principal, tenant, method, route, body hash. Claim with SET NX.
  2. Put a UNIQUE constraint on the external reference. It costs one migration and it is the only layer that works for every code path, including the ones you have not written yet.

There is a third layer (a transactional outbox with SKIP LOCKED, for the side effects that run after the ledger commits) and a circuit-breaker story about why a wrong PIN must not count as an outage. Both are in the full teardown here: Building 3-layer idempotency and webhook resilience across 8 payment gateways.

I build fixed-scope payment integrations and audit existing ones. kiragu@alkenacode.dev or Contra. Case studies and more of this work at kiragu.alkenacode.dev.

Source: dev.to

arrow_back Back to Tutorials