The offset parameter that silently does nothing (and would have billed for it)

Quick answer

The Wayback Machine's CDX API accepts an offset parameter. It is documented. It returns HTTP 200. And on web.archive.org/cdx/search/cdx it does nothing at all — offset=0, offset=1000 and offset=100000 return byte-identical results. A paginator built on it loops over page 1 forever. Under Pay-Per-Event that isn't just a bug, it's an overbilling bug: every repeated page writes duplicate rows the customer pays for. We caught it before shipping the Wayback Machine CDX Scraper because we re-probed the live API instead of trusting a green test suite.

How a parameter that does nothing passes every test 🟢

This is the uncomfortable part. The first implementation had:

• 73 passing tests
• clean ruff
• both pre-publish verifiers OK
• fixtures captured from the real API

All green. All meaningless for this defect, because the tests asserted that the client sent offset and parsed what came back — and it did, and it parsed correctly. Nothing in that loop can notice that the server ignored the parameter, since the response is a perfectly valid page of results. It's just the same page.

The only way to catch it is to ask the live API a question whose answer you already know:

offset=0      -> first record: com,stripe)/ 19961218234205
offset=1000   -> first record: com,stripe)/ 19961218234205
offset=100000 -> first record: com,stripe)/ 19961218234205

Confirmed across matchType=exact, host and domain. A test suite written against your own client can never tell you this. A fixture proves your parser works; it cannot prove the server agreed with you.

What actually paginates: resumeKey 🔑

The working mechanism is a resume token:

1. Send showResumeKey=true.
2. The response ends with a blank line and then the token.
3. Pass it back as resumeKey=<token> for the next page.

There's a wrinkle worth recording: the resume-key mechanism is widely assumed to be a text-mode-only feature. It is not — it works in output=json mode too, which is what let us keep structured parsing and correct pagination at the same time.

Verified with a real multi-page live run: 12 records across 3 pages, 12 unique timestamps, zero duplicates.

Why duplicate rows are worse than a crash 💸

A crash is honest. You see it, you fix it.

Silent duplication is the failure mode we're most careful about here, because it passes every health check we own. The run finishes SUCCEEDED. The dataset is full. The dashboard is green. And the customer has paid for the same archived URL fifteen times.

We've been bitten by the neighbouring version of this — a SUCCEEDED run with zero rows scores 100% on every success-rate metric while delivering nothing — so "the run succeeded" has stopped counting as evidence around here. What counts is a property only correct output can satisfy. For this Actor, that property is uniqueness of (timestamp, original_url) across the whole result set, and it's checked, not assumed.

What you can actually do with it 🕵️

The CDX index is not "fetch me an old page". It's a queryable index of every capture the archive holds, and the server-side filters are the useful part:

• matchType=domain — every URL ever archived under a domain, subdomains included.
• filter=statuscode:200 — only captures that were actually live, so you're not recovering a list of 404s.
• filter=mimetype:text/html — pages, not assets.
• collapse=urlkey — one row per unique URL instead of one per capture.
• from / to — restrict to a date window.

Put together, that's the query an SEO team needs after a botched migration: every HTML URL on this domain that returned 200 before the relaunch, deduplicated. That is a recoverable sitemap of lost inventory, and it's one run.

For OSINT and domain diligence the same index answers a different question: what did this domain used to be, and when did it change hands?

What you get per row

That row is real output. digest is the archive's own content hash — two captures sharing a digest are byte-identical, which is how you find when a page actually changed rather than when it was merely re-crawled.

FAQ

What does it cost?
$0.05 per run start plus $0.002 per snapshot row — $2.05 per 1,000 rows.

Do I need an API key or account?
No. The CDX API is public and keyless.

Will it return every URL for a big domain?
It will page through as far as you let it. Set the result cap deliberately: matchType=domain on a large site can return a very large number of captures, and you pay per row. Use collapse=urlkey when you want inventory rather than history.

What if a domain has nothing archived?
That's a valid empty answer, not an error. The run finishes SUCCEEDED and says so — the archive genuinely holding nothing is information.

Does it fetch the archived page content?
No. It returns the index plus a direct archive_url for each capture. Indexing and fetching are different jobs with very different costs, and bundling them would make the cheap one expensive.

Is the Internet Archive OK with this?
The CDX API is a public, documented interface. We pace requests and back off rather than hammering it — it's a nonprofit archive, and being a bad neighbour there would be both rude and self-defeating.

Built by Devil Scrapes. We do the dirty work so your dataset stays clean. 😈