rdap.org answers 302, and a client that doesn't follow it sees nothing

python dev.to

Quick answer

WHOIS is a 1980s text protocol with no schema — every registrar formats it differently, so "parsing WHOIS" means maintaining a pile of regexes per registrar. RDAP is its IETF replacement: same data, structured JSON, no key, no account. The catch that bites first: the bootstrap service at rdap.org answers HTTP 302, not 200. It redirects you to the authoritative registry (rdap.verisign.com for .com). A client with redirects disabled sees a 302 with a zero-byte body and concludes the endpoint is dead. Ours follows it — that is the whole trick, and it is why the Domain WHOIS RDAP Scraper needs no paid WHOIS reseller behind it.

Why not just use a WHOIS API? 💸

Because you'd be paying a reseller for a worse version of free public data.

Most domain-lookup tools on the market wrap a commercial WHOIS API. That means credit metering, rate caps set by someone else's business model, and output that is often just the raw WHOIS blob — a wall of text you still have to parse. The reseller's margin buys you nothing structural.

RDAP (RFC 7480–7484) is the registries' own successor protocol. It is:

  • Keyless. No signup, no token.
  • Structured. JSON with defined object classes, not free text.
  • Authoritative. You end up talking to the registry that actually holds the record.

The one thing it is not is convenient, and that is where the engineering goes.

The three things that actually break 🔧

1. The 302 bootstrap. https://rdap.org/domain/stripe.com returns 302. Follow it and you land on rdap.verisign.com with the real record. Don't follow it and you get nothing — and because the body is empty rather than an error, it looks like a broken endpoint rather than a redirect. We record which server answered in rdap_server on every row, so you can always see who told us.

2. 404 is an answer, not an error. For an unregistered domain, RDAP returns 404. That is the most commercially useful response the protocol has — it means available. An Actor that treats non-200 as failure throws away exactly the result a domain investor is paying for. We emit a row with status: "available" instead. The same logic means a run that finds nothing still finishes SUCCEEDED: a completed, genuinely-empty answer is correct, not a crash.

3. Entity layouts differ by registry. The registrar's name is not a field — it lives inside an entities array, in whichever entry carries the role registrar, encoded as jCard. Dates are not fields either: they sit in an events array keyed by eventAction (registration, expiration, last changed). ccTLD registries nest entities differently and add event actions the gTLDs don't use. Our fixtures include a ccTLD with a genuinely different nesting shape, captured as real bytes, precisely so the parser is tested against reality rather than against itself.

About redacted registrant data 🕵️

Since GDPR, most registries redact registrant name, email and phone. You will see registrant_org and registrant_country populated sometimes and null often.

We return null. We do not guess, infer, or backfill from a third-party enrichment source and present it as registry data. A null that means "the registry redacted this" is more useful than a plausible-looking value you can't trust — and if a tool shows you a registrant name for every domain, ask where it came from.

What a row looks like

Real output, verified against the live registry outside the Actor:

Field Value
domain stripe.com
status registered
registrar SafeNames Ltd.
created_date 1995-09-12T04:00:00Z
expiration_date 2027-09-11T04:00:00Z
last_changed_date 2025-10-01T01:39:51Z
nameservers 4 × AWSDNS-*
epp_status_codes client/server delete, transfer, update prohibited
dnssec_enabled false
rdap_server rdap.verisign.com

Those epp_status_codes are the field people overlook. Six prohibition codes on a domain is a registrar lock — a signal about how tightly held a name is, which matters if you are trying to buy it or watching for a hostile transfer.

FAQ

What does it cost?
$0.05 per run start plus $0.005 per domain — $5.05 per 1,000 domains. Pay per result, no subscription.

Do I need an API key or an account?
No. RDAP is public. That is the point.

Can I check thousands of domains in one run?
Yes — bulk input is the primary use case. Registry RDAP endpoints throttle, so we pace requests and back off on 429/503 rather than hammering them, and a single failing domain is skipped rather than killing the batch.

Does it tell me if a domain is available?
Yes, that's the status: "available" row, derived from the registry's own 404. Note that availability at the registry is not the same as purchasable — a name can be reserved, premium-priced, or in a redemption period.

Why is registrant_name never populated?
GDPR redaction at the registry. We report the absence honestly rather than filling it in.


Built by Devil Scrapes. We do the dirty work so your dataset stays clean. 😈

Source: dev.to

arrow_back Back to Tutorials