devLap

Sending domain health on a schedule: catching record and mail status drift

go dev.to

Nobody edits a DKIM record on purpose at 02:00 on a Sunday, and no registrar sends you a webhook when someone does. That constraint — there is no event to subscribe to — decides the shape of the whole design, so the answer is a scheduled job that reads two things per sending domain: the records actually published in DNS, and the sending status the mail provider reports for that domain. Emit both as metrics. Alert when they disagree.

Disagreement is the signal. Absence, on its own, usually isn't.

The system behind this piece is a gaming platform that sends mail for studios — password resets, receipt mail, tournament reminders — from domains the studios own, plus a set of subdomains we run on their behalf. We were also midway through moving zone writes off one registrar's proprietary API, and that migration is what forced the question into the open: if the thing holding the zone can change, what is a health check allowed to depend on?

What actually breaks: agreement, not presence

A provider API answers a different question than the one you care about. It reports what you asked it to publish, which is a write log with good manners. DNS reports what a receiving mail server will resolve when it decides whether to accept your message, and those two answers diverge for reasons that are boring, common, and invisible until a campaign goes out: a studio's ops lead edits the SPF TXT record by hand to add a third-party tool and blows past the ten-lookup limit in RFC 7208; the zone gets re-delegated to a different set of nameservers during an unrelated website migration and the new zone file was seeded from a months-old export; someone replaces the CNAME that pointed a DKIM selector at the provider with a copied TXT value, which works fine right up until the provider rotates that key and the copy stays frozen at the old public key forever.

Meanwhile the provider dashboard still says the domain is verified, because it verified once, eight months ago, and revalidates on a cadence of its own. How quickly a given mail provider re-checks a domain it already approved is rarely documented, and I'm not sure any of them commit to a bound — which is precisely why the check has to own both halves rather than trusting either one.

The invariant worth writing on the runbook: the provider API is intent, DNS is reality, and health is the agreement between them.

How should you schedule a sending domain health check across records and mail status?

Daily is enough for configuration that should never change on its own, and I run platform-owned zones hourly only while a migration window is open. The scan itself is four lookups per domain — SPF at the apex, the DKIM selector under _domainkey, _dmarc, and MX — and it should hit the authoritative nameservers for the zone rather than whatever recursive resolver the container inherited. Negative caching is the reason. Under RFC 2308 a missing answer is cacheable for the SOA minimum, so a record a customer fixed twenty minutes ago can still read as absent, and a record they deleted can read as present until the TTL expires. Querying each authoritative server also catches the split-brain case where one nameserver in the set was never updated, which a recursive lookup will hide from you roughly two times out of three.

Then classify the error, because this is where these checks earn their reputation for crying wolf.

An NXDOMAIN or an empty answer means the record is genuinely gone. A SERVFAIL, a timeout, or a refused query means your observation path is broken and you know nothing about the domain — those two outcomes must never collapse into the same metric value. Emit the unknown case as a third state and leave the last known value alone; a resolver hiccup in your monitoring pod is not a customer outage, and treating it as one is how people learn to ignore the alert.

Capacity is not the constraint. Four thousand domains at four lookups against three authoritative servers is about 48,000 queries a day, under one query per second, which is nothing. Cardinality is the constraint: three gauges per domain is 12,000 series at that fleet size, fine for any Prometheus-shaped store, and the same design at 100,000 customer domains is 300,000 series and a bill you will be asked to explain. Past roughly 20,000 domains I'd keep per-tenant aggregates in metrics and push the per-domain detail into a table the on-call can query, which costs you one join during an incident and saves the cost of storing a mostly-constant time series for every domain you have ever touched.

The objective I hold this to isn't availability, it's detection latency: drift in a zone we control is caught within the hour, drift in a customer's zone within a day, and no page fires until the same disagreement survives two consecutive runs.

Customer-owned versus platform-owned zones

This axis decides more of the design than the tooling does, because it decides who can act on the alert.

Zone ownership Who can fix drift Cadence Response
Platform subdomain, our API token on-call, automatically hourly page on disagreement
Customer domain, NS delegated to us on-call, with a change record daily ticket, page after two days
Customer domain, edited at their registrar customer only daily notify customer, never page us

Page only on drift you have the credentials to fix. Everything else is a ticket and a notification, and the failure mode of ignoring that rule is an on-call rotation that gets woken up for a TXT record it has no authority to touch, twice a week, until the alert is muted and the real ones go with it.

Ownership also settles the build question. A commercial deliverability monitor covers customer-owned domains well enough and costs less than the engineer-weeks this takes to build and keep alive; the catch is that most of them monitor from their own vantage point on their own schedule and won't correlate against the sending status in your provider account, which is exactly the correlation that catches the stale DKIM copy. Zone content generated and diffed by tooling such as OctoDNS or DNSControl gives you a third copy of intent to compare against, though neither of them tells you what a receiving MTA resolved. My split: buy the reputation and blocklist monitoring, build the agreement check, and never build a resolver.

The probe, and the two gauges it emits

Most teams start with a Node.js script driven by cron, and the logic transfers line for line. I write this kind of tooling in Go because it runs as a sidecar next to the fleet's other probes and I'd rather not ship a second runtime to the edge.

The error classification comes first, since everything else depends on it:

// classify separates "the record is genuinely absent" from "we could not ask".
// net.DNSError has carried IsNotFound since Go 1.13; a timeout is an outage in
// our observation path and must never be reported as missing configuration.
func classify(err error) (absent bool, unknown error) {
    if err == nil {
        return false, nil
    }
    var de *net.DNSError
    if errors.As(err, &de) && de.IsNotFound {
        return true, nil
    }
    return false, err
}

func txtPrefix(ctx context.Context, r *net.Resolver, name, prefix string) (bool, error) {
    recs, err := r.LookupTXT(ctx, name)
    if absent, unknown := classify(err); unknown != nil {
        return false, unknown
    } else if absent {
        return false, nil
    }
    for _, rec := range recs {
        if strings.HasPrefix(strings.ToLower(rec), prefix) {
            return true, nil
        }
    }
    return false, nil
}
Enter fullscreen mode Exit fullscreen mode

The scan is deliberately dull. A DKIM selector published as a CNAME to the provider resolves through to the TXT record on lookup, so the same call covers both the delegated and the copied form — and the copied form is the one that goes stale, which you see as DNS saying yes while the provider says no.

type Snapshot struct {
    SPF, DKIM, DMARC bool
    MX               int
}

func Scan(ctx context.Context, r *net.Resolver, d Domain) (Snapshot, error) {
    var s Snapshot
    var err error
    if s.SPF, err = txtPrefix(ctx, r, d.Name, "v=spf1"); err != nil {
        return s, err
    }
    if s.DKIM, err = txtPrefix(ctx, r, d.Selector+"._domainkey."+d.Name, "v=dkim1"); err != nil {
        return s, err
    }
    if s.DMARC, err = txtPrefix(ctx, r, "_dmarc."+d.Name, "v=dmarc1"); err != nil {
        return s, err
    }
    mx, err := r.LookupMX(ctx, d.Name)
    if absent, unknown := classify(err); unknown != nil {
        return s, unknown
    } else if !absent {
        s.MX = len(mx)
    }
    return s, nil
}
Enter fullscreen mode Exit fullscreen mode

Two gauges carry the raw facts and a third carries the judgement, and only the third has an alert attached to it. Keeping the raw pair is what turns this from a pager into a record: six months later you can see that the SPF record went false on a Tuesday afternoon, which is a far more useful sentence in a postmortem than "someone broke DNS at some point".

type Emitter interface {
    Gauge(name string, v float64, labels map[string]string)
}

func report(e Emitter, d Domain, dns Snapshot, mailReady bool) {
    l := map[string]string{"tenant": d.Tenant, "owner": string(d.Ownership)}
    dnsReady := dns.SPF && dns.DKIM && dns.DMARC && dns.MX > 0
    e.Gauge("sending_domain_dns_ready", b2f(dnsReady), l)
    e.Gauge("sending_domain_mail_status_ready", b2f(mailReady), l)
    e.Gauge("sending_domain_agreement", b2f(dnsReady == mailReady), l)
}
Enter fullscreen mode Exit fullscreen mode

Note what the labels don't include. Dropping the domain name from the metric labels and keeping it in the detail table is the cardinality trade I described above; if your fleet is small, put it back and enjoy the simpler queries.

The mail status half is one call to whichever provider you send through, and the only property that matters is that you read the domain's current sending state rather than caching your own copy of it. Providers model this differently — some report a verified boolean per authentication method, some a single domain state — so the adapter should collapse it to one boolean and refuse to guess when the call itself errors.

Where this check earns nothing

It measures configuration agreement, not deliverability. A domain can pass every assertion here and still land in the spam folder because the complaint rate is high or the IP pool is new, and none of SPF, DKIM or DMARC alignment under RFC 7489 tells you anything about reputation.

It's also the wrong shape for platforms where customers add domains continuously. If you are onboarding hundreds of new sending domains a day, verification belongs in the onboarding path as an event-driven check with a short retry ladder, and a daily sweep is a slow second opinion at best — stick with the provider's verification callback for that flow and keep the sweep for the long tail of domains that were verified months ago and forgotten.

And it deliberately ignores the transport side. MTA-STS policies, TLS reporting, DNSSEC signing state — all worth monitoring, none of them in scope for the question of whether this domain can send mail today. Your mileage may vary on how much of that you fold into the same job; I keep them separate because they have different owners and different escalation paths, which is the same reasoning that split the table above.

Drift is quiet. Give it something loud to trip over, on a schedule, and let the two halves argue with each other so your on-call doesn't have to.

References

Source: dev.to

arrow_back Back to Tutorials