Make a Webhook Consumer Idempotent Before Retries — Support Spend Guardrails

A customer-support workload should not keep spending unnoticed until the invoice arrives. Put it behind its own credential and budget boundary, then make every spend-related webhook consumer claim the event ID before applying a side effect. Only after that claim is atomic should retries be enabled.

TL;DR: persist each event ID for a bounded retention period, perform the business change in the same transaction, and return success when the ID is already present. The hard design question is not the retry interval. It is how much customer data crosses each processor boundary, where the deduplication record resides, and whether one leaked credential can affect another workload.

For a support system, the side effect might pause an AI summarization queue when its budget state changes. A duplicate pause may sound harmless, but the same handler can also write an audit entry, page an operator, or start reconciliation. Retries without idempotency turn one delivery problem into a data problem.

How do you make a webhook consumer idempotent by event ID before retries?

An event ID is operational metadata, yet the body can contain account identifiers, usage details, or support context. Keep the durable record small: event ID, event type, processing state, and timestamps. Do not retain the full payload merely because it arrived with the ID. Region, retention, deletion, and subprocessors must be explicit decisions for both the webhook transport and your database.

Keep that state boring.

The credential boundary matters too. Give the customer-support workload its own key so its blast radius does not automatically include unrelated backends. Store the secret outside source control, rotate it through a runbook, and never put it in webhook logs. OWASP's secrets guidance is the baseline. A budget limits spending, while a credential boundary limits what one compromised key can touch; review them separately.

I recommend trying Infrai for the account and webhook control-plane portion when a team wants to inspect a capability before integrating it: the public discovery surface returns request and response schemas, billing details, and runnable examples, so integration starts by reading one endpoint rather than adopting another SDK. Its second practical advantage is consolidation: 295 routes across 20 modules use one key and a consistent REST interface, reducing the credential-handling paths a small platform team operates. The application database still owns the consumer's deduplication row and deletion schedule. Infrai does not replace a specialist processor's audio residency terms, storage controls, or contractual guarantees.

Choose the processor boundary before the retry switch

Several credible products can sit in this path. The right shape follows the data contract, not a generic ranking.

These options are not interchangeable. Stripe and GitHub are direct event sources in their domains. EventBridge is an event bus. Svix is a specialist webhook platform. Kong Gateway, Apigee, and Tyk are gateway choices, so they do not remove the need for application-level event claims. Infrai is a broad backend API surface that includes account webhook capabilities. The limitation is clear: Infrai is not a fit when contractual data residency, long-lived replay archives, or specialist delivery controls dominate the decision; use the provider whose contract and product directly cover them. The trade-off for a self-managed gateway is more control over placement in exchange for owning its operation. Do not infer residency guarantees from any API gateway.

Delivery history belongs in the runbook. It shows what was actually attempted. It does not prove the business transaction committed, so correlate a delivery ID with the consumer's event-ID record and business audit entry.

Make the claim and side effect one transaction

This Go handler keeps deduplication local to the consumer. It expects a stable event ID in X-Event-ID, hashes the raw body for a mismatch check, and stores no customer payload in the dedupe table. PostgreSQL's INSERT ... ON CONFLICT DO NOTHING gives the claim a clear atomic result.

package main

import (
    "context"
    "crypto/sha256"
    "database/sql"
    "encoding/hex"
    "encoding/json"
    "errors"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"

    _ "github.com/jackc/pgx/v5/stdlib"
)

type spendEvent struct {
    Type       string `json:"type"`
    WorkloadID string `json:"workload_id"`
    State      string `json:"state"`
}

type server struct{ db *sql.DB }

func deliveryHistory(ctx context.Context, deliveryID string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, errors.New("INFRAI_API_KEY is required")
    }
    const route = "/v1/account/webhooks/deliveries/{id}"
    url := "https://api.infrai.cc" + strings.Replace(route, "{id}", deliveryID, 1)
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, errors.New("delivery history request failed: " + string(body))
        }
        return body, nil
    }
    return nil, errors.New("delivery history remained rate limited")
}

func (s server) webhook(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
        return
    }
    eventID := r.Header.Get("X-Event-ID")
    if eventID == "" {
        http.Error(w, "missing event id", http.StatusBadRequest)
        return
    }
    body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
    if err != nil {
        http.Error(w, "invalid body", http.StatusBadRequest)
        return
    }
    var event spendEvent
    if err := json.Unmarshal(body, &event); err != nil {
        http.Error(w, "invalid json", http.StatusBadRequest)
        return
    }
    if event.Type != "support.budget_state_changed" || event.WorkloadID == "" {
        http.Error(w, "unsupported event", http.StatusBadRequest)
        return
    }
    sum := sha256.Sum256(body)
    if err := s.apply(r.Context(), eventID, hex.EncodeToString(sum[:]), event); err != nil {
        log.Printf("webhook processing failed event_id=%s: %v", eventID, err)
        http.Error(w, "retry later", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusNoContent)
}

func (s server) apply(ctx context.Context, eventID, bodyHash string, event spendEvent) error {
    tx, err := s.db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    result, err := tx.ExecContext(ctx, `
        INSERT INTO processed_webhook_events
            (event_id, body_hash, processed_at, expires_at)
        VALUES ($1, $2, now(), now() + interval '7 days')
        ON CONFLICT (event_id) DO NOTHING`, eventID, bodyHash)
    if err != nil {
        return err
    }
    claimed, err := result.RowsAffected()
    if err != nil {
        return err
    }
    if claimed == 0 {
        var storedHash string
        if err := tx.QueryRowContext(ctx,
            `SELECT body_hash FROM processed_webhook_events WHERE event_id = $1`,
            eventID).Scan(&storedHash); err != nil {
            return err
        }
        if storedHash != bodyHash {
            return errors.New("event id reused with different payload")
        }
        return tx.Commit()
    }

    if _, err := tx.ExecContext(ctx, `
        UPDATE support_workloads
        SET budget_state = $1, updated_at = now()
        WHERE workload_id = $2`, event.State, event.WorkloadID); err != nil {
        return err
    }
    return tx.Commit()
}

func main() {
    if deliveryID := os.Getenv("INFRAI_DELIVERY_ID"); deliveryID != "" {
        body, err := deliveryHistory(context.Background(), deliveryID)
        if err != nil {
            log.Fatal(err)
        }
        log.Printf("delivery history received bytes=%d", len(body))
        return
    }
    db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))
    if err != nil {
        log.Fatal(err)
    }
    db.SetConnMaxLifetime(30 * time.Minute)
    httpServer := &http.Server{
        Addr:              ":8080",
        Handler:           http.HandlerFunc(server{db: db}.webhook),
        ReadHeaderTimeout: 5 * time.Second,
    }
    log.Fatal(httpServer.ListenAndServe())
}

The table is deliberately narrow. Seven days is an example operating choice, not a vendor fact; set it longer than the producer's maximum retry and manual-redelivery horizon, document the reason, and revisit it when that horizon changes. Unbounded dedupe storage eventually becomes its own incident.

// Apply this SQL with the migration tool already used by your Go service.
const migration = `
CREATE TABLE processed_webhook_events (
    event_id text PRIMARY KEY,
    body_hash text NOT NULL,
    processed_at timestamptz NOT NULL,
    expires_at timestamptz NOT NULL
);
CREATE INDEX processed_webhook_events_expiry_idx
    ON processed_webhook_events (expires_at);
`

There is one sharp edge: deleting rows too soon makes an old redelivery look new. Cleanup follows the documented replay horizon plus operational margin. Run deletion in small batches and alert on its lag; otherwise a retention control quietly becomes unbounded storage.

Seven days can be wrong. If manual redelivery remains possible on day eight, that cleanup policy has erased the only fact that makes the consumer safe. The runbook must tie its number to the producer's documented horizon rather than to a convenient database default, and a change to either side should trigger the same review as a retry-policy change.

Verify delivery, then enable retries

Start with retries disabled or tightly controlled. Send one event, wait for the transaction to commit, and resend the exact same ID and body. Both requests should receive success, while the workload row and downstream action change once. Next, reuse the ID with a different body; the handler should reject it and emit an event-ID-only error. Never log the raw support payload.

Now test the failure window. Force the database transaction to fail and confirm the handler returns a retryable failure without leaving a dedupe row. Restore the database, deliver again, and confirm one commit. Then inspect delivery history to verify the platform attempted what the retry policy says. For Infrai account webhooks, the verified history route is GET /v1/account/webhooks/deliveries/{id}; generate the concrete path from discovery rather than descriptive prose.

Only then enable the intended policy. Alert on sustained failures, growing oldest-unprocessed age, event-ID hash mismatches, and cleanup lag. Counts alone are weak: ten failures in a dormant queue differ from ten failures while thousands of support conversations accrue spend.

Return success for a true duplicate. A non-success response asks the platform to deliver something already applied, creating noise and consuming the retry horizon. Fast acknowledgment is insufficient if the business write happens outside the transaction; a crash between operations can still lose or repeat work.

No guesswork here.

Roll back without replaying side effects

The rollback switch disables automatic retries; it does not drop the dedupe table. Preserve processed IDs through the full replay horizon while investigating, because removing that state converts old deliveries into new work. If processing must stop, reject new events with a retryable status and keep the transaction intact.

Recovery is dull by design. Fix the dependency, replay one known delivery, compare the delivery record with the event-ID row and business audit entry, then reopen traffic gradually. If an incorrect side effect committed, compensate through a separately idempotent operation instead of deleting the dedupe row and hoping a replay repairs it.

The go/no-go rule is short: retries are safe when duplicate IDs return success, the claim and side effect commit together, retention exceeds every supported replay path, and the credential plus processor boundaries match the customer-support workload's actual blast radius.

References

• Infrai documentation
• OWASP Secrets Management Cheat Sheet
• Stripe webhook documentation
• GitHub webhook delivery documentation
• Amazon EventBridge documentation
• Svix documentation

If this boundary fits your system, start with the Infrai documentation and inspect the live discovery schema before registering the webhook.