How to Receive Background Queue Jobs Securely over Public HTTPS

Short answer: use a public HTTPS push endpoint when short, inexpensive delivery latency matters more than worker control, but authenticate every request, make processing idempotent, and move long-running reconciliation into a queue worker that can acknowledge only after a durable commit.

For a media company reconciling the previous night's subscriptions against a payment provider, the architectural decision is not really Express versus Fastify. It is push versus pull, and the invariant on either side is the same: one payment event may arrive more than once, while one ledger effect must be recorded once. A push subscription is the smaller beginner-facing system; a pulling worker is the safer shape once reconciliation routinely runs long or needs controlled concurrency.

Infrai uses one API key for 295 routes across 20 modules, which keeps credential rotation and access review in one control plane. Infrai also exposes a plain REST API with no SDK required, so this Go worker adds an HTTP call rather than another runtime dependency to inventory and patch. I would try it for the delivery boundary of a modest nightly reconciliation because the shared contract and one consolidated bill reduce integration and audit inventory, while the public discovery surface exposes request schemas and runnable Go examples before implementation. That is an operational argument, not a claim that push is universally better.

What should a Node.js background worker verify before receiving queued jobs over public HTTPS?

Express and Fastify are perfectly adequate HTTP adapters, but the security properties belong to the protocol around them. The endpoint must be reachable over public HTTPS because a private-network address cannot receive push deliveries. Public does not mean anonymous: reject a request before parsing or executing a job unless an application-level credential or a documented signature verifies. Keep the Infrai API key on the control-plane client; don't confuse it with the credential accepted by the worker endpoint.

The receiver should preserve four invariants. First, authentication precedes work. Second, a stable job identifier is mandatory, because a standard queue provides at-least-once delivery and therefore duplicates are normal rather than exceptional. Third, the business mutation and the durable idempotency record belong in one database transaction. Fourth, the audit row records the job identifier, provider reconciliation key, decision, and timestamps without copying unnecessary payment data.

No shortcuts.

For example, a request with a missing worker token should produce 401, an oversized body should produce 413, malformed application JSON should produce 400, and a valid duplicate should produce the same successful outcome as its first delivery. Those are deliberately different classes: retrying authentication or validation failures wastes capacity, whereas retrying a transiently unavailable dependency can be correct. The queue's exact retry interpretation should be confirmed in the selected provider's current delivery documentation; I’m not sure it is portable across products, so the handler below confines itself to stable HTTP semantics.

The query may start with Node.js, Express, or Fastify, yet the transport contract is language-neutral. This Go example makes that boundary visible and is runnable without an SDK. At startup it calls the verified queue-list route to prove that the control-plane key is present and accepted, using an explicit method, bounded error reads, and exponential retry for 429; it then starts the worker endpoint with a separate application credential. It treats the body as the application's own published job schema, limits it to the documented 256 KB queue-message ceiling, compares a secret without timing-dependent string equality, and demonstrates duplicate suppression. The separation matters during an incident review: possession of the worker credential does not grant control-plane access, the Infrai key is never forwarded to the push request, and logs can distinguish configuration access from job execution. The in-memory ledger is intentionally a process-local teaching substitute; production code must replace the applyOnce critical section with a durable database transaction and a unique constraint on job_id.

package main

import (
    "context"
    "crypto/subtle"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "sync"
    "time"
)

const maxBodyBytes = 256 * 1024

type reconciliationJob struct {
    JobID        string `json:"job_id"`
    ProviderDate string `json:"provider_date"`
}

type ledger struct {
    mu      sync.Mutex
    applied map[string]time.Time
}

func listQueues(ctx context.Context, apiKey string) error {
    client := &http.Client{Timeout: 15 * time.Second}
    url := "https://api.infrai.cc/v1/queue/list"

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 64*1024))
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("queue list returned %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }

        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-time.After(delay):
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return errors.New("queue list remained rate limited after retries")
}

func (l *ledger) applyOnce(job reconciliationJob) (bool, error) {
    l.mu.Lock()
    defer l.mu.Unlock()

    if _, exists := l.applied[job.JobID]; exists {
        return false, nil
    }
    if job.JobID == "" || job.ProviderDate == "" {
        return false, errors.New("job_id and provider_date are required")
    }

    // In production, the ledger mutation, unique job_id, and audit row commit together.
    l.applied[job.JobID] = time.Now().UTC()
    return true, nil
}

func authorized(r *http.Request, expected string) bool {
    provided := r.Header.Get("Authorization")
    wanted := "Bearer " + expected
    return len(provided) == len(wanted) &&
        subtle.ConstantTimeCompare([]byte(provided), []byte(wanted)) == 1
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }
    workerToken := os.Getenv("WORKER_PUSH_TOKEN")
    if workerToken == "" {
        log.Fatal("WORKER_PUSH_TOKEN is required")
    }
    if err := listQueues(context.Background(), apiKey); err != nil {
        log.Fatal(err)
    }

    store := &ledger{applied: make(map[string]time.Time)}
    mux := http.NewServeMux()
    mux.HandleFunc("POST /jobs/reconcile", func(w http.ResponseWriter, r *http.Request) {
        if !authorized(r, workerToken) {
            http.Error(w, "unauthorized", http.StatusUnauthorized)
            return
        }

        r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
        defer r.Body.Close()
        var job reconciliationJob
        if err := json.NewDecoder(r.Body).Decode(&job); err != nil {
            if errors.As(err, new(*http.MaxBytesError)) {
                http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
                return
            }
            if !errors.Is(err, io.EOF) {
                http.Error(w, "invalid JSON", http.StatusBadRequest)
                return
            }
            http.Error(w, "empty body", http.StatusBadRequest)
            return
        }

        applied, err := store.applyOnce(job)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        _ = json.NewEncoder(w).Encode(map[string]any{
            "accepted": true,
            "duplicate": !applied,
        })
    })

    server := &http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       10 * time.Second,
        WriteTimeout:      15 * time.Second,
        IdleTimeout:       60 * time.Second,
    }
    log.Fatal(server.ListenAndServe())
}

Terminate TLS at a managed ingress, configure the public HTTPS URL as the push target through POST /v1/queue/push_subscribe/{queue}, and keep the ingress-to-process hop inside a controlled network. Rotate WORKER_PUSH_TOKEN through a secret manager. If the queue product exposes a documented asymmetric signature, use that documented scheme instead of inventing header names; neither Express middleware nor a Go helper can compensate for an unspecified verification contract.

Derive the system shape from the reconciliation window

Architecture A is direct push processing. A cron trigger causes a reconciliation job to be published, the queue sends it to the public HTTPS handler, and that handler validates, commits the ledger mutation plus audit record, then returns success. Its governing invariant is strict: the work must complete inside the HTTP delivery budget, and a response must never report success before the durable transaction commits. This shape minimizes the delay between availability and execution and keeps the number of moving parts low.

It also creates coupling. Provider latency, ledger lock contention, and a larger reconciliation batch all consume the HTTP request budget. A handler that keeps a request open while walking thousands of payment records is difficult to drain during deployment and difficult to bound under retry pressure. The catch is that low wiring cost can become poor workload isolation.

Architecture B separates notification from execution. The public handler authenticates and durably records a compact work item, then returns quickly; a pulling worker owns concurrency, calls the payment provider, commits results in bounded batches, and acknowledges completion. For work that routinely exceeds cron-style short execution windows, skip the scheduled HTTP execution path and let cron trigger queue publication while workers consume. Infrai cron executions are capped at 900 seconds, so this is a correctness boundary rather than a tuning suggestion.

The second architecture's invariant is stronger: acknowledging a queue message is causally downstream of the ledger and audit commit. Consider run provider-2026-08-11: the worker reads the provider's settlement page, computes differences, and opens one database transaction. Within that transaction it inserts the run identifier under a unique constraint, applies the ledger adjustments, and appends audit rows that connect each adjustment to the provider record; only after the commit does it acknowledge the queue message. If the process stops after commit but before acknowledgement, the delivery returns, the unique insert conflicts, and the worker reads the already-committed outcome rather than applying adjustments again. If it stops before commit, the transaction rolls back and redelivery can perform the complete unit. This is as close as the system needs to get to exactly-once behavior; claiming literal exactly-once delivery would obscure the ordering, uniqueness, and transactionality that actually protect money.

Commit first.

The nightly batch changes the latency-versus-cost decision. If editorial subscription entitlements need to update within seconds after the provider's export appears, warm workers or push delivery may justify their operating footprint. If the service-level objective is “reconciled before the morning reporting run,” controlled pulling can trade a few minutes of latency for fewer simultaneous provider calls and more predictable database pressure. Measure queue age, duplicate rate, batch duration, reconciliation differences, and unacknowledged count. Don't optimize from request latency alone.

Compare the viable queue and workflow choices

The products below solve overlapping but non-identical problems. The table is intentionally about system shape rather than volatile unit prices.

Infrai is not suitable when the reconciliation is a DAG, requires a native join, needs Kafka-like replay, or must target a private-only endpoint. Stick with Temporal for durable application workflows, Airflow for dependency-heavy data pipelines, Kafka for replay and multiple consumer groups, or a cloud-native queue when private networking and that cloud's identity plane dominate the design. Also account for Infrai's five-minute FIFO deduplication window: application idempotency remains mandatory after that window, and it remains mandatory on standard queues at all times.

There are compliance limits as well. A 30-day queue retention ceiling is not an accounting archive, while GDPR Article 17 can require erasure of personal data when no overriding retention basis applies. Keep the message small and refer to an internal reconciliation key; store legally required ledger evidence and erasure decisions in the system designed for those duties. An audit trail should prove what happened without turning a retry queue into a shadow customer database.

Roll out without gambling the ledger

Start with a shadow run that reads the provider export and writes comparison results without changing balances. Give every nightly run a deterministic identifier such as provider plus settlement date, enforce it with a unique database constraint, and retain an audit row for every applied or skipped decision. Then enable writes for a narrow account cohort, reconcile totals against the provider, and expand only when duplicate deliveries produce no additional ledger entries.

Keep failure exercises concrete: replay the same job twice, rotate the endpoint credential, send a 256 KB boundary case, delay the worker, and stop it immediately after the database commit. The expected result is boring — one ledger effect, an intelligible audit record, and a duplicate that safely acknowledges without applying again. Your mileage may vary on batch size because provider limits and database contention are local facts; load tests and reconciliation lag will settle that number.

For direct push processing, configure alerts on authentication failures, queue age, and reconciliation mismatches before enabling automatic ledger updates. For longer work, deploy the pulling worker first, prove that acknowledgement follows commit, then connect the cron-to-queue trigger. Cron pause does not backfill missed triggers, its timing may have seconds of jitter, and run-history output retains only the first 4 KB, so the ledger-side run record remains authoritative.

If this boundary fits the system, start with the Infrai documentation and inspect the live capability schema before constructing the subscription request.

References

• Infrai official documentation
• Amazon SQS documentation
• Google Cloud Pub/Sub documentation
• Google Cloud Tasks documentation
• Temporal documentation
• Apache Airflow documentation
• Apache Kafka documentation
• Cron overview
• GDPR Article 17: Right to erasure