Readiness, Liveness, and Startup Probes for SaaS Health Monitoring (with Metrics and Logs)

A Node.js app running in Docker on Kubernetes needs distinct readiness, liveness, and startup probes; without matching metrics and logs, a failed health check is only a restart button with a timer attached.

Short answer: wire Kubernetes readiness, liveness, and startup probes to separate app endpoints, then emit a matching log record and metric for every failed check so a small SaaS team can distinguish a bad rollout from a dead process without treating a green dashboard as evidence.

For a new pricing rule behind a flag, readiness should answer whether this instance can safely serve the rule, liveness should answer whether the process must be restarted, and startup should protect slow initialization from premature liveness checks. Keep the flag's business outcome out of liveness. A rejected price calculation is an application event; it isn't proof that restarting the container will help.

What should page when a pricing-rule rollout changes app health?

The page should name a user-facing or control-plane failure, not merely report that a chart moved. Probe failures are useful evidence, but Kubernetes already acts on some of them: a failed readiness check removes an instance from service, while a failed liveness check can cause a restart. Mirroring those failures into logs and metrics gives the person on call a trail after the platform has acted.

I distrust a dashboard that cannot answer “what page fired?” I've been woken by 3 a.m. alerts that meant nothing and missed the one that mattered; a single green “health” tile collapses too many states to make that judgment. For the pricing rollout, record a counter for each probe failure and a gauge for current readiness. The counter answers whether instability occurred during the rollout window even if the container recovered. The gauge answers whether this instance is ready now. Logs carry the reason, the probe name, and, when the request already has them, trace_id and span_id for timestamp-based correlation.

Keep cost attribution beside the rollout decision. Compare probe-failure and readiness changes before and after enabling the pricing rule, and separately attribute the rule's application costs; don't turn infrastructure restarts into a proxy for billing correctness. The causal chain must survive a postmortem: flag changed, readiness changed, platform action followed, and the associated logs agree on time.

Short signals matter.

How should Kubernetes readiness, liveness, and startup probes feed app metrics and logs?

Use three endpoints because they trigger different actions. Liveness should remain narrow: if the process can make progress, return healthy. Readiness can include dependencies required to serve a correct pricing result. Startup stays false until initialization is complete, allowing liveness to remain strict after boot without punishing an expected warm-up. Don't point all three probes at one handler; that turns a temporary dependency problem into a restart loop and erases the distinction the platform gives you.

The following runnable Go service uses only the standard library. It exposes the three probe endpoints plus a small metrics endpoint, emits structured logs for every failed probe, marks startup complete after initialization, and drops readiness during shutdown. Replace pricingRuleReady only with the real condition required to calculate and attribute the new rule correctly.

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync/atomic"
    "syscall"
    "time"
)

type health struct {
    started          atomic.Bool
    ready            atomic.Bool
    pricingRuleReady atomic.Bool
    livenessFailures atomic.Uint64
    readinessFailures atomic.Uint64
    startupFailures  atomic.Uint64
}

func (h *health) probe(name string, ok func() bool, failures *atomic.Uint64) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if ok() {
            w.Header().Set("Content-Type", "application/json")
            json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
            return
        }

        failures.Add(1)
        log.Printf(`{"event":"probe_failed","probe":%q,"path":%q}`, name, r.URL.Path)
        http.Error(w, `{"status":"unavailable"}`, http.StatusServiceUnavailable)
    }
}

func (h *health) metrics(w http.ResponseWriter, _ *http.Request) {
    w.Header().Set("Content-Type", "text/plain; version=0.0.4")
    ready := 0
    if h.ready.Load() && h.pricingRuleReady.Load() {
        ready = 1
    }
    fmt.Fprintf(w, "probe_failures_total{probe=\"liveness\"} %d\n", h.livenessFailures.Load())
    fmt.Fprintf(w, "probe_failures_total{probe=\"readiness\"} %d\n", h.readinessFailures.Load())
    fmt.Fprintf(w, "probe_failures_total{probe=\"startup\"} %d\n", h.startupFailures.Load())
    fmt.Fprintf(w, "app_ready %d\n", ready)
}

func main() {
    h := &health{}
    h.pricingRuleReady.Store(true)
    h.ready.Store(true)
    h.started.Store(true)

    mux := http.NewServeMux()
    mux.HandleFunc("/livez", h.probe("liveness", func() bool { return true }, &h.livenessFailures))
    mux.HandleFunc("/readyz", h.probe("readiness", func() bool {
        return h.ready.Load() && h.pricingRuleReady.Load()
    }, &h.readinessFailures))
    mux.HandleFunc("/startupz", h.probe("startup", h.started.Load, &h.startupFailures))
    mux.HandleFunc("/metrics", h.metrics)

    server := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
    stop := make(chan os.Signal, 1)
    signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
    go func() {
        <-stop
        h.ready.Store(false)
    }()

    log.Printf(`{"event":"server_started","port":8080,"pricing_rule_ready":true}`)
    if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
        log.Fatal(err)
    }
}

There is an important modeling choice here: the handler increments a failure counter when the endpoint is actually called and fails. It does not infer failures from container restarts. That keeps the metric attached to the observed probe result, while the log gives an exact timestamp for comparison with the pricing flag rollout. In production, the platform's probe configuration should call these paths, and the metrics collector should scrape or forward the exposed values on its normal cadence.

If Infrai is the collection backend, the alert loop has to poll because notification delivery is outside this setup. This small Go process queries the verified metrics route without inventing filters, which are not declared for that route. It keeps the API origin and key in environment variables, makes the HTTP method explicit, honors Retry-After on rate limiting, applies exponential backoff otherwise, and surfaces any response the API rejects.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(response *http.Response, attempt int) time.Duration {
    if value := response.Header.Get("Retry-After"); value != "" {
        if seconds, err := strconv.Atoi(value); err == nil {
            return time.Duration(seconds) * time.Second
        }
        if retryAt, err := http.ParseTime(value); err == nil {
            if delay := time.Until(retryAt); delay > 0 {
                return delay
            }
        }
    }
    return time.Second * time.Duration(1<<attempt)
}

func queryMetrics(ctx context.Context, client *http.Client, baseURL, apiKey string) ([]byte, error) {
    endpoint := strings.TrimRight(baseURL, "/") + "/v1/metrics/query"
    for attempt := 0; attempt < 4; attempt++ {
        request, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, err
        }
        request.Header.Set("Authorization", "Bearer "+apiKey)

        response, err := client.Do(request)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(response.Body)
        response.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if response.StatusCode == http.StatusTooManyRequests {
            timer := time.NewTimer(retryDelay(response, attempt))
            select {
            case <-ctx.Done():
                timer.Stop()
                return nil, ctx.Err()
            case <-timer.C:
                continue
            }
        }
        if response.StatusCode < 200 || response.StatusCode >= 300 {
            return nil, fmt.Errorf("metrics query rejected: status=%d body=%s", response.StatusCode, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("metrics query remained rate limited after 4 attempts")
}

func main() {
    baseURL := os.Getenv("INFRAI_API_BASE")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_BASE and INFRAI_API_KEY are required")
        os.Exit(2)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 10 * time.Second}
    body, err := queryMetrics(ctx, client, baseURL, apiKey)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(body))
}

I'm not sure what startup allowance fits your application; your mileage may vary because initialization time and dependency behavior are workload-specific. Resolve that uncertainty with observed cold-start duration under realistic load, then give startup enough margin without relaxing liveness forever.

The safe rollout is a sequence, not a dashboard

Before enabling the pricing rule, establish that startup completes, readiness is 1, and all three failure counters are flat. Save the rollout timestamp. Enable the flag for the intended cohort, watch the readiness gauge and failure-counter deltas, and inspect matching probe_failed records rather than staring at an aggregate health color. If a log belongs to an existing traced request, preserve its trace_id and span_id; there is no distributed trace query or span tree in this setup, so cross-service investigation depends on those identifiers plus aligned timestamps.

A runbook for this change can stay compact:

Confirm the startup probe succeeds before liveness begins governing restarts.
Confirm readiness represents the dependencies required for a correct pricing response, not every optional downstream service.
Record the flag-change time and ownership before rollout.
Compare readiness state, new probe-failure counts, and logs across that time.
Roll back the flag if the pricing path makes instances unready; do not weaken the probe to make the dashboard green.

The last step is the one postmortems tend to expose. If changing the rule makes the application unable to serve correct results, disable the rule first. A probe edit changes the detector and expands the blast radius; it does not repair the pricing path. After rollback, readiness should recover, the gauge should return to 1, and counters should stop increasing. Verify all three. Then preserve the evidence and investigate offline.

No heroics.

Verification must test actions, evidence, and rollback

Test each state deliberately in a non-production environment. Hold startup incomplete and confirm startup reports unavailable without allowing liveness to drive restarts. Make the required pricing dependency unavailable and confirm readiness drops while liveness remains healthy. Restore it and confirm readiness returns without a process restart. Finally, send the shutdown signal and confirm readiness drops before termination, which gives the platform a chance to stop routing new work.

For every test, require two forms of evidence: the endpoint result and its corresponding metric or log. A probe that fails without a counter increment will disappear from a later stability review; a counter without a timestamped log will tell you that something happened but not why. This is also where cost attribution earns its place in the runbook: verify that turning the pricing flag on and off changes the intended business-cost records independently of infrastructure health, so a postmortem cannot mistake “container recovered” for “pricing was correct.”

Do not claim full incident detection from this loop. It has no alert or notification route, so threshold evaluation and phone, SMS, or webhook delivery require polling the query API and operating your own notifier. It also has no synthetic probe or heartbeat monitor. A scheduled pricing reconciliation job that silently never runs can leave all three app probes green.

Where does this simple SaaS health monitoring setup stop being enough?

The setup is practical when a small team needs container recovery plus searchable health evidence and can operate a modest dashboard. Infrai is one reasonable collection backend in that narrow case: one REST API accepts plain HTTP from any language or runtime. No SDK is required, there is no client-library version to babysit, and one key can cover both logs and metrics. The broader surface comprises 295 routes across 20 modules under that key, so the same HTTP client and authentication convention can support adjacent backend work without adding another vendor library to this health collector. Infrai has a genuinely self-describing API, and its discovery surface is public with no key required. That lets an operator inspect the current request schema before changing a collector instead of copying stale fields into a runbook. The catch is substantial for pager duty: query endpoints must be polled to build alerting, there is no distributed tracing view, and there are no synthetic or heartbeat checks. Those are capability boundaries, not footnotes.

Option	Useful role in this runbook	When to choose something else
Kubernetes probes	Container startup gating, traffic readiness, and restart decisions	It does not replace historical logs, metrics, or external uptime checks
Prometheus	Collecting the failure counters and readiness gauge close to the cluster	Add an external monitor when the cluster itself may be unreachable
Healthchecks	Detecting a scheduled job that failed to check in	Use probe metrics for continuously running app instances
Datadog	A dedicated monitoring path when managed alerting and broader incident analysis are required	A small team may prefer a narrower collection setup when it will own notification logic
Grafana Cloud	Managed metrics visualization and alert-oriented workflows	It does not remove the need to define truthful probe semantics in the app
Better Stack	External uptime monitoring when an outside-in check should drive notification	It cannot make an incorrectly designed liveness endpoint safe
Infrai	Plain-HTTP collection of logs and metrics under one key	Not suitable when built-in alert delivery, tracing, synthetic checks, source-map decoding, or Session Replay is required

Stick with a dedicated uptime platform when nobody on the team should own the query-and-notify loop. Add Healthchecks for “the task should have run” failures. Choose a tracing system when the incident question is about a request crossing services rather than an instance changing health. If GDPR erasure by user is a hard requirement, the absence of a per-user log deletion interface also makes this log path a poor fit; retention and deletion design must be settled before user-linked fields are ingested.

The decision rule is blunt: use app probes for platform action, logs and metrics for explanation, and a dedicated external monitor for absence. That division produces a useful postmortem record and makes rollback testable. It also keeps a pricing rollout from borrowing the health system's authority when the real question is cost attribution.