Last winter I got paged at 02:00 because a viewer in São Paulo could not start a stream. Our origin dashboard was a wall of green. Origin was fine. What was not fine was the edge in gru (São Paulo): its cache had gone cold after a botched purge, every segment request was a miss punching through to origin across the Atlantic, and time-to-first-byte for the first .ts segment had climbed past four seconds. A single global uptime ping — the kind that hits your homepage from one datacenter every minute — will never see this. It hit our US edge, got a 200, and reported 100% uptime while an entire continent buffered.
That night I stopped trusting single-point uptime checks and built a multi-region health-check aggregator that probes every edge the way a real player does. This post is the design I run in production at TrendVidStream, a global streaming-discovery service spread across eight regions, and the tradeoffs behind each piece. The stack is deliberately boring: a small Go probe, PHP 8.4 for aggregation and scoring, SQLite (with FTS5 for incident search), plain cron, and FTP to push a static status snapshot to each origin. No Kubernetes, no Prometheus, no time-series database. It fits on one box and it catches the failures that mattered.
Why a single uptime ping lies about video
Most uptime tools were built for web pages. They fetch one URL from one location, check the status code, maybe assert a string is present, and move on. That model works when your product is a page. It falls apart for video, because video delivery is not one request — it is a manifest plus a long tail of segment requests, and each of those is served by a regional edge that caches independently.
Here is what a single homepage ping structurally cannot see:
- Per-region edge health. Your US edge can be perfect while your Sydney edge is serving stale manifests or cold-missing every segment.
-
Segment latency versus manifest latency. The
.m3u8manifest is tiny and almost always cached. The segments are where bytes and buffering live. A fast manifest tells you nothing about whether segment zero arrives before the buffer drains. -
Cache-miss cliffs. After a purge or a cold deploy, an edge returns
200on everything but every response is a miss going back to origin. Status codes look healthy; latency is catastrophic. -
Partial transfers. A response can start with a
200header and then stall mid-body. If your check only reads headers, it scores that as success.
The fix is not a better uptime vendor. It is to measure the thing your viewers actually experience — a manifest fetch followed by a segment fetch — from the perspective of each region, and to aggregate those measurements into one score you can alert on.
What healthy actually means for an edge
Before writing a single probe I wrote down the service-level indicators. For a video edge, four signals cover the vast majority of real incidents:
-
Manifest availability — does
GET /master.m3u8return200with a body that parses as a playlist? -
Segment availability — does the first
.ts/.m4ssegment return200? - Segment time-to-last-byte — how long to fully transfer that segment, not just receive the header?
- Cold-connection cost — TLS handshake plus first byte on a fresh connection, because most viewers arrive without a warm keep-alive.
The critical number is the segment budget. Our health playlist uses 6-second segments. If a 6-second segment takes longer than roughly 6 seconds to arrive, the player's buffer drains faster than it refills and the viewer stalls. In practice you want a comfortable margin, so I set a hard budget of 1500 ms for the health segment. Anything slower is degraded even if it eventually returns 200. That single threshold turned out to be the most valuable line in the whole system — it is what would have caught the São Paulo incident an hour before the page.
Probing an edge like a player, not a monitor
The probe is the one component I wrote in Go, for two reasons: goroutines make probing eight regions concurrently trivial, and I wanted precise control over connection reuse. A real cold viewer does not benefit from a warm keep-alive, so the probe disables it and fully drains every response body to measure true time-to-last-byte.
package main
import (
"crypto/tls"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"sync"
"time"
)
type Probe struct {
Region string `json:"region"`
Edge string `json:"edge"`
ManifestMS float64 `json:"manifest_ms"`
SegmentMS float64 `json:"segment_ms"`
Status int `json:"status"`
OK bool `json:"ok"`
Err string `json:"err,omitempty"`
TS int64 `json:"ts"`
}
// edges maps a logical region to the edge hostname that serves it.
var edges = map[string]string{
"us-east": "edge-iad.trendvidstream.com",
"us-west": "edge-sjc.trendvidstream.com",
"eu-west": "edge-fra.trendvidstream.com",
"eu-north": "edge-arn.trendvidstream.com",
"sa-east": "edge-gru.trendvidstream.com",
"ap-south": "edge-bom.trendvidstream.com",
"ap-east": "edge-hkg.trendvidstream.com",
"oceania": "edge-syd.trendvidstream.com",
}
const manifestPath = "/hls/health/master.m3u8"
const segmentPath = "/hls/health/seg0.ts"
func timedGET(client *http.Client, url string) (float64, int, error) {
start := time.Now()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("User-Agent", "tvs-healthcheck/1.0")
resp, err := client.Do(req)
if err != nil {
return 0, 0, err
}
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body) // force full transfer, not just headers
return float64(time.Since(start).Microseconds()) / 1000.0, resp.StatusCode, nil
}
func probe(region, edge string) Probe {
client := &http.Client{
Timeout: 8 * time.Second,
Transport: &http.Transport{
TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12},
DisableKeepAlives: true, // measure a cold connection like a real viewer
},
}
p := Probe{Region: region, Edge: edge, TS: time.Now().Unix()}
mms, status, err := timedGET(client, "https://"+edge+manifestPath)
if err != nil {
p.Err = "manifest: " + err.Error()
return p
}
p.ManifestMS, p.Status = mms, status
if status != 200 {
return p
}
sms, sstatus, err := timedGET(client, "https://"+edge+segmentPath)
if err != nil {
p.Err = "segment: " + err.Error()
return p
}
p.SegmentMS = sms
p.OK = sstatus == 200 && sms < 1500 // 1.5s budget for a 6s segment
return p
}
func main() {
var wg sync.WaitGroup
var mu sync.Mutex
results := make([]Probe, 0, len(edges))
for region, edge := range edges {
wg.Add(1)
go func(r, e string) {
defer wg.Done()
p := probe(r, e)
mu.Lock()
results = append(results, p)
mu.Unlock()
}(region, edge)
}
wg.Wait()
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(results); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Three details carry the weight here. DisableKeepAlives: true forces a fresh TCP and TLS handshake per probe, matching a first-time viewer rather than a warm CDN connection. io.Copy(io.Discard, resp.Body) reads the entire segment body, so SegmentMS is a true time-to-last-byte and a stalled mid-transfer counts as slow. And the whole thing is a two-stage check: if the manifest is not 200, we never bill the region for a slow segment — the failure is already the manifest. The binary emits a JSON array on stdout, which keeps the boundary between languages dead simple: Go measures, PHP decides.
Aggregating eight regions into one store
I store everything in a single SQLite file. For a service that already deploys over FTP, SQLite is the natural fit: it is one file you can copy, it needs no server, WAL mode gives me concurrent reads while a write is in flight, and FTS5 ships in the amalgamation so incident search costs nothing extra. There are three tables — raw probes, a rolling score per region, and a full-text incident log.
-- health_probe: one row per region per run (the raw history)
CREATE TABLE IF NOT EXISTS health_probe (
id INTEGER PRIMARY KEY,
region TEXT NOT NULL,
edge TEXT NOT NULL,
manifest_ms REAL,
segment_ms REAL,
status INTEGER,
ok INTEGER NOT NULL DEFAULT 0,
err TEXT,
ts INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_probe_region_ts
ON health_probe(region, ts DESC);
-- health_score: current smoothed state, one row per region (upserted each run)
CREATE TABLE IF NOT EXISTS health_score (
region TEXT PRIMARY KEY,
score REAL NOT NULL, -- EWMA, 0..100
state TEXT NOT NULL, -- healthy | degraded | down
last_ok_ts INTEGER,
updated_ts INTEGER NOT NULL
);
-- incident_fts: human-readable notes so ops can grep the history
CREATE VIRTUAL TABLE IF NOT EXISTS incident_fts USING fts5(
region, state, note, ts UNINDEXED, tokenize = 'porter'
);
Ingest is a thin PHP 8.4 CLI script. It reads the probe JSON from stdin and writes one transaction per run. Batching the inserts into a single transaction matters a lot on SQLite — without it, each INSERT is its own fsync and eight rows can take longer than the entire probe.
<?php
declare(strict_types=1);
final class HealthIngest
{
public function __construct(private readonly PDO $db) {}
/** Ingest one probe run (decoded from the Go probe's JSON array). */
public function ingest(array $probes): void
{
$stmt = $this->db->prepare(
'INSERT INTO health_probe
(region, edge, manifest_ms, segment_ms, status, ok, err, ts)
VALUES (:region, :edge, :manifest_ms, :segment_ms, :status, :ok, :err, :ts)'
);
$this->db->beginTransaction();
foreach ($probes as $p) {
$stmt->execute([
':region' => $p['region'],
':edge' => $p['edge'],
':manifest_ms' => $p['manifest_ms'] ?? null,
':segment_ms' => $p['segment_ms'] ?? null,
':status' => $p['status'] ?? null,
':ok' => !empty($p['ok']) ? 1 : 0,
':err' => $p['err'] ?? null,
':ts' => $p['ts'],
]);
}
$this->db->commit();
}
}
$pdo = new PDO('sqlite:' . __DIR__ . '/data/health.db');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->exec('PRAGMA journal_mode = WAL;');
$pdo->exec('PRAGMA busy_timeout = 5000;');
$raw = stream_get_contents(STDIN);
$probes = json_decode($raw, true, flags: JSON_THROW_ON_ERROR);
(new HealthIngest($pdo))->ingest($probes);
fprintf(STDERR, "ingested %d probes\n", count($probes));
Turning raw probes into a stable score
Raw probes are noisy. A single slow segment — a routing hiccup, a garbage-collection pause on an edge, a transient packet loss — should not page anyone. But two bad runs in a row absolutely should. The classic mistake is to alert on the latest sample; you get a firehose of flapping alerts and your team learns to ignore them, which is worse than no monitoring at all.
Two techniques fix this. First, an exponentially weighted moving average (EWMA) smooths the per-region score so one bad sample nudges it rather than crashing it. Second, hysteresis: the thresholds for entering a bad state and for leaving it are different. A region has to climb back above a clearly-good line before we call it healthy again, so a region hovering right at the boundary does not oscillate between degraded and healthy every two minutes.
<?php
declare(strict_types=1);
final class HealthScorer
{
private const ALPHA = 0.3; // EWMA weight for the newest run
private const DEGRADED_BELOW = 85.0;
private const DOWN_BELOW = 50.0;
private const RECOVER_ABOVE = 90.0; // hysteresis: must clear this to leave a bad state
public function __construct(private readonly PDO $db) {}
public function recompute(int $now): void
{
$regions = $this->db->query(
'SELECT DISTINCT region FROM health_probe'
)->fetchAll(PDO::FETCH_COLUMN);
foreach ($regions as $region) {
$latest = $this->db->prepare(
'SELECT ok, manifest_ms, segment_ms FROM health_probe
WHERE region = :r ORDER BY ts DESC LIMIT 1'
);
$latest->execute([':r' => $region]);
$row = $latest->fetch(PDO::FETCH_ASSOC);
if ($row === false) {
continue;
}
$sample = $this->sampleScore($row);
$prev = $this->currentScore($region);
$score = $prev === null
? $sample
: (self::ALPHA * $sample) + ((1 - self::ALPHA) * (float) $prev['score']);
$state = $this->nextState($prev['state'] ?? 'healthy', $score);
$lastOk = (int) $row['ok'] === 1 ? $now : null;
$this->upsert($region, $score, $state, $lastOk, $now);
}
}
/** Turn one probe into a 0..100 sample: latency-weighted, hard zero on failure. */
private function sampleScore(array $row): float
{
if ((int) $row['ok'] !== 1) {
return 0.0;
}
$seg = (float) ($row['segment_ms'] ?? 1500.0);
// 300ms segment => 100, 1500ms segment => 0, linear in between.
return max(0.0, min(100.0, (1500.0 - $seg) / 12.0));
}
private function nextState(string $current, float $score): string
{
// Hysteresis: leaving a bad state requires clearing RECOVER_ABOVE,
// so a region hovering near a threshold does not flap alerts.
if ($current !== 'healthy' && $score < self::RECOVER_ABOVE) {
return $score < self::DOWN_BELOW ? 'down' : 'degraded';
}
if ($score < self::DOWN_BELOW) return 'down';
if ($score < self::DEGRADED_BELOW) return 'degraded';
return 'healthy';
}
private function currentScore(string $region): ?array
{
$stmt = $this->db->prepare(
'SELECT score, state FROM health_score WHERE region = :r'
);
$stmt->execute([':r' => $region]);
$row = $stmt->fetch(PDO::FETCH_ASSOC);
return $row === false ? null : $row;
}
private function upsert(string $region, float $score, string $state, ?int $lastOk, int $now): void
{
$stmt = $this->db->prepare(
'INSERT INTO health_score (region, score, state, last_ok_ts, updated_ts)
VALUES (:r, :s, :st, :lok, :now)
ON CONFLICT(region) DO UPDATE SET
score = excluded.score,
state = excluded.state,
last_ok_ts = COALESCE(excluded.last_ok_ts, health_score.last_ok_ts),
updated_ts = excluded.updated_ts'
);
$stmt->execute([
':r' => $region, ':s' => $score, ':st' => $state,
':lok' => $lastOk, ':now' => $now,
]);
}
}
Tune ALPHA to your probe cadence. At a two-minute interval, 0.3 means a region has to be bad for roughly three or four consecutive runs before it crosses into degraded — about six to eight minutes of sustained trouble, which is short enough to matter and long enough to ignore a single flaky sample. The COALESCE on last_ok_ts is a small but important touch: it preserves the last-known-good timestamp even across failing runs, so an alert can say down for 14 minutes instead of just down.
Searching incident history with FTS5
When a region flips state, a small hook writes a human-readable note into incident_fts — something like "eu-north segment_ms 2100 after purge, cache cold". Months later, when the same edge misbehaves, I do not want to scroll a Grafana board; I want to grep. FTS5 makes that a one-liner: SELECT region, note FROM incident_fts WHERE incident_fts MATCH 'gru AND (cold OR purge)' ORDER BY ts DESC LIMIT 20;. The porter tokenizer stems purge/purged/purging together, so I find the pattern regardless of how tired I was when I wrote the note. This is the payoff of keeping everything in SQLite rather than a metrics store: the operational log and the metrics live in the same file and speak the same query language.
Wiring it into multi-region cron and FTP deploy
The whole thing runs from one cron entry every two minutes. The last step is the one that matches how the rest of the platform ships: it exports a static status.json snapshot and pushes it to each origin over FTP, so every site can render a live status widget without ever querying the aggregator directly. Static file, cache-friendly, zero runtime dependency.
#!/usr/bin/env bash
set -euo pipefail
ROOT="/opt/tvs/healthcheck"
# 1. Probe all 8 regions in parallel (Go binary emits a JSON array on stdout).
"$ROOT/bin/probe" > "$ROOT/tmp/run.json"
# 2. Ingest raw probes, then recompute smoothed scores (PHP 8.4 CLI).
php "$ROOT/ingest.php" < "$ROOT/tmp/run.json"
php "$ROOT/score.php"
# 3. Export a static snapshot and push it to each origin via FTP.
php "$ROOT/export_status.php" > "$ROOT/public/status.json"
lftp -u "$FTP_USER,$FTP_PASS" "$FTP_HOST" <<'EOF'
set ftp:ssl-allow yes
set ssl:verify-certificate no
cd public_html/health
put -O . /opt/tvs/healthcheck/public/status.json
bye
EOF
The cron line is simply */2 * * * * /opt/tvs/healthcheck/run.sh >> /var/log/tvs-health.log 2>&1. Because probing is parallel in Go, eight regions finish in about the time of the slowest single region — usually under two seconds — so a two-minute cadence leaves enormous headroom. If you run more origins, you scale by adding rows to the edges map in Go, not by adding infrastructure.
One operational note that cost me an afternoon: keep the FTP push idempotent and never let it block the scoring step. If the upload fails, the aggregator has already recorded truth in SQLite; the stale status.json on the edge is a cosmetic problem, not a data-loss one. Order the script so measurement and scoring happen first and the cosmetic push happens last.
What this caught that green dashboards did not
Since it went live, the aggregator has flagged failures that our old single-point ping was structurally blind to:
- A cold-cache cliff in
sa-eastafter a purge —200s everywhere, segment time-to-last-byte at 3–4s, exactly the São Paulo pattern that started this. - A TLS certificate that renewed on seven edges but silently failed on
ap-east, caught by the probe's fresh-handshake-per-run behavior. - A stale manifest in
oceaniapointing at deleted segments — the manifest returned200, but the segment fetch 404'd, soOKcorrectly stayed false. - Intermittent mid-transfer stalls in
eu-norththat a header-only check scored as healthy and that only the full-body read exposed.
None of these tripped a status-code-only monitor. All of them affected real viewers in exactly one region.
Conclusion
The lesson is not build a fancier monitor. It is measure what your viewer measures. For video that means a manifest fetch plus a full segment transfer, from every region, on a cold connection, smoothed enough to avoid flapping and stored somewhere you can search later. The implementation is small — a Go probe, three SQLite tables, two PHP scripts, and a cron entry — and it runs comfortably on one box next to everything else. If you deliver video across regions and your dashboard is a single green light, that light is telling you about one datacenter and lying by omission about the rest. Probe every edge, smooth the noise, and let the score — not a status code — decide when to wake someone up.