devLap

Hot Reloading Go Video Services with Air During Local Development

go dev.to

Our transcoding-metadata service is written in Go. It sits behind the PHP 8.4 monolith that renders TopVideoHub, and its only job is to normalize trending-video payloads from a dozen Asia-Pacific regions before they hit SQLite. It is a small service — maybe 40 files — but during a heavy feature week I was rebuilding it by hand forty or fifty times an hour. Ctrl+C, up-arrow, go run ./cmd/aggregator, wait for the region clients to re-dial, paste a test payload, read the log, repeat. Each cycle cost eight to twelve seconds of dead time. That is not a rounding error. Over a day it is close to an hour of me staring at a terminal waiting for a binary that already knew how to compile itself.

The fix is Air, a file-watcher that rebuilds and restarts your Go binary the instant you save. It is the closest thing Go has to the PHP edit-refresh loop I was spoiled by for years. But dropping it into a real service — one with a warm SQLite connection, background region pollers, and a graceful-shutdown handler — takes more care than the README suggests. This is how we wired it up, what broke, and the config that finally made saves feel instant.

Why a compiled service needs this more than PHP does

When I edit WatchController.php, LiteSpeed re-executes the file on the next request. There is no build step, no process to restart, no in-memory state to warm back up. The CJK tokenizer that feeds our FTS5 search index is a loaded SQLite extension, and it just sits there between requests. Iteration is free.

Go is the opposite. Every change means a full compile-link-restart, and my aggregator does real work at startup:

  • It opens a SQLite handle with WAL mode and a busy-timeout pragma.
  • It spins up eight goroutines, one per region, each holding an HTTP client keyed to a different Accept-Language (ja-JP, ko-KR, zh-TW, and so on).
  • It preloads a small in-memory map of category slugs so it does not re-query on every payload.

Doing that by hand on every save is the tax. Air pays it for you, but only if you tell it exactly which files matter and give the old process time to release its resources before the new one grabs them. Skip that and you get a service that half-restarts, leaks file descriptors, or deadlocks on a SQLite lock the previous process never let go of.

Installing Air and generating a baseline config

Air ships as a single binary. Pin the version in your tooling so every engineer on the team runs the same watcher — a newer Air occasionally changes default exclude behavior, and you do not want that surprise mid-sprint.

# Install a pinned version into ./bin so it is repo-local, not global
GOBIN=$(pwd)/bin go install github.com/air-verse/air@v1.61.7

# Generate a starter config you can commit
./bin/air init   # writes .air.toml to the current directory
Enter fullscreen mode Exit fullscreen mode

The generated .air.toml is a fine starting point but it is tuned for a toy main.go, not a service with a cmd/ layout and generated code you never want to trigger a rebuild. Here is the config we actually run, annotated:

# .air.toml — TopVideoHub aggregator service
root = "."
tmp_dir = "tmp"

[build]
  # Build the real entrypoint, not the package root
  cmd = "go build -o ./tmp/aggregator ./cmd/aggregator"
  bin = "./tmp/aggregator"
  # Pass the dev env file through so we hit the local SQLite copy
  full_bin = "APP_ENV=dev ./tmp/aggregator"
  include_ext = ["go", "tmpl", "sql"]
  # Never rebuild on these — they change constantly and mean nothing to the binary
  exclude_dir = ["tmp", "testdata", "vendor", "data", ".git"]
  exclude_regex = ["_test\\.go", ".*\\.gen\\.go"]

  # Debounce: wait 200ms after the last change before building.
  # Editors that save-all can fire five events in 30ms; this collapses them.
  delay = 200

  # CRITICAL: give the old process time to run its shutdown handler
  # before SIGKILL. Default is 0, which kills instantly and strands
  # the SQLite WAL lock.
  send_interrupt = true
  kill_delay = "2s"
  stop_on_error = true

[log]
  time = true

[misc]
  clean_on_exit = true
Enter fullscreen mode Exit fullscreen mode

The two lines that took me the longest to get right are send_interrupt and kill_delay. I will come back to them, because they are the difference between a clean reload and a service that intermittently refuses to start.

The SQLite lock trap

Here is the failure that cost me an afternoon. I would save a file, Air would rebuild, and roughly one restart in five the new process would die immediately with:

database is locked
Enter fullscreen mode Exit fullscreen mode

The reason: by default Air sends SIGKILL to the running binary and starts the new one right away. But my old process was mid-flight holding the WAL lock, and SQLite in WAL mode does not release a write lock until the connection is actually closed. SIGKILL cannot be trapped, so my graceful-shutdown code — the code that calls db.Close() — never ran. The kernel eventually reclaims the fd, but not before the new process has already tried and failed to open the database.

The fix has two halves. First, tell Air to send SIGINT instead of SIGKILL and to wait a beat (send_interrupt = true, kill_delay = "2s"). Second — and this is the part people forget — your service must actually handle that signal. If your main does not trap SIGINT, all the kill_delay in the world just delays the inevitable SIGKILL.

This is the shutdown handler that made reloads deterministic:

package main

import (
    "context"
    "database/sql"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    _ "github.com/mattn/go-sqlite3"
)

func main() {
    db, err := openDB("file:data/aggregator.db?_journal_mode=WAL&_busy_timeout=5000")
    if err != nil {
        log.Fatalf("open db: %v", err)
    }

    srv := &http.Server{Addr: ":8082", Handler: routes(db)}

    // Listen for the signal Air sends on file change.
    ctx, stop := signal.NotifyContext(context.Background(),
        syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    go func() {
        log.Println("aggregator listening on :8082")
        if err := srv.ListenAndServe(); err != nil &&
            !errors.Is(err, http.ErrServerClosed) {
            log.Fatalf("listen: %v", err)
        }
    }()

    <-ctx.Done() // Air fired SIGINT — begin graceful shutdown
    log.Println("shutting down, releasing SQLite lock...")

    shutCtx, cancel := context.WithTimeout(context.Background(), 1500*time.Millisecond)
    defer cancel()

    if err := srv.Shutdown(shutCtx); err != nil {
        log.Printf("http shutdown: %v", err)
    }
    // Close the DB LAST so in-flight handlers finish first.
    if err := db.Close(); err != nil {
        log.Printf("db close: %v", err)
    }
    log.Println("clean exit")
}

func openDB(dsn string) (*sql.DB, error) {
    db, err := sql.Open("sqlite3", dsn)
    if err != nil {
        return nil, err
    }
    // One writer for SQLite; more just contend on the lock.
    db.SetMaxOpenConns(1)
    return db, db.Ping()
}
Enter fullscreen mode Exit fullscreen mode

Notice the ordering: HTTP server drains first, database closes last. If you close the DB while a handler is still running you will spray sql: database is closed errors into your logs and mistake them for a bug in your code. The 1500ms shutdown timeout is deliberately shorter than Air's 2s kill_delay, so a well-behaved shutdown always finishes before Air escalates to SIGKILL. That gap is the whole trick.

Keeping region pollers from stacking up

My aggregator does not just serve HTTP. It runs background pollers that hit each region's trending feed on a ticker. Early on, every hot reload leaked these. The old process's goroutines were killed with the process, so that was fine — but I had briefly experimented with running the pollers as a separate long-lived worker that Air did not restart, and that is where it got ugly: two generations of pollers hammering the same upstream, doubling my quota burn against the regional APIs.

The lesson: one Air process, one binary, one lifecycle. Do not try to be clever and keep part of the service warm across reloads. Let the whole thing die and come back. To make that cheap, tie every background loop to the same context that the shutdown handler cancels:

func startRegionPollers(ctx context.Context, db *sql.DB, regions []string) {
    for _, region := range regions {
        go func(region string) {
            client := &http.Client{Timeout: 8 * time.Second}
            ticker := time.NewTicker(90 * time.Second)
            defer ticker.Stop()

            // Poll once immediately so a fresh reload has data fast.
            pollRegion(ctx, client, db, region)

            for {
                select {
                case <-ctx.Done():
                    log.Printf("[%s] poller stopping", region)
                    return
                case <-ticker.C:
                    pollRegion(ctx, client, db, region)
                }
            }
        }(region)
    }
}
Enter fullscreen mode Exit fullscreen mode

Because the ticker loop selects on ctx.Done(), the same SIGINT that drains the HTTP server also stops every regional poller in lockstep. When Air brings the new binary up, pollRegion fires once on startup, so I am never waiting a full 90-second tick to see fresh ja-JP or ko-KR data after a save. Small quality-of-life detail, big difference when you are iterating on the parser for a specific region's payload.

Tuning what triggers a rebuild

The fastest reload is the one that never happens. Half of feeling instant is making sure Air only rebuilds when something that affects the binary actually changed. A few rules that earned their place in our .air.toml:

  • Exclude your data directory. My data/ holds the dev SQLite file and its WAL/SHM sidecars. Those files change on every request the service serves — if Air watches them, the service rebuilds itself in an infinite loop the moment it handles traffic. This is the single most common Air footgun.
  • Exclude generated code. We generate *.gen.go from SQL. Regenerating touches a dozen files at once; without the .*\.gen\.go exclude, one go generate triggers a rebuild storm.
  • Exclude test files. Editing _test.go should run your tests, not restart your server. _test\.go in exclude_regex keeps the two loops separate.
  • Debounce with delay. Format-on-save plus a language server can emit several write events for one keystroke. A 200ms delay collapses them into one build. Too low and you build twice; too high and saves feel laggy. 200ms is the sweet spot for our repo.

One more measurement worth doing: time your actual build. If go build for your service takes more than a second or two, you feel it on every save. Ours was creeping up because the SQLite driver recompiles CGO on a cold cache. Warming the build cache once (go build ./...) and keeping include_ext narrow — we only watch go, tmpl, and sql — kept incremental builds under 700ms. Anything you can shave here is multiplied by every save you make all day.

Running it next to the PHP stack

On my machine the Go service does not run alone. The PHP monolith, LiteSpeed, and a local Cloudflare tunnel all need to be up so I can test the full path from a rendered watch page down to the aggregator. I do not want to manage four terminals, so Air is just one process in a small compose file. The key is that Air runs inside the container with the source bind-mounted, so file events from my host editor reach the watcher:

# docker-compose.dev.yml — the Go service slice
services:
  aggregator:
    image: golang:1.23
    working_dir: /app
    # Bind-mount source so host saves are visible to Air inside the container
    volumes:
      - ./services/aggregator:/app
      - ./data:/app/data
      - go-build-cache:/root/.cache/go-build
    environment:
      APP_ENV: dev
    command: >
      sh -c "go install github.com/air-verse/air@v1.61.7 &&
             air -c .air.toml"
    ports:
      - "8082:8082"

volumes:
  go-build-cache:
Enter fullscreen mode Exit fullscreen mode

The named go-build-cache volume is not optional. Without it, every docker compose up starts from a cold build cache and your first reload after boot takes 20+ seconds while CGO recompiles the SQLite driver. With it persisted, the cache survives restarts and the first save is as fast as the hundredth. On macOS and Windows, bind-mount file events can be slow through the VM layer; if Air seems to miss saves, enabling Air's polling mode ([build] poll = true, poll_interval = 500) trades a little CPU for reliable detection. On my Linux box native inotify is instant and polling is off.

What actually changed for me

The honest measure of a dev tool is whether you stop thinking about it. Two weeks in, I do. I save, glance at the log, and the new behavior is there before my eyes finish moving from editor to terminal. The forty-restarts-an-hour tax is gone, and — less obviously — I write smaller changes now, because testing a one-line tweak no longer costs ten seconds, so I do it constantly instead of batching up a big risky edit.

If you are adding Air to a real Go service rather than a demo, the whole game is in three things:

  • Handle SIGINT yourself and close stateful resources (SQLite, connection pools) in the right order, so a reload never strands a lock.
  • Set kill_delay longer than your shutdown timeout, so graceful exit always wins the race against SIGKILL.
  • Exclude your data and generated files, so the service does not rebuild itself into an infinite loop the moment it does real work.

Get those right and a compiled Go service iterates almost as fast as the PHP pages sitting in front of it. For a small backend team shipping against a moving target — new regions, new payload shapes, new CJK edge cases every week — that reclaimed hour a day is the difference between shipping the fix today and shipping it tomorrow.

Source: dev.to

arrow_back Back to Tutorials