Go GC Write Barriers Under Concurrent Mutation: Tricolor Invariants, STW Phases, and the Allocation Pressure Tradeoff

The Problem Nobody Measures Correctly

Go's GC pause story is usually told as: "STW is under a millisecond, you're fine." That framing collapses three distinct cost categories—write barrier overhead during mark, STW mark termination latency, and sweep amortization—into a single number that obscures where your service is actually losing time. For a microservice processing 50k RPS with shared heap state mutated across goroutines, those categories have different optimization paths, and conflating them leads to cargo-culted GOGC tuning that improves one metric while worsening another.

This article examines the GC mechanics that matter operationally: the tricolor invariant and how write barriers enforce it under concurrent mutation, where STW phases actually occur in a Go 1.21+ collector, and how allocation rate and object graph shape determine whether you pay in pause time, CPU overhead, or both.

Tricolor Invariant and Why Concurrent Mutation Breaks It

Go's mark phase runs concurrently with your application goroutines. The collector classifies every object as white (unvisited), grey (enqueued for scanning), or black (scanned, children enqueued). The invariant the collector must maintain: no black object holds a pointer to a white object at mark termination. Violation means a live object gets collected.

Concurrent mutation creates two hazard patterns:

1. A goroutine writes a pointer to a white object into a black object's field.
2. A goroutine destroys the only grey-reachable path to a white object before the collector processes it.

Either pattern can make a live object invisible to the mark phase.

Write Barriers: What the Compiler Actually Emits

Go uses a hybrid write barrier (introduced in Go 1.17, refined since) that satisfies Dijkstra's insertion barrier and Yuasa's deletion barrier simultaneously. Every pointer write in your Go code that the compiler cannot prove is stack-local gets instrumented. The emitted pseudocode for a heap pointer write is:

// Compiler-generated around: obj.field = ptr
if writeBarrierEnabled {
    shade(obj.field)  // grey the old value (Yuasa)
    shade(ptr)         // grey the new value (Dijkstra)
}
obj.field = ptr

shade marks the target grey and enqueues it for the mark worklist if it's currently white. This runs in your goroutine, on your goroutine's P, inline with your mutation. It is not a safepoint; it is synchronous overhead on every instrumented pointer store.

The critical implication: write barrier cost scales with pointer store frequency, not allocation rate. A service that allocates rarely but mutates shared pointer-heavy structs heavily pays more in write barrier overhead than a service that allocates aggressively but stores into fresh, short-lived objects.

Demonstrating Write Barrier Sensitivity

Consider a concurrent LRU cache backed by a doubly-linked list and a map—a pattern common in Redis-backed services that maintain a local hot tier:

type entry struct {
    key        string
    val        []byte
    prev, next *entry
}

type LRU struct {
    mu   sync.Mutex
    m    map[string]*entry
    head *entry
    tail *entry
    cap  int
}

func (l *LRU) promote(e *entry) {
    // Unlink and move to head—four pointer writes per operation
    if e.prev != nil { e.prev.next = e.next }  // write barrier
    if e.next != nil { e.next.prev = e.prev }  // write barrier
    e.next = l.head                            // write barrier
    e.prev = nil
    if l.head != nil { l.head.prev = e }       // write barrier
    l.head = e                                 // write barrier
}

Each promote call under an active mark phase triggers up to five instrumented pointer stores. At 50k cache hits/sec this is 250k barrier invocations per second—not catastrophic, but measurable as CPU overhead distinct from mark work. You can observe it via runtime/metrics:

samples := []metrics.Sample{
    {Name: "/gc/scan/globals:bytes"},
    {Name: "/cpu/classes/gc/mark/assist:cpu-seconds"},
    {Name: "/cpu/classes/gc/mark/dedicated:cpu-seconds"},
}
metrics.Read(samples)

/cpu/classes/gc/mark/assist:cpu-seconds rising proportionally with cache promotion rate, not just allocation rate, is the signature that write barrier overhead is your primary GC cost—not heap size.

Where STW Actually Occurs in Go 1.21+

The narrative that Go GC is "mostly concurrent" is true but incomplete. STW phases still exist:

STW Mark Setup: Enables write barriers, takes stack snapshots, roots the mark worklist. Duration is proportional to goroutine count (each goroutine must reach a safepoint). A service with 10k live goroutines during a bursty period pays more here than one with 500.

STW Mark Termination: Verifies no grey objects remain after concurrent mark drains. The time here is proportional to residual grey objects that mutation introduced after the last drain pass—effectively, it is a function of write rate in the final drain window. High write rates mean more shading, more grey objects, longer termination.

Sweep: Concurrent and amortized across allocations. Not STW but adds latency to allocation paths proportional to unswept span density.

The common failure mode: a team observes p99 latency spikes correlating with GC cycles and reduces GOGC expecting shorter pauses. Lower GOGC increases GC frequency, which increases total time in mark setup and write barrier overhead per second, while reducing individual pause duration only if concurrent mark terminates quickly. For write-heavy workloads the net effect can be negative.

Allocation Shape Drives GC Character

Two allocation patterns that appear similar at the pprof heap level produce radically different GC behavior:

Pattern A: Many small, short-lived pointer-bearing structs

High allocation rate, low mark work per object, high escape rate forces heap allocation, mark worklist churns fast. Symptoms: frequent GC cycles, high mark/assist CPU, moderate pause duration.

Pattern B: Few large, long-lived structs with deep pointer graphs

Low allocation rate, high mark work per object, each GC cycle does more scanning per collection. Symptoms: infrequent GC cycles, low assist CPU between cycles, but longer mark termination when cycles do occur.

For Pattern A, the lever is reducing escapes—keep objects stack-local where the compiler can prove it:

// Escapes to heap: returned pointer forces heap allocation
func newRequest(id string) *Request {
    return &Request{ID: id}
}

// Stays stack-local if caller inlines and doesn't store the address
func processRequest(id string) Result {
    req := Request{ID: id}  // stack-allocated if not captured
    return compute(req)
}

Verify with go build -gcflags='-m=2'—the compiler reports every escape decision and its reason. "moved to heap: req" with reason "too large" or "address taken" tells you exactly what forced the allocation.

For Pattern B, the lever is flattening pointer graphs. Replace linked structures with index-based slices where traversal patterns allow:

// Pointer graph: each node escapes, each pointer is a barrier site
type Node struct { Val int; Children []*Node }

// Index graph: single backing slice, no interior pointers
type FlatNode struct { Val int; ChildrenIdx []int32 }
var nodes []FlatNode

The flat representation keeps the entire structure in one heap object. The mark phase scans []int32 fields but finds no pointers—the GC skips them. Mark work drops proportionally.

The GOGC and GOMEMLIMIT Interaction

Go 1.19 introduced GOMEMLIMIT, which caps heap growth absolutely. The interaction with GOGC is non-obvious and operationally significant:

• GOGC=100 (default): GC triggers when live heap doubles. With unlimited memory this is permissive.
• GOMEMLIMIT=512MiB: GC also triggers when heap approaches the limit, regardless of GOGC. This is a hard ceiling enforced by a separate pacing algorithm.

In a container environment with memory limits (ECS, Kubernetes), setting GOMEMLIMIT to ~90% of the container's memory limit prevents OOM kills from heap growth bursts while giving the GC pacing algorithm a target to optimize against. The GC will increase collection frequency to stay under the limit, trading CPU for memory headroom.

The failure mode: setting GOMEMLIMIT too close to the container limit (100%) while running high-allocation workloads causes the GC to thrash—continuous collection to stay under the limit, high assist overhead, degraded throughput. Monitor /memory/classes/heap/released:bytes and /gc/cycles/total:gc-cycles; if cycles/sec spikes with heap near limit, the limit is too tight.

Decision Framework

Before tuning, classify your GC cost:

Measure before tuning. runtime/metrics provides GC cycle timestamps, pause durations, heap size at trigger, and CPU class breakdowns without external tooling. Build a metrics pipeline from it before reaching for GOGC changes—the symptom you see in p99 latency almost never maps to the knob you'd intuitively turn first.