Pack Four Docs Without the Block Mask and Doc 4 Spends 83.78% of Its Attention on Documents It Never Saw

Sequence packing concatenates short documents into one row so you stop paying for padding. On the length distribution here that lifts utilisation from 27.98% to 99.81%.

But attention inside a row is global. Without a block-diagonal mask, the fourth document in the row spends 83.78% of its layer-1 attention on documents it has never seen — worst single token 94.90%.

👉 Live, runs in your browser: https://dev48v.infy.uk/dl/day80-sequence-packing.html

The ground truth is declared; everything else is measured

Every document is first pushed through the stack on its own, ordinary causal mask, positions 0…L−1. That unpacked run is the ground truth, and every packed configuration is scored against it token by token, coordinate by coordinate — not against a loss curve, which is exactly the instrument that cannot see either bug.

The only thing that differs between a correct packed row and a broken one:

const allow = cfg.block
  ? (i, s) => s <= i && p.docIds[s] === p.docIds[i]   // block-diagonal ∩ causal
  : (i, s) => s <= i;                                 // causal only: the mask bug
const posIds = cfg.reset ? p.local : p.tokens.map((_, i) => i);

The tolerance is a prediction, not a fudge

With the mask and per-document position reset, the packed row reproduces the unpacked run to 2.22e-16 — across five independently seeded models, every token, every coordinate.

It is not exactly zero for a stated reason. A batched attention kernel subtracts a row maximum before exponentiating, and takes that maximum over the whole causal prefix of the row, not over the current document. Subtracting a constant leaves a softmax mathematically unchanged, so this is a reassociation that moves the last bit and nothing else. The engine reproduces that kernel behaviour rather than hiding it.

The verifier then tests the explanation instead of trusting it: an independently written attention stack that subtracts no maximum at all gives the identity at exactly 0. So the residual is the shift, and nothing else. The asserted tolerance is 1e-12 — four orders above the residual, twelve below the smallest real bug.

Two independent bugs, and the arithmetic proves they are two

0.7773 is neither the sum (1.1214) nor the max (0.7607). They interact, so you cannot subtract one to estimate the other.

Two controls separate them. Zero the position table and the position bug has to vanish exactly — there is nothing left for a wrong id to index — and it drops to 1.11e-16 while the mask bug is untouched at 0.3498. And across a growing-neighbour sweep the two respond completely differently to the same change: the mask error correlates with leaked mass at r = 0.9814, the position error at r = 0.1384, because a position offset is already wrong once the neighbour has a single token.

Nothing throws, and here is why

Every attention row still sums to 1 to 4.44e-16 in all five configurations. A masked softmax renormalises over whatever it is left with, so the shape is right, the loss falls, and the model just learns something slightly wrong.

Both bugs also leave the first document in the row bit-identical — exactly 0, not nearly zero. Causal attention cannot look forward, and its positions start at 0 whether or not anyone resets them. Spot-check a packed batch on the first sequence and both bugs are invisible.

Three honest subtractions

Damage is not monotone in neighbour length. The foreign mass is strictly increasing by construction, from 0.2261 to 0.8772 mean. The error is L×gap and the second factor moves with the content of whatever tokens get appended — there is a visible dip at n = 4. "More neighbour, more damage" is a trend here, not a law.

Length-sorted batching already gets most of the win. Bucketed batches reach 99.06% utilisation against packing's 99.81%. The 3.57× is measured against arrival-order batching, which is the baseline packing is sold against but not the only alternative.

Packing does not make attention cheaper unless the mask does. Attention cost is quadratic in the row, so a packed row computed densely costs 1.027× the padded baseline — slightly more. It is the block-diagonal structure, exploited by a kernel that skips the off-diagonal blocks, that brings it to 0.1625×. The mask is not only what makes packing correct; it is what makes it fast.

The models are not trained, positions are a learned absolute table rather than RoPE, one head, two layers, float64 throughout. So which bug is larger is a property of this toy's scales and is not a general claim. The claim is that both are non-zero, independent, and invisible to the loss.

38 in-page checks, 126 verifier asserts, 0 failures.