Model A Beats B on Every Slice and Loses 57.60% to 79.06%: 9,510 Reversals, 0 on a Shared Mixture

Two models, one benchmark, three slices. Model A is ahead on all three — 96.0 / 60.0 / 50.0 against 88.0 / 50.0 / 40.0.

Pooled, A scores 57.60% and B scores 79.06%. A 21.5-point defeat assembled entirely out of victories.

👉 Live, runs in your browser: https://dev48v.infy.uk/ml/day80-simpsons-paradox.html

Declared, not sampled — and no winner decided by a float

Every count on the page is a declared integer. Every accuracy is an exact rational, and every comparison is settled by BigInt cross-multiplication, so a reversal is proved rather than observed through rounding:

// every winner is decided by cross-multiplication, never by a float
const rcmp = (a, b) => { const l = a.n * b.d, r = b.n * a.d;
                         return l < r ? -1 : l > r ? 1 : 0; };

There is no p-value anywhere, because nothing is estimated. What is measured is an exhaustive sweep.

The identity the whole thing rests on

Pooled (micro) accuracy is not a different kind of quantity from the per-slice rates. It is their mean, weighted by slice size:

micro(M) = Σ c_g / Σ n_g = Σ (n_g/N) · (c_g/n_g)
macro(M) = (1/G)  ·  Σ (c_g/n_g)

The two metrics differ in exactly one place — the weights — and the entire paradox lives there. A weighted mean with fixed non-negative weights is monotone in its inputs, so per-slice dominance would be inherited by the pooled number. The reversal requires the two models to carry different weights.

Which is why macro and micro disagree in sign in 9,510 of 9,510 reversals found. Not usually. All of them.

The smallest reversal in existence is nine observations

Sweeping every two-group table with per-cell n from 1 to 8 — 3,748,096 tables, of which 774,400 put A strictly ahead in both groups — gives 9,510 reversals. The minimum total sample across all of them is 9, reached by exactly 4 tables:

group    model A        model B
g1       1/1 = 100%     2/3 = 66.7%
g2       1/4 =  25%     0/1 =  0%
pooled   2/5 =  40%     2/4 = 50%

Four cells and nine data points are enough to rank two models backwards. There is nothing subtle or large-sample about it.

At the other end: the largest pooled reversal found is 5/9 against 2/9 — 55.6 points — and the largest per-group lead that still loses is 3/8, so A is ahead by at least 37.5 points in every group and loses anyway. There is no margin of per-group superiority large enough to make a pooled scalar safe to read.

The negative result is the one worth keeping

Re-run the same sweep with the two models forced onto the same slice mixture:

Zero is not "rare". With the weights shared, pooled accuracy is a weighted mean using the same weights for both models, and monotonicity does the rest. Matching only the proportions is already enough. Matching only the sample size is not.

Every zero is printed next to the number of comparisons behind it, so an impossibility is distinguishable from an empty search — and the degenerate control (one group, where a reversal would be a contradiction) duly reports 0 over 1,936 comparisons.

What it does not say

A third slice does not make it cheaper: each extra group adds two more cells that must each be strictly won, and the smallest three-group reversal costs 13 observations against nine. The sweep covers two groups at per-cell n ≤ 8, so the minimum is exact for two groups and a claim about nothing else. And the page never says which number is right — macro and micro answer different questions, and choosing needs a deployment mixture a benchmark does not carry.

The narrow claim is the defensible one: a pooled scalar published without the slice sizes that produced it is not a ranking anybody can check.

27 in-page checks, 134 verifier asserts, 0 failures.