Builder Journal · ARC Prize 2026
Every boxing movie has the sparring montage. The hero looks untouchable, gloves snapping, trainer grinning in the corner. Then the bell rings on the real fight and none of it holds, because sparring was never the fight. This month I built an AI agent that was crushing it in sparring. The real fight went differently, and the gap between the two finally taught me how this competition actually works.
This is a thread in my builder's log from inside the ARC Prize 2026, where my agent has to win small games it has never seen before, with no instructions, and the games it actually gets graded on are hidden from me. An earlier entry laid out what the competition is and why it might be the most important benchmark in AI. This one is the day I learned to stop trusting my own practice scores.
The gym every competitor has to build
To improve the agent I had to measure it, and I cannot measure it on the real test, because the real games are secret. There is no way around that wall. So, like everyone in this competition, I built a practice gym at home: the handful of example games the organizers do hand out, a scoring script, and a way to run two versions of the agent head to head and see which did better.
That gym is supposed to be the edge. It is where you try ten ideas cheaply, keep the one that wins, and only then spend one of your scarce real submissions. The leaderboard gives you only a trickle of attempts, so the whole game is supposed to be won at home, on the bench, long before you ever submit. Get the gym right and you compound. Get it wrong and you are sprinting on a treadmill.
The number that looked like a win
One change tested beautifully. On my home bench it gained about five hundredths of a point, and in a competition scored from zero to one, where the leaders are separated by thin margins, that is a real, ship-it-with-confidence number. Every signal pointed the same way. So I sent it.
It came back worse. Not by a hair. The change that gained roughly 0.05 at home cost me around 0.14 on the real, hidden board. Nearly three times the size of the supposed win, pointing the opposite direction. My practice gym had not merely failed to predict the game. It had confidently marched me the wrong way down the road and told me I was winning the whole time.
The part that actually scared me
A bad submission is survivable. You lose a slot, you learn something, you move on. What unsettled me was what happened when I tried to understand the loss. I had a known regression sitting right there on the official scoreboard. A fact. Undeniable, in writing, public. And my own tooling, run against every practice game I had, could not reproduce it. I went hunting for the failure on my bench and the bench kept calmly insisting nothing was wrong.
That is worse than a wrong number. That is an instrument that is structurally blind to the exact thing that decides your score. Digging further, I found that my home proxy had at one stage been overstating performance by something like twenty times. I had been steering by a gauge that was not just miscalibrated but pointed at the wrong quantity entirely, and I had trusted it because it was the only gauge I had.
What it actually means
This rewired how I work here, and the new rule is blunt: the leaderboard is the only oracle. It is the single place the real, hidden behavior ever shows its face, and it rations submissions to a precious few, so each one has to be a sharp question with a real answer attached.
My home gym does not get a vote on whether something works anymore. I demoted it. Its job now is smaller and more honest. It invents candidates worth trying. It guarantees, before I risk a submission, that a change physically cannot make things worse than the version already on the board. And after a score lands, it helps me explain why. What it is no longer allowed to do is pretend to be the truth, because the one time I let it, it lied to my face, twice, with total composure.
There is a cousin to this lesson over in my other competition's journal, where a test set I had carefully built told me a confident lie for almost the opposite reason. Same disease, different host. A measurement you cannot fully trust is more dangerous than no measurement at all. No measurement keeps you humble. A lying one makes you bold, right before it walks you off a cliff.
Where I'm standing right now
Still 0.25 on the board. That number has become the wall I keep failing to climb, because the clever upgrades keep losing to it. The floor is stubborn precisely because it is honest: it is the version of my agent that does the safe, simple thing and never overreaches, and so far nothing fancier has reliably beaten it where it counts. Climbing past it for real, on games I will never get to see, is the entire problem, and I now know for certain it cannot be faked from the bench.
Next entry: the change I was certain made my agent smarter, that made it measurably dumber, and the uncomfortable reason why.
More in this series
- ← Start here: I'm trying to teach a machine to play a game it has never seen, what ARC-AGI-3 is, and why it's AI's uncomfortable mirror.
- The cousin lesson from my other competition: My own test set lied to me, same disease, different host.
- The full ARC Prize 2026 thread, every entry from this competition.
- The Builder Journal, the live log across all my competitions.
This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.