Judge Agent Patches on Tests They Never Saw
python
dev.to
A green run on tests an agent was allowed to read is not a merge signal. It is a training-set score. Split the suite first. Judge the patch on a held-out oracle, on fixtures the agent did not generate, and on a flake ledger it cannot edit. Agent patches overfit in a specific way. They do not only chase coverage. They also rewrite the exam. A copied constant, a fixture dumped from the new code, and a widened skip all produce the same CI color. The color is not the defect. Shared information is.