A large evaluation of coding agents produced an awkward result. Naming a testing technique did not reliably make the agents use that technique well. In the Zstd implementation study described by Dan Luu, agents were given instructions for TDD, QuickCheck, property-based testing, fuzzing, differential testing, mutation testing, formal methods, and other tools. The default condition, with no extra testing instruction, performed above average. Several named techniques performed worse.
The point is not that testing is useless. The point is that a test library name is not a testing method. Agents can produce tests that compile, run, and miss the bug that matters.
The uncomfortable result is not a missing test command
Coding agents already know that a finished implementation should have tests. They can create a test module, run the project command, repair a failing assertion, and report a green result. That workflow looks like verification from a distance. It often checks only whether the agent's implementation agrees with the agent's own assumptions.
Luu's evaluation makes that gap visible. Agents did not simply ignore every instruction. They often used the requested library or wrote something that resembled the requested technique. The problem was the missing judgment underneath it. A property-based test with a weak generator is still a weak test. A formal proof about an irrelevant invariant does not protect the code path where the defect lives. A differential test that implements the same mistake twice creates agreement, not an oracle.
That distinction matters more as agents take on larger changes. A human reviewer can ask whether a test attacks the risky assumption. An agent tends to optimize for a local signal: a command finished successfully, the assertion passed, or the requested tool appeared somewhere in the diff. The harness must make the stronger question visible.
What the evaluation actually measured
The primary evaluation reused a Rust implementation task for Zstd. It compared 26 prompt conditions, including Default, TDD, QuickCheck, property-based testing, fuzzing, differential testing, mutation testing, Lean 4, Verus, Alloy, and other formal or testing tools. Four additional skills were tested. Correctness came from hidden tests rather than from the tests the agent wrote for itself.
The main graph averaged 80 runs for each condition and effort level and compared medium with higher-effort runs. The article also discusses an IMAP RFC evaluation. The exact ranking should not be treated as a universal league table. The task, model, harness, prompt wording, and available dependencies all affect the result. The author repeatedly warns that the observed differences are noisy and that agents often failed to apply a technique in a meaningful way.
The broad pattern is more useful than the ordering. Default performed above average. At higher effort, fuzzing and property-based conditions did somewhat better than formal methods on average, while the medium-effort picture was mixed. The testing skills recommended by the model underperformed, while a small custom skill did better because it pushed agents toward examining likely mistakes instead of presenting a long tutorial.
This is a study of minimal instructions, not a study of what an expert can achieve with Verus, QuickCheck, or TDD. It asks what happens when an agent receives a technique label and limited guidance. That is close to how many teams deploy skills today, which is why the limitation is also the practical lesson.
Why naming a technique is not using it
QuickCheck was a clear example. The agents generally wrote simple smoke tests, used random inputs that often followed rejection paths, and checked few meaningful properties. In 63 of 160 QuickCheck runs discussed in the study, only one property was checked. The library was present, but the reasoning that makes property-based testing useful was absent.
The formal tools showed a similar split. Verus agents often proved abstract arithmetic facts instead of properties tied to the implementation. Some proofs were effectively of the form A => A: valid, but not useful for finding a Zstd defect. Lean, Alloy, and related conditions also relied heavily on ordinary Rust tests while adding proofs or models that did not reach the risky behavior.
TDD changed the workflow more visibly. Agents wrote about twice as many tests and created failing tests earlier, but the condition still underperformed. More test-first activity did not guarantee better cases. In the four-stream jump-table feature, agents often made all four streams identical. That is a legal fixture for a simple path, but it cannot reveal a bug that appears only when the streams differ.
None of this is an indictment of the tools. It is an indictment of the assumption that a tool name contains its own operating knowledge. An expert knows how to choose generators, construct an oracle, shrink a failure, inspect a counterexample, and decide whether an invariant touches the risk. A prompt that says “use QuickCheck” supplies none of that.
The difference between test presence and test power
Test presence is easy to count. Test power is harder to observe. A suite can contain hundreds of assertions and still leave the important behavior unconstrained.
The Zstd examples show why. Agents noticed that bitstream reversal could be risky, then used palindromic inputs for a relevant test. Reversing a palindrome produces the same sequence, so an encode/decode implementation with the wrong order can pass. The test is related to the feature but powerless against the defect.
The same problem appeared in tests for four Huffman streams. Making every stream identical exercises the shape of the input without exercising the relationship between distinct streams. A fixture that looks realistic can still erase the dimension where the bug occurs.
Differential testing adds another warning. It works when independent implementations receive the same inputs and disagree. In the study, most agents that attempted it did not build two genuinely independent implementations. They wrote the same idea twice, allowing the same wrong assumption to appear in both versions. Agreement then became a false signal of correctness.
A passing test proves only that one execution satisfied one assertion. It does not prove that the assertion represents the specification, that the fixture reaches the risky state, or that the oracle is independent from the code under test. Those are separate questions, and an agent needs a separate artifact for each one.
A verification loop an agent can follow
A useful verification loop starts with a claim instead of a command. The agent states what must remain true, identifies the failure mode that would violate it, chooses an oracle, and records the evidence. A small artifact can make that sequence explicit:
verification:
claim: decode(encode(data)) equals data for valid inputs
risk: bitstream order can be reversed in one direction
targeted_cases:
- empty input
- one-symbol input
- non-palindromic multi-block input
adversarial_case: distinct values in every stream
independent_oracle: reference decoder or specification
required_evidence:
- test result
- mutation result
- failure log when a case is rejected
The claim gives the agent a target that can be reviewed. The targeted cases connect the claim to boundaries and ordinary behavior. The adversarial case tries to falsify the agent's interpretation. The independent oracle prevents the implementation from grading its own homework. The evidence record makes it possible to inspect what the agent actually checked.
The adversarial case should be shaped around the suspected failure, not chosen because it is convenient to serialize. For a stream-order bug, a non-palindromic input is more informative than another randomly generated input that happens to collapse to the same path. For a parser, malformed delimiters and ambiguous boundaries may matter more than another valid example.
A small Rust test can express the difference without pretending to be a complete proof:
#[test]
fn distinct_streams_exercise_ordering() {
let input = StreamSet::new([
vec![1, 2, 3],
vec![4, 7],
vec![8, 13, 21, 34],
vec![55],
]);
let encoded = encode(&input);
let decoded = reference_decode(&encoded);
assert_eq!(decoded, input);
}
This test is stronger than a palindrome because each stream carries different structure. It still does not prove that the encoder is correct for every valid input. It only supplies one independent, reviewable check aimed at a particular failure mode.
The loop should also have a stop condition. If a mutation survives, the agent must either add a case that kills it or explain why the mutation is outside the specification. A green command with surviving mutations is not a green verification result.
Turn verification into an observable interface
A prompt or skill can nudge an agent, but the environment decides what the agent can learn from failure. The evaluation found that agents could name risky areas such as FSE, Huffman coding, and bit readers without testing them effectively. That is a feedback problem as much as a reasoning problem.
A harness should separate implementation context from verification context where practical. It should provide bounded tools, expose mutation results, preserve failing inputs, and make the independent oracle easy to call. It should also stop runaway work and attach each artifact to a change identifier. The agent then receives information that a longer instruction cannot provide.
Here is a compact interface sketch:
class VerificationHarness:
def __init__(self, oracle, mutation_tool, fuzzer, threshold):
self.oracle = oracle
self.mutation_tool = mutation_tool
self.fuzzer = fuzzer
self.threshold = threshold
def verify(self, implementation, specification):
claim = propose_claim(specification)
tests = generate_targeted_tests(implementation, claim)
if not oracle_checks(self.oracle, tests, specification):
return Reject("tests do not check the specification")
mutation_score = self.mutation_tool.run(implementation, tests)
if mutation_score < self.threshold:
return Reject("important mutations survived")
fuzz_result = self.fuzzer.run(implementation, timeout=60)
if fuzz_result.found_failure:
return Reject(f"fuzzer found: {fuzz_result.case}")
return Accept(claim=claim, tests=tests, mutation_score=mutation_score)
The mutation score is not a correctness certificate. It is a pressure signal: tests that merely mirror the implementation should fail to kill useful mutations. Fuzzing is not a certificate either. It adds cases and failure modes that the agent did not choose by hand. The oracle check asks a different question again.
Technique selection should follow the risk and the available oracle. Differential testing fits a protocol with a trusted reference implementation. Property-based testing fits a domain with a clear invariant and a useful generator. Fuzzing fits parsers and state machines with robust crash or semantic oracles. Mutation testing evaluates the tests themselves. Formal methods fit a specification that is precise enough to prove. TDD can structure feedback, but it does not replace hard-case design.
What teams should change first
Start with one high-risk path, not an attempt to impose every testing technique on every task. Write down the independent oracle before asking an agent to implement the path. If the oracle cannot be described, the team is not ready to treat a green agent-generated suite as strong evidence.
Require an adversarial case tied to the likely defect. Do not accept “edge cases included” as a summary. Name the input dimension, state transition, permission boundary, or protocol ambiguity that the case exercises. A reviewer should be able to see why the case could distinguish a correct implementation from a plausible wrong one.
Record verification artifacts with the change. Keep the claim, test rationale, adversarial input, oracle source, mutation result, and preserved failures. This turns review from “the agent says tests pass” into an inspection of the evidence. It also gives a later agent something concrete to revisit when a test fails in production.
Measure escaped defects and holdout failures instead of generated test counts. The study's TDD condition wrote more tests without improving correctness. Counting files or assertions rewards activity, not constraint. Run holdout cases that the agent did not see, mutate the implementation, and compare results against a reference where one exists.
The practical fix is not a longer prompt. It is a verification system that makes weak tests fail visibly, gives the agent an oracle it cannot rewrite, and leaves a reviewer enough evidence to judge the gap between tests that exist and tests that matter.
Originally published on Dispatch.