When the Tests Pass but the Spec Is Still Wrong

A study of Verus/Rust verification tasks found that models surfaced more evidence of incomplete specifications when they searched without corrective feedback than when they used a feedback loop: a 59.0% certified-witness yield versus 27.1% ¹. The implication for engineering teams is uncomfortable. A system that helps code satisfy a written contract can, at the same time, make it harder to notice that the contract never captured the behavior you actually wanted.

That distinction holds whether a team relies on formal verification, AI-assisted review, or ordinary unit tests. A passing check answers one bounded question: does this implementation meet the requirements as written? It says nothing about whether those requirements cover everything the product is supposed to do.

Passing the check isn't the same as being right

A verifier can confirm that an implementation satisfies a written contract. On its own, that result can't confirm the contract captures everything it should. The paper focuses on this gap: programs that satisfy a written contract while violating intended behavior ¹.

Here's a concrete, illustrative version of the problem (not a case from the paper):

fn save_payment_method(method: PaymentMethod) -> Result<()>
  requires: method.is_valid()
  ensures:  stored_method() == method

Nothing in that contract says what should happen if a payment method already exists. An implementation that silently overwrites it passes every check above, and still does something the product team never intended.

The review question that matters is: what behavior would still be wrong if every current check passed? That question hunts for omitted requirements, not implementation bugs. It also forces a useful separation: an implementation can satisfy a spec while the spec itself stays incomplete.

Why correction can erase evidence of an incomplete specification

The paper argues that correctness feedback can suppress evidence of incomplete specifications, meaning programs that meet a written contract but violate intended behavior ¹. Its evaluation compares a corrective-feedback loop against discovery that runs without that feedback ¹.

The two tasks have different objectives. Correction asks whether a candidate violates the current contract. A completeness audit asks whether the contract leaves out behavior stakeholders meant to require or forbid. They can use the same verifier and still not be the same job.

Analysis: when a system is repeatedly told to fix contract violations, it has every incentive to narrow its search toward whatever makes the stated checks pass. That narrowing leaves less room to explore the behaviors the contract never constrained in the first place. The paper's result supports investigating that mechanism in the workflow it studied; it doesn't show that all correctness feedback is harmful everywhere.

The authors also ran a format-matched iterative control. They report that feedback content accounts for a larger share of the yield gap than independent restarts do ¹. This points to feedback content as a larger contributor in this evaluation, while not showing that restarts had no effect. It doesn't establish that feedback suppresses specification evidence in every tool or development process.

What the Verus/Rust verification evaluation showed

The evaluation covered 12 Verus/Rust tasks across four model families, with matched call budgets ¹. Independent discovery without feedback produced a 59.0% certified-witness yield, against 27.1% for the corrective-feedback loop, IntentShield ¹. A certified witness is the paper's unit of evidence for a specification failure; the available summary doesn't give its full operational definition or the uncertainty bounds behind these percentages.

The advantage held across every model family and survived parser repair ¹. This is evidence that the result was not confined to one model family and that it persisted after the parser-repair check used in the study. It says nothing about how representative these twelve tasks are of production codebases.

A separate, phase-separated execution matched the aggregate yield without reusing any responses, and most certified witnesses also failed held-out cases ¹. That's a reason to test discovery as its own phase, distinct from correction. It is not a production defect rate, and it doesn't guarantee the same split will reproduce on your code.

What a phase-separated specification completeness audit looks like

Phase separation means finding evidence that a contract may be incomplete before using feedback to fix the contract or the implementation. In the paper's evaluation, a discovery run executed separately from correction matched the aggregate yield without reusing responses ¹.

A practical version of this for a team:

1. Search for permitted-but-unintended behavior. Ask for behaviors the current contract allows that might conflict with product intent, before giving the system any corrective feedback.
2. Preserve and assess each candidate. Check whether a witness is accepted under the current contract, and whether it reflects a real requirement rather than a misreading.
3. Revise the contract. Make the missing requirement explicit before using the revised contract to judge the implementation.
4. Verify against the revised contract, while staying open to the possibility that it still needs more work.

This sequence is a recommended practice, not a procedure whose every prompt and threshold is specified in the paper ¹. The checkpoint that matters is simple: preserve surprising behavior for human review before a repair loop touches the candidate or its explanation.

For each finding, a short review record helps: the intended behavior, the current contract, the witness behavior, evidence the current verifier accepts it, the proposed contract change, and the tests that should fail before the change and pass after. That artifact is what separates a genuine omission from an invalid counterexample or a misunderstanding.

What happened when contracts got repaired

Phase-Revised contracts reached 81.3% implementation success, versus 62.5% for IS-Revised contracts; Direct-Revised came close behind ¹. Both no-feedback witness sources substantially outperformed deliberately weakened contracts ¹.

These are task-level results, not a general benchmark for developer productivity or software reliability. They suggest that where a spec's gaps are first surfaced, and what evidence gets preserved along the way, affects the quality of the resulting contract ¹.

But implementation success isn't proof of completeness. A contract can be easy to implement and still miss an important requirement. The distinction from the top of this piece holds: implementation checks assess whether stated requirements are satisfied; a completeness audit asks whether the stated requirements are enough.

Where this breaks down

The reported effect did not reproduce in the available Dafny and Lean evaluations. The paper describes this as an empirical boundary around the Verus/Rust workflow it studied ¹. That limitation is worth sitting with, because it means these results shouldn't be read as evidence the effect generalizes across formal verification tools.

The available summary doesn't name the individual tasks or model families, give per-task breakdowns, report uncertainty estimates, or establish how representative the benchmark is of production work ¹. Treat those as open questions for anyone weighing how much to lean on this result, not gaps to paper over with assumptions.

Nothing here shows that every corrective-feedback system suppresses specification evidence, that phase separation improves production software outcomes generally, or that any particular review process will catch every missing requirement. What it supports is testing the failure mode inside your own workflow.

What to change in AI code review

Add a completeness checkpoint that is explicitly distinct from "does the implementation satisfy the contract?" Ask reviewers, or an AI system used as a discovery aid, to look for adversarial behaviors the current spec permits, including formally valid outcomes that still conflict with product intent. If you already scale review effort to risk, our guide to risk-based code review is a natural place to decide which changes earn this extra checkpoint.

Where practical, run discovery before exposing it to corrective feedback, and keep the raw witnesses. This follows the paper's phase-separated result, but validate it against your own tools and risk profile before trusting it ¹.

Four questions worth asking at the checkpoint:

• Which intended behaviors are actually written down, and which are only assumed?
• What evidence would reveal that a requirement is missing?
• Who decides whether a witness reflects a real requirement rather than noise?
• What happens when a witness reveals an ambiguous requirement rather than a clearly broken implementation?

What to measure

Pass rates alone won't catch this failure mode. Track witness yield, the share of witnesses that survive held-out checks, contract changes accepted by reviewers, and defects found post-release. The paper reports witness and implementation outcomes ¹; the rest of this list is a proposed set of operational metrics, not a reported finding. The same logic applies beyond code: launch-time tests can pass while behavior drifts, which is why agent monitoring needs drift checks, not just launch tests.

For high-risk systems, keep a human responsible for interpreting requirements. A verifier can expose behavior a contract permits. Whether that behavior violates product intent is a specification and governance call, not a verification result.

Questions to ask before adopting this

• Does our review process search for behavior the contract allows, or only for behavior that violates it?
• Can we preserve evidence found before corrective feedback changes the candidate?
• Do we distinguish "the implementation satisfies this contract" from "this contract captures the intended requirements"?
• Which languages, verification tools, model families, and task types have we actually evaluated ourselves? The reported boundary between Verus/Rust and the Dafny/Lean evaluations is a reason not to assume this transfers across tools ¹.
• When a witness exposes ambiguity rather than a clear bug, who owns the decision about what the product should do?

The useful change isn't to stop giving systems correctness feedback. It's to stop treating correction as a substitute for a completeness review. The Verus/Rust results make the case for testing that separation in your own workflow ¹. They don't make the decision for you.