False Completion Is the Real Failure Mode of Coding Agents

A deliberately strict process for preventing false completion in autonomous software development

In an earlier article, I argued that the model should have to earn every agent in an agentic system. One agent is a valid answer. Every additional boundary has to justify its cost.

This article starts one level lower. Assume we have selected the topology. Who gets to decide that the resulting product is actually the product we asked for?

After experiencing dozens of AI-development efforts and reviewing the results of many more, I no longer think the main danger is that the development procedure will manifestly fail. A crashed build, a red test or an explicit error report is relatively easy to deal with. The system has admitted that it has a problem.

The far more serious failure is that of FALSE COMPLETION: the AI system believes it has accomplished the development goal, reports success with complete confidence, and delivers a product that has little to do with what the system design had in mind.

The code may compile and the interface may look polished. Yet core flows are missing, persistence is half-wired, or the implementation has quietly solved a simpler problem than the one it was given.

And this is not merely anecdotal. An analysis of over 20,000 coding-agent sessions identified inaccurate self-reporting and misreading developer intent as recurring forms of developer-agent misalignment. Most visible resolutions still required explicit human correction. The agent’s confidence was not evidence that the job was done. The study is worth reading.

My response is a deliberately strict development process. It costs more tokens and wall-clock time than letting one agent run until it declares victory. I deliberately pay that cost. For substantial autonomous builds, this is now the only way I work. A failed process is visible. A successfully completed wrong product may survive until a user discovers it.

1. Make the product book executable

Before any code is written, the product book, call it a PRD, specification or product contract, must be reviewed by an AI agent whose task is not to improve the prose. Its task is to attack the design.

Can every important requirement be converted into observable behavior? Are the user flows, data rules, failure behavior and non-functional requirements clear? Is anything important hiding behind words such as “fast,” “intuitive” or “secure” without a measurable meaning?

The objective is not a specification that deterministically produces one implementation. Many implementations may be valid. The objective is a specification that lets us determine, with as little interpretation as possible, whether an implementation is acceptable.

The product book must also define kill conditions. If an acceptance condition cannot be tested or two requirements contradict each other, the correct output is not heroic improvisation. It is STOP.

2. Lock your acceptance tests

The acceptance tests derived from the product book must be fixed before implementation and remain immutable throughout the development run.

The coding process may run them and may fail because of them. It may not change, weaken, delete or reinterpret them.

This is intentionally more rigid than ordinary iterative development. If we later discover that a test cannot be met or that the test itself is wrong, the process stops. The product book and acceptance set are updated through a separate decision, their version changes, and execution restarts from the specification and planning boundary. The finish line is not quietly moved by the participant currently failing to reach it.

Locked does not mean sacred. Tests can be wrong. OpenAI found difficult SWE-bench Verified tasks where tests rejected valid solutions or required behavior absent from the problem statement. That is why changing a test should be an explicit design event, not an opportunistic edit inside the coding loop. OpenAI’s audit shows how dangerous a bad oracle can be.

3. Break into steps before any code is written

The complete task must be broken into discrete execution steps before code is written. Each needs defined inputs, outputs, dependencies and review conditions, and must fit comfortably inside the model’s usable context, not merely its advertised context window.

This may be the single most important protection against an AI-development effort going sideways. Long contexts accumulate abandoned hypotheses, obsolete instructions, failed repairs and irrelevant tool output. Eventually the agent spends its reasoning budget navigating its own history.

Small should not mean arbitrary. Splitting one tightly coupled behavior across five agents simply moves the difficulty into integration. A good boundary ends in a coherent change another agent can verify without reconstructing the entire project.

4. Assign each step a new coder instance

Using one coding agent and looping it through every step looks compelling. The developer of step five already knows what happened during steps one through four, and fewer tokens are spent bringing it up to speed.

But the same accumulated context also contains every wrong assumption, abandoned path and local compromise made along the way. Anyone who has executed long-running agent tasks has seen performance deteriorate as the context shifts from one problem to another.

For non-trivial development, I want a fresh coding context for every step. Continuity should come from explicit artifacts: the product contract, architecture decisions, interfaces, git history, accepted outputs, open risks and a short task packet. It should not depend on a long conversation correctly remembering which parts are still true.

Anthropic’s recent experiments suggest that stronger models can sometimes sustain longer builds, allowing sprint decomposition to be removed and evaluation to happen less frequently. I understand the argument. My own experience leads me to a more conservative default: I would rather pay the visible cost of fresh contexts than accept the invisible risk of context corruption and false completion. Their current reasoning and trade-offs are described here.

5. The builder never grades his own work

The coder responsible for a step must never be the authority that approves it.

When the work is ready, the coder hands the diff, task contract and evidence to a fresh reviewer. The reviewer should not inherit the coder’s reasoning. It should inspect the result on its own terms, preferably with read-only access and the ability to run the system.

Where practical, I prefer a reviewer from a different model family. A different model is not automatically better, but it is less likely to reproduce exactly the same assumptions and blind spots. Deterministic tests remain a stronger oracle than either model; the second model exists to challenge what the tests do not express well.

The reviewer compares the change with the product intent, architecture, regressions and end-to-end behavior. It reports material gaps, not style objections manufactured merely to demonstrate activity.

Here current guidance and my experience agree: Anthropic recommends adversarial review in a fresh context so the evaluator sees the result rather than the story that produced it. See its current Claude Code guidance.

6. Final acceptance tests must be invisible to coder

This is probably my most controversial rule: the coder should not have read access to the final acceptance-test implementation.

It must know the required behavior. A hidden test may never demand a private method name, particular internal structure or functionality absent from the product contract. But the exact executable suite should belong to the control plane, not to the participant being evaluated.

The coder may write unit tests and receive visible examples or contract tests. What it cannot do is inspect the final oracle and shape the implementation around its precise assertions.

Agents are no better than us in this respect. Give the builder direct sight of the scoring mechanism and the target can quietly shift from “build the intended product” to “make these checks turn green.” The protected suite should therefore be largely black-box. Failures should identify the violated requirement without exposing or surrendering control of the examination.

7. Place hard cap on coder/reviewer cycles

It is legitimate for a skeptical reviewer, especially one using a different model, to reject a step and request repairs. That is the process doing its job.

But the loop must be strongly bounded. My red line is around four coder/reviewer attempts, and I stop earlier if the same defect returns twice. By then, the problem is probably not a missing if statement. Requirements may conflict. The step may be too large. The technology may be at its limit. The acceptance test may be wrong. Or the accumulated repair dialogue may already be corrupting both agents' judgment.

What I do not want is endless negotiation until the reviewer becomes tired, confused or sufficiently conditioned to approve bad code.

When the budget is exhausted, the reviewer produces a failure report: what remains wrong, the evidence, what was attempted, which assumptions are doubtful, and whether the likely fault is in implementation, decomposition, architecture, specification or the test oracle. Control returns to planning.

Failure is an acceptable result. Counterfeit success is not.

“Done” must be a system state

These rules lead to one architectural principle: the agent does not get to decide it is done.

Every feature begins in a failing state. A coder may propose completion, but only protected tests and an independent evaluator can accept it. Every step leaves code, evidence and a clean checkpoint. Integration checks confirm that individually valid steps still form one working product.

This is not the cheapest possible harness, and it is not my recommendation for changing a label or adding one obvious validation. It is my default for substantial autonomous product development, where a plausible but wrong result can consume days of review or reach production.

The current fashion is to assume that better models will let us remove more of this structure. They probably will remove some of it. Anthropic explicitly recommends retesting which pieces remain load-bearing as models improve. That is sensible engineering.

My own bias is to remove safeguards only after repeated evidence that they no longer catch meaningful failures. Model capability is improving quickly. The ability of a confident agent to make a wrong product look finished has not disappeared.

So let the model write the code. Let it plan, search, test and repair. Give it real autonomy inside clear boundaries.

But do not let it define the finish line, rewrite the examination, grade its own paper and then announce that it passed.

“Done” is not an agent opinion. It is a state the system must prove.

I am a principal software and protocol engineer. I have written production Solidity since 2017, led protocol-layer engineering on a live EVM L1, and built an AI-assisted audit workflow that I validated against a 219-contract corpus. It is now open source: github.com/giladHaimov/Smart-Contract-AI-Audit-Skill