Building a Verification-First Repair Harness

A method for putting a language model inside a maintenance pipeline without letting it decide anything.

Abstract

A large class of engineering maintenance work has the same shape. A machine-readable artifact (a selector, a query, a mapping, a parser rule, a configuration) silently stops producing correct output because the world it points at changed. The artifact is small, the failures are many, and a competent engineer can fix any single instance in minutes given the evidence. The cost is not difficulty; it is volume and triage.

This is an attractive target for a language model and a dangerous one. The model can propose a repair from evidence, but a wrong repair is worse than no repair: it produces plausible output that no alarm fires on. The naive pipeline, which collects failures, shows them to a model and applies its answers, fails in three independent ways at once, and each failure is invisible in aggregate metrics.

This document describes a harness that makes the approach work, derived from a production deployment repairing extraction rules across roughly 1,400 targets and 1.6 million monitored items. It is written to be domain-independent. The running example is web extraction, but the method has been checked against configuration migration and flaky-test repair, and the invariants are stated so they can be tested in any domain.

The central claim is measurable and, in our deployment, measured. The quality of such a system is governed by the oracle and the evidence given to the generator, not by the capability of the generator. Doubling model size produced no usable improvement. Constraining the prompt and narrowing the evidence roughly doubled precision.

Contents

1. When this method applies
2. Terminology
3. Five invariants
4. Reference architecture
5. Stage A. Attribution: is this even our problem?
6. Stage B. Compression: how many distinct problems?
7. Stage C. Generation: the one place the model sits
8. Stage D. The oracle
9. Stage E. The human gate
10. Designing the outcome taxonomy
11. Evaluation protocol
12. Diagnosing failure: model or evidence?
13. Cost model
14. Anti-patterns
15. Design checklist
16. Instantiating the method in a new domain

1. When this method applies

The method fits a problem when six conditions hold together. If any of them fails, a different design is appropriate, and section 16 says which.

The first condition is that the artifact is small and structured: a selector, a path expression, a regular expression, a field mapping, a configuration key. It must be small enough to generate in full and, more importantly, small enough to verify exhaustively rather than sample.

The second is that correctness is machine-checkable against evidence you already hold. This is the load-bearing condition. If you cannot decide, mechanically, whether a proposed repair is correct using evidence already in your possession, you do not have this problem; you have a research problem. Do not build a harness. Build the oracle first, and then reconsider whether the harness is still needed. In our deployment the oracle existed before the model did and could be executed against historical evidence, which is what made everything downstream tractable.

The third is that failures are numerous and repetitive. Below some volume a human simply fixes them faster than you can build the machinery, and the machinery will be obsolete before it pays for itself.

The fourth is that a wrong repair is silent: it produces output, just wrong output. This is what separates the problem from code generation. A generated function that is wrong usually throws, fails a test, or refuses to compile. A wrong extraction rule returns a number. A wrong mapping returns a record. The pipeline stays green and the data quietly rots. Every design decision in this document follows from taking that seriously.

The fifth is that the evidence for a repair is local, meaning one or two examples suffice. If a repair requires reasoning across the whole system, the generator has the wrong shape and no amount of harness will fix it.

The sixth is that a human can adjudicate a proposal in under a minute given the right display. The gate is deliberate, but it must not become the bottleneck, and whether it does is a property of your interface rather than of your reviewers.

2. Terminology

The vocabulary below is used consistently throughout, and it is worth fixing before the design discussion begins, because most disagreements about systems like this turn out to be disagreements about which component is being described.

An artifact is the small structured object being repaired. A target is the entity that artifact belongs to: a monitored site, a source table, a service. An instance is one failing case, meaning an artifact together with the evidence of its failure, and evidence is the stored input on which the artifact produced wrong output or none.

The generator is the model, in its single narrow role: evidence in, candidate artifact out. The oracle is deterministic code that decides whether a candidate is acceptable, and a verdict is the oracle's typed judgement on one candidate. A gate is a particular kind of oracle check, one that can only demote a verdict and never promote it. A trial is empirical evaluation of a candidate against a held-out population of real instances. The adjudicator is the human who approves or rejects, and is the only writer to production. The harness is everything except the generator.

The asymmetry in that list is deliberate and worth stating explicitly. The generator has one job and no authority. Everything that decides anything is either deterministic code or a person.

3. Five invariants

State these in your design document and test them. They are not style preferences. Each one was learned by violating it.

I1. Deterministic before generative

Every filtering, grouping, classification and verification step is ordinary code. The generator is invoked only on instances that deterministic code could not dispose of, and only for the one judgement that genuinely requires reading unstructured evidence.

The consequence is that the generator sees a tiny fraction of the input, in our deployment roughly one instance in three thousand. This is what makes the economics work, and it is also what makes the results interpretable: when quality moves, you know which stage moved it.

I2. The oracle runs the production code path

Verify a candidate by executing the same code that will consume it in production, with the same parser, the same normalisation and the same coercion. Never a reimplementation, never an approximation, and never "the model says it matches".

The consequence is that a candidate which passes verification passes because production would pass. A reimplemented oracle drifts from production silently, and its drift is indistinguishable from a model failure, which means every subsequent measurement is contaminated. Where a lightweight check must exist elsewhere, for example inside the generation loop for speed, measure the divergence between it and the real oracle explicitly. We measured 3.2% of candidates decided by parser differences alone: small enough to tolerate, and large enough that assuming zero would have been wrong.

I3. Every claim is downgraded to a measurement

Do not let a qualitative judgement survive into the pipeline. "The candidate looks right" becomes a verdict from the oracle. The verdict becomes a hit rate over a held-out population. The hit rate becomes a comparison against what production currently achieves on the same evidence.

In our deployment, three candidates carrying an identical verdict turned out under trial to be a genuine repair, pure churn with behaviour identical to the incumbent, and an unmeasurable case with no test population at all. Three correct decisions, three different actions, one verdict. Without the trial layer all three would have been treated alike, and two of those treatments would have been wrong.

I4. Precision over recall, deliberately

In a human-gated system the scarcest resource is the adjudicator's trust. A queue at high precision gets reviewed. A queue at low precision gets ignored, and once ignored it is dead regardless of what it contains.

The consequence is that most tuning should make the generator answer less often. In our sequence of prompt changes the count of genuinely good candidates barely moved, staying between eight and eleven across every variant, while precision rose from 40% to 91%. The entire gain was the model learning to decline. Design the generator's contract so that "no answer" is an explicitly correct response, and say so in the prompt rather than hoping it is inferred.

I5. Decisions are durable

An approval is never silently overwritten by a later run. A rejection is never resurrected: the same candidate must not reappear as though it were new, and a genuinely different candidate for the same slot should arrive annotated with what was already rejected there.

The consequence is that the adjudicator's effort accumulates instead of resetting each cycle. Without this property, a periodic pipeline re-proposes rejected candidates forever and trains its reviewers to stop reading.

4. Reference architecture

flowchart TD
    IN["Failing instances"] --> A
    A["A. ATTRIBUTION<br/>metadata only, no payload decoded"] --> Aq{"Can repairing the<br/>artifact fix this?"}
    Aq -->|no| ROUTE["Routed to the owning<br/>discipline, with the reason"]
    Aq -->|yes| B
    B["B. COMPRESSION<br/>group by structural equivalence"] --> Bq{"Is this evidence<br/>a valid basis<br/>for a repair?"}
    Bq -->|no| EXCL["Excluded, reason recorded"]
    Bq -->|yes| C
    C["C. GENERATION<br/>focused evidence to candidate"] --> CC["critique loop<br/>(mechanical, one round)"]
    CC --> D
    D["D. ORACLE<br/>production code path + gates"] --> Dq{"verdict"}
    Dq -->|not acceptable| FILE["Filed with a typed verdict"]
    Dq -->|acceptable| Q["Review queue"]
    Q --> E["E. ADJUDICATION<br/>evidence displayed, trial on demand"]
    E -->|approve| PROD["Production artifact"]
    E -->|reject| MEM["Durable rejection"]

Figure 1. The five stages, and the single path into production.

Two properties of this structure matter more than the individual stages. The first is that the funnel is monotone in cost: each stage costs more per instance than the one before it, so each stage must remove work rather than add it. If a stage does not reduce the population by roughly an order of magnitude, it is not earning its place and should be merged into its neighbour. The second is that exactly one arrow reaches production, and a person is standing on it. Everything else writes to a queue, a file, or a log.

5. Stage A. Attribution: is this even our problem?

The purpose of this stage is to partition failures by which discipline can fix them, using metadata alone, with no payload decoded and no model invoked. It is the highest-value stage relative to its cost, and it is almost always skipped by teams who begin from "we have a lot of failures, let us use a language model".

Four rules govern its design.

Buckets must be mutually exclusive and exhaustive. Compute them with a single first-match-wins rule chain rather than with independent conditions. Independent conditions allow one failure to count in two buckets and allow some failures to count in none, and both errors make every downstream number wrong. Assert that the buckets sum to the total, and fail loudly when they do not.

Order the chain by ownership rather than by frequency. The question the chain answers is "who fixes this?", so the most specific owner should be tested first.

Each bucket names a team, not a symptom. "Blocked" is not a bucket. "Access control, owned by the anti-abuse team" is a bucket, because it tells you where the work goes.

A bucket with no owner is a bug in the taxonomy. Add an explicit catch-all and watch it. Ours surfaced several thousand failures that had previously been invisible because they matched none of the named conditions.

The measurement that justifies the stage: across one estate, 78% of failures attributed to something other than the artifact, namely access blocks, deleted entities and infrastructure faults. In a single run, 29,852 failures traced to one unreachable gateway. Sending that population to a model would have produced confident and useless repairs for a problem no repair could touch.

One blind spot must be documented rather than solved here. A failure can look healthy at the metadata level and be semantically dead, for example a "this item no longer exists" page served with a success status and a full body. Metadata cannot see that. Stage B must.

6. Stage B. Compression: how many distinct problems?

The purpose of this stage is to collapse N failing instances into K distinct modes, where K is the number of genuinely different repairs required. Four rules govern it.

Group by structure, never by content. Two instances belong together when the shape of the evidence matches: the vocabulary of identifiers, the schema, the template. Content differs within one mode by definition, so keying on content makes every instance its own mode and the stage does nothing.

Hard-split before you cluster softly. Partition first on cheap categorical facts, such as whether the evidence carries structured data, whether it is a stub, and which size bucket it falls into. Apply similarity only within each partition. This prevents a similarity threshold from merging categories that are qualitatively different but superficially close.

Take one representative per mode, and carry a sibling. The representative is what the generator sees. The sibling, a second instance of the same mode, is what makes the resulting candidate testable for generality rather than memorisation. This rule is not optional: a candidate validated on exactly one example is indistinguishable from a candidate that encodes that example, and requiring it to hold on a second instance is the cheapest generality test available. It is the difference between "works here" and "works on this template".

Classify the representative's validity, and exclude with a recorded reason. Not every piece of evidence is a legitimate basis for repair. Evidence drawn from the wrong kind of entity produces an artifact that is correct for that evidence and wrong forever afterwards.

The measurement: 26,481 failing instances compressed to 114 representatives. Most targets had exactly one mode; a minority had two or three genuinely distinct template variants, and that minority is precisely the population a naive one-instance-per-target deduplication would have silently mis-repaired.

The trap this stage exists to catch is worth stating concretely. Before compression, our top-ranked target by failure count carried thousands of failures and admitted no repair at all: 99.9% of its evidence was the wrong kind of page and none of it contained the value. Ranking by raw failure count points at the loudest target. Ranking by attributable, valid failures points at the fixable one.

7. Stage C. Generation: the one place the model sits

The purpose of this stage is narrow by construction: given focused evidence and the current artifact, propose a candidate or decline.

7.1 Evidence selection is the highest-leverage variable

Do not hand the model the whole payload. Anchor on the strongest signal for the value you want, expand to a bounded context around that anchor, and attach only the structured metadata that could plausibly carry the answer. In our deployment the model reads roughly 5% of the payload.

This is not primarily a cost optimisation. The excluded region is where the wrong answers come from: the repeated blocks, the recommendation modules, the alternate renderings of the same value. Removing that region removes a class of error rather than trading it for another.

The corresponding hazard is real and must be handled explicitly, because a focused view that omits the answer is worse than an unfocused one. Fall back to the full payload whenever the anchor is not found with confidence, and measure how often the answer lies outside the focused region. In our case, for one field, that was roughly one instance in ten, which converts an unknown into a stated bound on that field's recall.

7.2 The contract

The prompt is a contract rather than a request, and it must state five things.

It must state the task as repair, not discovery. "This artifact stopped working, here is the evidence, what should it be" outperforms "find the value", because the framing keeps the model anchored to the same semantic slot the original artifact meant rather than to any plausible-looking value on the page.

It must state the current artifact and the observed failure, which are context the model cannot infer.

It must state hard constraints as prohibitions with reasons: uniqueness, generality across instances, and forbidden constructs such as positional indexes, instance identifiers and state-dependent conditions. Give the reason for each one. A constraint with a stated failure mode is followed more reliably than a bare rule.

It must state that declining is a correct answer, explicitly, with an example of when. This single clause carries a large share of the precision gain described in I4.

It must specify a machine-parseable output shape, with a place for the model to record what it saw and why. That rationale is not for the pipeline; it is for the adjudicator.

7.3 Determinism

Fix the sampling temperature at zero and pin the seed. Without this you cannot attribute a metric change to a design change, and every comparison is contaminated by sampling noise. Before pinning, we measured disagreement on 4 of 46 slots between two runs of an identical configuration, which is enough to move a headline number by two points and invite a wrong conclusion. Residual nondeterminism from batched inference may remain; measure it once, state it, and stop worrying about it.

7.4 The critique loop, exactly one round

After the first answer, run the cheap mechanical checks. If any fail, return them to the model together with the evidence of what its candidate actually did, not a bare "invalid" but a statement of the form "it matched these three elements, whose contents are X, Y and Z". Allow exactly one revision.

The reason for exactly one is that iterating against the same evidence teaches the model to satisfy that evidence, which is overfitting with extra steps. Empirically the second round mostly produces withdrawal to a null answer rather than a better candidate, and withdrawal is a win under I4, so the round pays for itself without needing to produce repairs at all.

7.5 A note on reasoning budgets

If your generator supports an explicit reasoning phase, be aware of a specific failure mode. When reasoning and answer draw from one output budget, the reasoning expands to consume the entire budget and the model returns nothing. We measured this at two different budget sizes: 19 of 23 instances returned empty, and increasing the budget made the problem worse rather than better.

The fix is a separate hard limit on the reasoning phase, leaving the answer budget intact. With a bounded reasoning phase, quality improved over no reasoning at all. The general lesson generalises past this one setting: verify that "more of a good thing" is monotone before assuming it, particularly where two behaviours draw from a shared pool.

8. Stage D. The oracle

The oracle is the component that makes the system trustworthy, and it deserves more design attention than the prompt.

8.1 Layers

Run the checks in increasing order of cost and stop at the first that rejects.

Begin with syntax, which asks whether the candidate parses or compiles at all, and costs microseconds. Then uniqueness, which asks whether it resolves to exactly one thing, at a cost of milliseconds. Then value, which asks whether the production consumer accepts what the candidate yields. Then generality, which asks whether it also holds on the sibling instance and on other evidence from the same target. Then form, which asks whether it violates a stated constraint such as an embedded instance identifier, a positional index or a state-dependent condition, and which is nearly free because it is a syntactic property of the candidate rather than a property of its execution.

Two further checks are more expensive and are described separately below, because they are the ones most systems omit: the regression gate, which asks whether the candidate still produces the right result on evidence the current artifact handles correctly, and the null-hypothesis gate, which asks whether the current artifact already works on the failing evidence.

8.2 Gates demote, never promote

A gate can only lower a verdict, never raise one. This asymmetry is what lets you add gates over time without re-validating everything that came before: a new gate can only make the queue more conservative, so its introduction can never invalidate a past approval.

8.3 The two gates people forget

The regression gate requires something most pipelines do not keep, namely evidence of success. Store, for each target, the most recent input on which the artifact worked, together with the value production extracted from it. Without that store, every candidate is evaluated only where the incumbent fails, which is the single place where the incumbent is guaranteed to lose, and the system will ship candidates that fix the broken 5% while breaking the working 95%.

The store is inexpensive, being bounded per target, compressed and time-limited, and it was the single highest-value addition we made after the first production run. It also supplies the population for the trial described in stage E.

The null-hypothesis gate asks whether the incumbent already works on the failing evidence. If it does, this artifact is not the cause of the failure, something else is, and the candidate should be demoted with a pointer to the real question. In our deployment this gate reclassified a large fraction of one run's queue, all of which would otherwise have been reviewed as though it were repair work.

8.4 Merging, not overwriting

If a target can have several modes, several candidates will arrive for the same slot. Do not key your queue by target alone. We did, and the consequence was that four candidates for one slot overwrote one another, the survivor was the candidate from the smallest mode, and three larger repairs vanished without a trace.

The correct behaviour is to verify everything first and then merge per slot. Identical candidates merge with their impact summed. Genuinely different candidates either compose, when the consumer supports ordered alternatives, in which case compose them and verify the composition as a unit, or are published as one winner with the alternatives attached and visible to the adjudicator.

A corollary applies as soon as more than one source feeds the queue, for example a failure-driven stream alongside a proactive audit. The queue key must include the source, or one stream's run will silently retire the other stream's work. We learned this in production, and the symptom was a queue that emptied.

9. Stage E. The human gate

The adjudicator is part of the system, and the display is part of the design. The card should show evidence rather than conclusions.

It should show the current artifact and the candidate side by side, with what each yields on the same evidence. It should show the counts behind the verdict: how many elements matched, how many sibling instances the candidate held on. It should show the impact, meaning how many instances the repair would affect, summed honestly across merged modes rather than reported from whichever mode happened to survive. It should show the reason string for any demotion, phrased so that a reviewer can argue with it. It should link to the raw evidence and state that evidence's classification. Finally, it should show the model's self-reported confidence in visually de-emphasised form, because it is the only unverified number present.

Make the candidate editable before approval. A large share of near-miss candidates need one qualifier added. An editable field converts them from rejections into repairs in seconds, and it is the cheapest recall you will ever buy.

Approval must be minimal and targeted. Write exactly one field, and write it to the location the consumer actually reads, which may depend on the target's configuration shape. We shipped a version in which approval wrote to a location the consumer never read for one whole class of targets: the review showed green and production behaviour did not change at all. Record, on the approval record itself, the exact path written.

9.1 The trial

The strongest instrument available to the adjudicator runs both the incumbent and the candidate over a sample of real instances drawn from both populations, meaning instances that currently fail and instances that currently succeed, and reports per-instance results.

Compute a verdict from that comparison and persist it, so that it survives the session. Five outcomes suffice: gain, where the candidate wins on failing evidence and loses nothing; no gain, where behaviour is identical to the incumbent and the change is therefore churn; loss; regression; and unverifiable, where no testable population exists.

The trial is the only stage that samples instances nobody selected, which is exactly why it catches what every earlier stage missed.

10. Designing the outcome taxonomy

Verdicts are the interface between the harness and the human, so design them as a typed enumeration with an action attached to each. Four rules apply.

Every verdict names an action. If two verdicts imply the same action, merge them. If one verdict implies "it depends", split it until it does not.

Distinguish "produces nothing" from "produces something different". These look alike in a diff and are opposites in meaning. The first is a break. The second is frequently the entire point of the repair, for example when the incumbent was reading an adjacent value and the candidate reads the intended one. Our regression gate emits both under a single verdict name and separates them in the reason string; in hindsight they should have been two verdicts.

Include a verdict for "not a repair at all". Some instances are telling you that the entity is dead, the reference is stale, or the access is broken. A taxonomy without that bucket forces them into a repair verdict, where they are reviewed forever and never resolved.

A verdict says the artifact is valid, not that the action is wise. A candidate can be perfectly correct and still extract a value from evidence you should not be processing at all. We hold real examples of technically flawless artifacts that faithfully extract "this item is no longer available" and "you do not have permission to view this page". The display must therefore carry the evidence classification next to the verdict, because the verdict alone cannot express this.

11. Evaluation protocol

Most reported numbers for systems of this kind are not comparable across time, because the checks change as the system improves. Fix that before anything else.

11.1 Re-score history with today's oracle

Keep every run's raw generator output. Score all runs, always, with the current oracle. A run from month one and a run from month three then differ only in what changed by design.

This has a consequence people find uncomfortable and should not avoid. A stricter oracle makes past runs look worse, and it makes the current run look worse than last month's. In our deployment, adding the regression gate dropped a headline precision figure from 51% to 49% on comparable populations, because the gate rejected candidates the previous run had happily called usable. That is the metric working correctly. A benchmark that only ever improves is measuring your optimism.

11.2 Define precision to exclude declining

Precision is acceptable candidates divided by candidates proposed. An instance the generator declined is not counted against it, because declining on evidence with no valid answer is correct behaviour, and penalising it would optimise directly against I4.

Report the decline rate separately. It is a real quantity, since it bounds recall, but it is not an error rate and must not be folded into one.

11.3 Two populations, reported separately

The laboratory population is a fixed, hand-picked set of twenty to fifty instances, used to attribute the effect of one change at a time. It is not representative and must never be quoted as system performance. Its job is causal attribution, and for that job a small fixed set is exactly right.

The field population is unselected production traffic. This is the honest number, and it will be lower.

Report both, labelled. In our deployment the laboratory figure was 91% and the field figures were 76% on healthy evidence and 49% on failing evidence. All three are true and each answers a different question.

11.4 Stratify by difficulty, and say why the strata differ

Do not average across populations of different difficulty. Our two entry points differ structurally rather than incidentally. One draws evidence where the artifact already fails, which is hard, because a substantial share of that evidence has no valid answer at all. The other draws evidence where the system is healthy, which is easier, because the value is present and a reference value exists for comparison. Averaging the two produces a number that describes no real workload.

11.5 One change at a time, with the question written down

For each experiment record five things: the question it tests, the single change made, the population it ran on, the result, and the decision taken. A sequence of such records is the most useful artifact your project will produce. It stops the team re-running a dead end six months later, and it lets a newcomer see that the gains came from constraints rather than from scale.

12. Diagnosing failure: model or evidence?

When precision plateaus, the instinct is to reach for a larger generator. Test that instinct, because it is cheap to test and it is usually wrong.

12.1 First, categorise every failure mechanically

Take a full run's rejected candidates and bucket them by why the oracle rejected them. In our deployment, across 442 rejections, the distribution was as follows.

Just under a third, 31%, resolved to nothing at all: a guessed chain of identifiers that matched no element on the page. A further 37% resolved to several things, which means the candidate was not scoped to the intended entity and would silently read a neighbouring one. Both of these are generator failures, and both are aggravated: the model had already been told, in its revision round, exactly what its candidate matched.

A further 13% violated a form constraint that had been stated in the prompt, typically a positional index or an embedded instance identifier. Another 9% resolved to exactly one element whose text was not a value at all, but a label, an identifier or an unrelated specification, which is a judgement failure rather than a mechanical one. A residual 2% did not parse.

The remaining 3% were ours: a defect in our own numeric coercion, which read a comma-decimal amount as an integer thousands of times larger on shops using European conventions. No model change would have fixed those.

Two conclusions follow immediately. First, 68% of the failures were candidates the generator's own in-loop check had already flagged, and the model returned them regardless. Those never reach a human, so queue quality is unaffected, but they establish that the ceiling is bounded by the model's compliance rather than by its ability to find the answer. Second, a measurable slice of the failures was a bug on our side, and had we not categorised them we would have attributed that slice to the model.

Alongside the failure taxonomy, compute the agreement rate on accepted candidates: where a candidate was accepted, did it produce the value production itself produced from the same evidence? Ours agreed in 98.5% of cases, which established that the evidence was sound and that the failures were not a data-quality problem in disguise.

12.2 Then run the escalation experiment

There is a cheap and decisive test of the "bigger model" hypothesis. Take only the candidates the oracle rejected. Send each to a larger or differently-trained generator, with the full context: the same evidence, the first generator's answer, and the oracle's explanation of exactly what that answer did. Score the results with the same oracle, and report four outcomes: repaired, withdrawn, still failing, and damaged, where damaged means slots that were correct before and are wrong now.

Our result, using a 30-billion-parameter code-trained model against a 9-billion-parameter general one, was as follows. On healthy evidence, 129 failing slots were sent; 28 of them, or 22%, were repaired; 4 were withdrawn; 97 remained wrong; and 24 slots that had been working were damaged. On failing evidence, 34 slots were sent; none were repaired; 14 were withdrawn; 20 remained wrong; and 5 working slots were damaged.

The larger model made the same mistakes as the smaller one, resolving to nothing and resolving to several things, while looking at an explicit list of what its predecessor's candidate had matched. When allowed to revise slots it had not been asked about, it damaged 29 that already worked.

The inference is that the limiting factor was the evidence window rather than the parameter count. The information needed to disambiguate lay outside the focused region we were sending. That is a retrieval and context problem, and it is fixed by widening and structuring the evidence, not by scaling the generator.

12.3 Escalate per-slot, never per-instance

If you do escalate, apply the second generator only to the specific slot that failed, and merge its answer back into the existing record. Our whole-instance replacement made the system worse, moving it from 50% to 46% on the same population, purely through collateral damage to slots that had been correct. Slot-scoped merging improved the same population from 75% to 80%.

13. Cost model

At realistic volumes, cost is dominated by fixed setup rather than by inference, and the intuition that inference is the expensive part leads to the wrong optimisations.

Attribution and compression are effectively free, running in seconds to minutes with no model involved. Generator provisioning is the dominant fixed cost: environment setup and weight loading consumed 21 of 46 minutes on a typical run of ours. Inference itself costs cents and a few seconds per instance. The oracle is free. Adjudication is the real cost, and it is measured in human minutes rather than in currency.

Five consequences follow for design. Batch aggressively, because two runs of a hundred instances cost far more than one run of two hundred. Parallelise by sharding the input rather than by scaling the machine, since three modest workers on three shards beat one large worker. Keep the environment warm across iterations during development, where the setup cost would otherwise be paid on every experiment. Make teardown independent of your session, because a crashed terminal must never leave a meter running: track resources by an external identifier, provide a one-command teardown, and, because local state files lie, verify against the provider's API that nothing is still running. Finally, optimise the adjudicator's minute rather than the inference cent: at our volumes, review time exceeds compute cost by orders of magnitude, and every display improvement that removes a click is worth more than a model change that removes a fraction of a cent.

14. Anti-patterns

Each of the following was either done by us or seriously proposed, and each carries a specific cost.

Going straight to the model, skipping attribution and compression, is the fastest thing to build and the most expensive to own. The model spends its budget on problems no repair can fix, and because nothing upstream partitions the input, the results cannot be attributed to any cause.

One instance per target looks like obvious deduplication and silently hides every target with more than one failure mode. You repair the smallest mode and declare the target fixed.

Reimplementing the consumer inside the oracle is faster to write and produces an oracle that drifts from production. The drift then masquerades as model error, and you tune the prompt to chase a bug in your own verifier.

Verifying only on failing evidence is tempting because that is the evidence you collected. It guarantees the incumbent loses every comparison, and you will ship regressions with confidence.

Auto-applying high-confidence candidates removes the bottleneck by removing the only component that can see semantic wrongness. Because wrong repairs are silent, you will not notice.

Trusting self-reported confidence is tempting because the number is right there in the output. It is uncalibrated, and it correlates with fluency rather than with correctness.

Iterating the critique loop until it passes looks like convergence and is overfitting to the single instance inside the loop.

Keying the queue by target alone is the simplest schema and causes multi-mode repairs to overwrite one another, and parallel sources to destroy each other's work.

Reaching for a bigger model at the plateau is the culturally default move. In our measurement it produced repair rates of 22% and 0% on the two populations, and damaged working slots as a side effect.

Reporting a single precision number makes for a cleaner narrative and averages populations of different difficulty, producing a figure that describes no real workload.

Normalising values into a fixed vocabulary before storing them produces tidier data and destroys the evidence. A substring match once mapped "not available" onto available; because only the mapped value was retained, the error was undetectable after the fact. Store what the source said, and normalise on read.

15. Design checklist

Before the first model call:

• [ ] The oracle exists, runs the production code path, and can be executed against historical evidence
• [ ] Attribution buckets are exclusive, exhaustive, owner-named, and assert to the total
• [ ] Compression groups by structure and carries a sibling instance per mode
• [ ] Evidence validity is classified, and invalid evidence is excluded with a recorded reason
• [ ] A store of successful evidence exists, with the value production extracted from it
• [ ] The queue key includes target, slot and source
• [ ] Verdicts are typed, each with an action attached; gates only demote
• [ ] The generator's contract states the constraints, their reasons, and that declining is correct
• [ ] Sampling is deterministic and seeded
• [ ] The critique loop is bounded at one round
• [ ] Approval writes one field, to the location the consumer reads, and records that path
• [ ] Teardown of paid resources works from a cold start and is verified against the provider

Before quoting a number:

• [ ] All runs re-scored with the current oracle
• [ ] Laboratory and field populations reported separately and labelled as such
• [ ] Populations of different difficulty reported separately
• [ ] Decline rate reported alongside precision, not folded into it
• [ ] The experiment log records the question, the single change, and the decision

When precision plateaus:

• [ ] Rejections categorised mechanically by oracle reason before any model change
• [ ] Agreement rate on accepted candidates computed, to rule out an evidence problem
• [ ] Escalation tested on rejected slots only, scored with the same oracle
• [ ] Damage to previously-correct slots counted, not just repairs

16. Instantiating the method in a new domain

Map your problem onto the terminology of section 2, then work through the checklist. Three worked mappings illustrate how little the method changes across domains.

In extraction rules, our own deployment, the artifact is a selector or path expression and the target is a monitored site. The evidence is the stored payload on which extraction failed. The oracle is the production parser together with value coercion. Uniqueness means the candidate resolves to exactly one element; generality means it holds on a second payload of the same template. The regression gate replays working payloads and requires their known values back, and the null-hypothesis gate asks whether the current selector already extracts. The trial runs both artifacts over fifty failing and fifty working payloads, and the adjudicator sees the value each one extracts, side by side.

In schema or configuration migration, the artifact is a field mapping or configuration key and the target is a source system. The evidence is a rejected record. The oracle is the real ingestion validator, not a copy of its rules. Uniqueness means the candidate maps to exactly one destination field; generality means it holds on a second record of the same shape. The regression gate replays previously-valid records and requires that they still validate, and the null-hypothesis gate asks whether the current mapping already accepts the record. The trial runs both mappings over a held-out batch, and the adjudicator sees which records each mapping accepts and rejects.

In flaky-test repair, the artifact is an assertion or wait condition and the target is a test suite. The evidence is a failing run's trace and timing. The oracle is the real runner executing the real suite. Uniqueness means the selector or wait target resolves to one thing; generality means the behaviour holds across N repeated runs rather than one lucky one. The regression gate requires that previously-passing tests still pass, and the null-hypothesis gate asks whether the test is failing for a reason that has nothing to do with timing. The trial runs both versions fifty times, and the adjudicator sees the pass rate and the timing distribution of each.

Four situations call for something other than this method.

When no mechanical oracle exists and condition C2 fails, build the oracle. If the oracle turns out to be the hard part, then the model was never your bottleneck and a harness will not help you.

When instances are few and individually valuable and condition C3 fails, use an interactive agent with tool access on individual cases. It costs an order of magnitude more per case and reads context a batch pipeline cannot, which makes it the right tool for a handful of hard targets and the wrong one for thousands. We run both, and the boundary between them is volume rather than difficulty.

When repairs require global reasoning and condition C5 fails, restructure the problem until the unit of repair is local, or accept that what you have is design work rather than maintenance work and staff it accordingly.

When wrong repairs are loud and condition C4 fails, you may be able to auto-apply with a rollback and skip the human gate entirely. Verify first that the failure really is loud in production rather than merely loud in a test environment, because the two are often confused and only one of them protects you.

Summary

The generator is the least important component of the system. What determines whether such a system produces trustworthy repairs is seven things: attribution, which refuses to work on failures no repair can fix, and which accounted for 78% of ours; compression, which yields one representative per genuine mode with a sibling for generality; evidence selection, a small and well-chosen window, because the excluded region is where the errors live; the oracle, running the production code path through layered checks and gates that only demote; a store of success, without which regression cannot be detected and will therefore be shipped; typed verdicts and durable decisions, so that reviewer effort accumulates rather than resetting; and honest evaluation, meaning re-scored history, separated populations, and a metric that is permitted to get worse.

Build those seven and a small general-purpose model is sufficient. Skip them and no model is.

Derived from a production deployment maintaining extraction rules across approximately 1,400 targets and 1.6 million monitored items. All quantitative claims are measurements from that deployment. They are offered as evidence for the design rules, not as benchmarks to reproduce.