Verdict: Evidence-First Agent Harness for Reproducible Bug Fixes

Most flaky bug reports end in one of two places: "cannot reproduce" or a patch nobody can prove fixed the problem. Verdict enforces a stricter contract: bugs are innocent until reproduced. No patch, no claim of success, no merge until the agent produces verifiable evidence that the failure exists and can be triggered on demand.

This is not an autonomous patch generator. It is an evidence-producing agent harness for the difficult step that comes before a patch: proving the bug is real, isolating the trigger, and building a regression test that prevents recurrence.

The Reproduction-Proof Boundary

Verdict treats bug investigation as a bounded experiment, not a conversation. An LLM can read a stack trace and propose a plausible explanation quickly. Plausible is not the same as reproduced. For an intermittent failure, the questions that matter are concrete:

• Which condition actually triggers it?
• How often does it fail under that condition?
• What happens under a contrasting control?
• Which repository range does the evidence support?
• What regression test would prevent the same failure from returning?

The harness refuses to claim a reproduction unless the evidence crosses a deterministic threshold. Every observation stays in the evidence ledger. An inconvenient result cannot disappear just because it weakens the story.

Three-Act Investigation Flow

Verdict uses three bounded subagents that run sequentially:

GitHub issue
    |
    v
Hunter: find the trigger
    |
    v
Surgeon: localize the change
    |
    v
Insurance: keep it fixed
    |
    v
Maintainer review

Each agent has a specific scope and cannot proceed without satisfying its evidence contract.

Hunter

Hunter searches only the condition matrix and command budget approved by the maintainer. It runs an approved command repeatedly under approved conditions. Successful, failed, partial, and unresolved runs all stay in the evidence ledger.

The agent does not get to cherry-pick results. If a condition fails 3 times out of 10, that ratio is part of the evidence. If a different condition fails 10 times out of 10, that difference is part of the evidence.

Hunter outputs:

• Trigger condition (environment variables, input files, timing constraints)
• Failure rate under that condition
• Control condition (what does not trigger the failure)
• Execution artifacts (logs, snapshots, traces)

Surgeon

Surgeon narrows the reproduced condition to the smallest suspect range the records support. It uses the trigger condition from Hunter to bisect the repository history or module boundary.

Static inspection stays visibly different from a proven execution boundary. Surgeon does not author a patch. It localizes the change that introduced the failure and hands that range to the maintainer.

Surgeon outputs:

• Suspect commit range or module boundary
• Execution evidence at each boundary
• Contrasting evidence from a known-good state

Insurance

Insurance converts the reproduction into a regression plan: the test name, fixture, failing assertion, and expected pass condition. This is not a patch. It is a test case that will fail until someone fixes the underlying issue.

Insurance outputs:

• Test case that reproduces the failure
• Fixture setup (environment, input data, timing)
• Assertion that captures the failure mode
• Expected behavior after fix

The maintainer reviews the test case, merges it (still failing), and then works on a patch. The patch is only considered successful if the test case passes.

Orchestration Shape

Verdict runs as a GitHub Action or standalone CLI. The maintainer configures the harness with:

• Allowed commands (no arbitrary shell execution)
• Allowed environment variables (no credential leakage)
• Allowed file paths (no write access outside workspace)
• Budget (max runs, max wall time, max cost)

The harness enforces these boundaries at runtime. An agent cannot escalate privileges or exceed the budget.

State Management and Artifact Storage

Every run produces:

• Command invocation (exact arguments, environment)
• Exit code and signal
• Stdout and stderr
• Timing (start, end, wall time)
• Snapshot (file diffs, memory state, network captures)

These artifacts are stored in a structured ledger, not a flat log. The ledger is versioned alongside the repository. An agent cannot rewrite history or hide a failed run.

The harness uses a content-addressed store for artifacts. Identical outputs (same stdout, same exit code) get deduplicated. This keeps the ledger compact even when an agent runs the same command 100 times.

Security Boundaries

Verdict does not trust the agent. The harness enforces:

• Command allowlist: Only pre-approved commands run. No shell interpolation, no piping, no subshells.
• Environment isolation: Only pre-approved environment variables are visible. Credentials stay in the maintainer's secret store.
• Filesystem boundaries: Agents read from the repository and write to a scratch directory. No access to home directories, no access to system paths.
• Network boundaries: Outbound requests go through a proxy that logs every destination. No arbitrary API calls.
• Budget limits: Max runs, max wall time, max cost. The harness kills the agent when the budget is exhausted.

The maintainer reviews the evidence ledger before merging any test case. The agent never gets write access to the main branch.

Observability and Failure Modes

The harness exposes:

• Run ledger: Every command invocation, every result, every artifact.
• Agent trace: LLM calls, tool calls, reasoning steps.
• Budget consumption: Runs used, time used, cost used.
• Boundary violations: Attempts to run unapproved commands, access forbidden paths, exceed budget.

Common failure modes:

• Cannot reproduce: Hunter exhausts the budget without finding a trigger. The ledger shows what was tried. The maintainer can adjust the condition matrix or budget.
• False positive: Hunter claims a reproduction, but the failure rate is too low or the control condition also fails. The evidence ledger exposes this. The maintainer rejects the claim.
• Scope creep: Surgeon tries to bisect a range that is too large or too vague. The harness enforces a max range size. The maintainer can narrow the suspect boundary manually.
• Weak test: Insurance produces a test case that is too brittle or too vague. The maintainer reviews the assertion and fixture before merging.

Deployment Shape

Verdict runs as:

• GitHub Action: Triggered by issue labels or comments. The harness runs in a GitHub-hosted runner. Artifacts are stored in GitHub Actions cache or S3.
• CLI: Triggered by the maintainer locally. The harness runs in a container. Artifacts are stored in a local directory.

The harness does not require a persistent server. It is stateless except for the evidence ledger, which is versioned alongside the repository.

Technical Verdict

Use Verdict when:

• You have intermittent failures that are hard to reproduce manually.
• You want a regression test before you invest time in a patch.
• You need a verifiable audit trail for bug investigations.
• You want to enforce a budget on agent exploration.

Avoid Verdict when:

• The bug is already reproducible with a one-line command. You do not need an agent harness for that.
• You want an autonomous patch generator. Verdict stops at the test case. The patch is still your job.
• You have a low tolerance for false negatives. Hunter may exhaust the budget without finding a trigger if the condition matrix is too sparse.

The core insight is the reproduction-proof boundary. An agent cannot claim success without evidence. An agent cannot hide a failed run. An agent cannot escalate privileges or exceed the budget. This turns a flaky bug report into a verifiable experiment.

Source Links

• Verdict: Evidence-First Agent Harness