Pin the Seed and Keep the Contract: A Merge Gate for Agent Patches

An agent patch that lands three new examples is not evidence. It is a story the patch told about itself. The merge gate should accept the change only when public-API contracts hold under a recorded random seed, and when flaky failures are frozen as seed records rather than skipped tests.

Example assertions collapse when the next legal input arrives. Contracts do not, if they sit on the boundary the caller actually uses.

The green bar is the wrong score

Agent patches optimize for the tests they can see. The usual failure is a fresh assertEqual that restates the implementation. The bar turns green. The bug remains one argument away.

Property checks at the public boundary reverse that incentive. They do not ask whether a fixture matches the patch. They ask whether every legal input still satisfies the contract the module advertised before the patch arrived.

That distinction is the method. Everything below is enforcement.

Three files, one gate

Keep the review surface small. Three files are enough.

1. contracts.yml — predicates on public functions only.
2. seed_ledger.jsonl — one line per property run: seed, trial count, result, hash of the failing input.
3. flake_freeze.yml — time-boxed freezes that pin a failing seed. They never call skip().

Internal helpers stay outside the agent's test budget. If a helper needs coverage, a human adds it after merge, not in the same diff that claims to fix production behavior.

Decision table

The table is policy. The harness below is the lock.

Workflow

1. Lock the public surface

List the functions callers import. Ignore _private names. Emit a candidate list if no contract file exists. Label the output as a proposal until a human edits the predicates.

# propose_contracts.py — proposal only; do not merge unreviewed output
from pathlib import Path
import ast
import sys

def public_functions(path: Path):
    tree = ast.parse(path.read_text())
    for node in tree.body:
        if isinstance(node, ast.FunctionDef) and not node.name.startswith("_"):
            yield node.name, [a.arg for a in node.args.args]

if __name__ == "__main__":
    src = Path(sys.argv[1])
    for name, args in public_functions(src):
        print(f"{name}({', '.join(args)})")

python propose_contracts.py src/billing/ledger.py

The printed names are a candidate list. They are not a specification. A patch that adds contracts for private helpers is a smell: the agent is testing its own scaffolding.

2. Write contracts as predicates, not examples

Idempotency, ordering, and error paths catch patches that example tests miss. Keep each predicate pure enough to call twice in one process, or replace it with a compare-after-commit check when the real system has side effects.

# tests/contracts/test_ledger_contracts.py
from hashlib import sha256
import json
import os
import random
import time
from pathlib import Path

LEDGER = Path("seed_ledger.jsonl")
TRIALS = int(os.environ.get("CONTRACT_TRIALS", "200"))
SEED = int(os.environ.get("CONTRACT_SEED", "0")) or int(time.time())

def shrink(value: str) -> str:
    cur = value
    while len(cur) > 1:
        trial = cur[:-1]
        if not holds(trial):
            cur = trial
        else:
            break
    return cur

def holds(invoice_id: str) -> bool:
    from billing.ledger import apply, apply_error
    if invoice_id.endswith("-"):
        try:
            apply(invoice_id)
        except apply_error:
            return True
        return False
    once = apply(invoice_id)
    twice = apply(apply(invoice_id))
    return once == twice

def record(seed: int, trials: int, ok: bool, witness: str | None) -> None:
    row = {
        "seed": seed,
        "trials": trials,
        "ok": ok,
        "witness_sha256": sha256(witness.encode()).hexdigest() if witness else None,
        "ts": int(time.time()),
    }
    with LEDGER.open("a") as fh:
        fh.write(json.dumps(row) + "\n")

def test_apply_contract():
    rng = random.Random(SEED)
    for i in range(TRIALS):
        suffix = rng.choice(["usd", "eur", ""])
        sample = f"INV-{rng.randint(1, 10_000):05d}-{suffix}"
        if not holds(sample):
            witness = shrink(sample)
            record(SEED, i + 1, False, witness)
            raise AssertionError(f"seed={SEED} witness={witness!r}")
    record(SEED, TRIALS, True, None)

Empty currency codes are legal in this sketch. That is the point. Agent patches often assume a field is always populated, then write an example that never leaves the happy path.

Two hundred trials is a budget, not a proof. Raise CONTRACT_TRIALS when the input space is wide. Lower it when each trial hits a network. Do not treat a large number as a substitute for a missing predicate.

3. Freeze the seed, never the test

A flake that disappears on rerun is not permission to skip. Capture the seed and the witness hash. Keep the test required on that seed. Other seeds stay non-blocking until the freeze expires.

# flake_freeze.yml — human-owned; agent patches that touch this file fail CI
freezes:
  - id: F-184
    test: tests/contracts/test_ledger_contracts.py::test_apply_contract
    seed: 1725781203
    witness_sha256: "9c56cc51b374c32ba11507e6e0c2484e"
    owner: "platform-payments"
    expires: "2026-09-15"
    reason: "empty currency code vs concurrent apply"
    skip: false

skip: false is load-bearing. A freeze that skips is a deleted test with extra YAML. The unit of quarantine is the seed plus the witness hash, not the test name alone. Two flakes in the same test are two ledger rows.

Validate the freeze file before CI reads it. The check is boring on purpose.

# tools/check_flake_freeze.py — proposal-grade validator
from datetime import date
from pathlib import Path
import sys
import yaml

ALLOWED = {"id", "test", "seed", "witness_sha256", "owner", "expires", "reason", "skip"}

def main(path: Path) -> int:
    data = yaml.safe_load(path.read_text()) or {}
    today = date.fromisoformat("2026-09-08")
    for row in data.get("freezes", []):
        extra = set(row) - ALLOWED
        if extra:
            print("unknown keys", extra)
            return 1
        if row.get("skip") is not False:
            print("skip must be false", row.get("id"))
            return 1
        if not row.get("owner") or not row.get("witness_sha256"):
            print("owner and witness required", row.get("id"))
            return 1
        exp = date.fromisoformat(str(row["expires"]))
        if exp < today:
            print("expired freeze is a hard fail", row["id"])
            return 1
    return 0

if __name__ == "__main__":
    sys.exit(main(Path("flake_freeze.yml")))

Shadow job, pinned to the frozen seed:

python tools/check_flake_freeze.py
CONTRACT_SEED=1725781203 CONTRACT_TRIALS=50 \
  pytest tests/contracts/test_ledger_contracts.py -q

Merge-blocking job uses a fresh seed every run, then replays every unexpired freeze seed.

python tools/check_flake_freeze.py
CONTRACT_TRIALS=200 pytest tests/contracts -q
python - <<'PY'
from datetime import date
from pathlib import Path
import yaml, os, subprocess, sys
today = date.fromisoformat("2026-09-08")
data = yaml.safe_load(Path("flake_freeze.yml").read_text()) or {}
for row in data.get("freezes", []):
    if date.fromisoformat(str(row["expires"])) < today:
        sys.exit(f"expired freeze {row['id']}")
    env = os.environ.copy()
    env["CONTRACT_SEED"] = str(row["seed"])
    env["CONTRACT_TRIALS"] = "50"
    r = subprocess.run(["pytest", "tests/contracts", "-q"], env=env)
    if r.returncode != 0:
        sys.exit(r.returncode)
PY

If the freeze expires on 2026-09-15 and the review is on 2026-09-08, the owner has a week. After expiry, the same seed is a hard fail. The only legal exit is deleting the freeze after the contract holds on that seed.

4. Keep the generator off the oracle

Do not let the same process invent contracts and mark them passing. A model can propose predicate names and edge cases from a public-API list. A separate process executes them. The proposal is untrusted input, same class as the agent patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access fits the proposal step: emit candidate contracts and shrinking ideas from signatures and types. The free server option fits the execution step: run the trial budget and the freeze-seed replay without folding those logs back into the model context. Neither step replaces review. The model does not edit flake_freeze.yml. If those options are unavailable, run proposals on a workstation and the harness in CI. The split is the method. The machines are interchangeable.

5. Reject patches that grow examples without a contract delta

examples=$(git diff origin/main -- tests/ | grep -c 'assertEqual' || true)
contracts=$(git diff origin/main -- tests/contracts/ | grep -c 'def test_' || true)
freezes=$(git diff origin/main -- flake_freeze.yml | wc -l)
if [ "$freezes" -gt 0 ]; then echo "agent must not touch flake_freeze.yml"; exit 1; fi
if [ "$examples" -gt 0 ] && [ "$contracts" -eq 0 ]; then
  echo "example-only test diff is unproven"
  exit 1
fi

If the example count rises and the contract count does not, fail the check. Humans can override with a signed note. Agents cannot. A contract delta that only restates an example in property clothing should fail review the same way: look for a predicate that can fail on an input the patch did not hard-code.

What a passing run does not prove

A green contract job with 200 trials is a sampled argument. It is not a proof of the predicate. Gaps in the generator are gaps in the gate. If invoices never include an empty suffix, the error-path branch never runs, and the ledger will still look healthy.

Seed recording does not make a concurrent bug deterministic. It only makes a sequential RNG bug replayable. If the flake is a race, add a thread or async harness. Do not freeze a race as if it were a bad seed. The witness hash will not save you.

Contracts on the wrong boundary lie. If the public function is a façade over two services, an idempotency check on the façade can pass while a downstream write duplicates. Push the predicate to the service the caller cannot see only when that service's tests are also in the gate.

Who should not use this

Do not use seed-ledger freezes on UI snapshot tests. Snapshots are examples. They belong in a different queue, with a different owner, and they should not share flake_freeze.yml.

Do not use sampled contracts on cryptographic code. "Probably holds for 200 samples" is the wrong claim. Use constant-time tests and known-answer tests instead.

Do not point a model at production credentials to help generate contracts. The proposal lane should see signatures and types, not live rows.

Do not install a freeze policy if nobody owns expiry. An unowned freeze is a skip with extra keys. The validator above exists to make that failure loud on 2026-09-08, not quiet six months later.

Limitations of the harness

The shrinker is linear and string-only. Structured inputs need a type-aware shrink, or the witness will be a long string that hides the field that actually broke. The ledger is append-only JSONL without rotation; trim it in a scheduled job or the file becomes the slowest part of CI. The idempotency sketch assumes apply is safe to call twice in one process, which is false for many ledgers. Replace it with a compare-after-commit check when writes are real.

None of those limitations excuse merging on examples alone.

Review checklist

1. Did the patch add or edit a public contract? If not, it is unproven.
2. Did any contract fail under a recorded seed? If yes, reject and attach the witness.
3. Did a fail vanish on rerun? Freeze the seed with an owner, a witness hash, and an expiry. Do not skip.
4. Did the agent edit flake_freeze.yml? Reject.
5. Did the freeze expire? The seed is now a required fail until the contract holds.

Park the new assertEqual. Merge on the contract and the seed.