Your evals pass. That doesn't mean they work.

Last week I wrote about tracelint, a linter that catches structural bugs in agent traces — the classic one being an agent that calls charge_card, gets a failure back, and just... keeps going and tells the customer their order shipped.

The response was better than I expected, and one question kept coming up in different forms: "Okay — but wouldn't my eval suite have caught that anyway?"

Fair question. So I checked. On the case I'll show you, it didn't. The suite was green. The charge was declined. The agent said "payment successful," and every eval passed.

This post is about the tool I used to find that — muteval — and what happened when I pointed it at the exact failure tracelint was built for.

The blind spot in a passing eval suite

Here's the thing about a green eval run: it tells you your system passed today's tests. It tells you nothing about whether those tests would notice if the system silently got worse.

That's not hypothetical. Your contains("refund") check passes whether or not the model still follows the refund rule someone deleted last week. Your faithfulness judge passes a confidently-wrong answer as long as it's grounded in the context. The eval is green, and the regression walks straight through it.

In normal software we have a name for "are my tests actually any good?" — mutation testing. You deliberately inject bugs (mutants) into the code, rerun the test suite, and measure how many the tests catch. mutmut and Stryker do this. muteval does it for eval suites — except the "code" it mutates is the system under test: the prompt, the retrieved context, the tool outputs, the model.

Degrade the system → rerun your existing evals → report the percentage of injected regressions they caught. The ones they miss are survivors — concrete coverage gaps.

The declined charge

Back to the agent. Here's a deliberately naive payments agent: it calls charge_card and reports success. Its eval suite has one realistic semantic check — does the reply confirm the charge went through?

muteval has an operator called deny_tool_output. It takes a tool's result and turns it into the nastiest kind of failure: a domain failure returned as transport success — an HTTP 200 whose body says {"status": "declined"}. Structured-error handling never fires, because nothing technically errored. The agent proceeds. The final answer still reads "Your payment was successful."

Rerun the eval suite against that mutant:

[HIGH] SURVIVED  [deny_tool_output]
          tool output #1 returned a domain failure (HTTP 200 + status:declined)

Mutation score: 0%. The suite caught nothing. The card was declined, the agent lied, and the contains("successful") check passed — because the answer does still say "successful." That's a survivor, and it's exactly the kind of blind spot you'd never see from a green CI run.

(To be clear about the number: 0% here means of the regressions I injected, your evals caught none of them. Not "your system is broken" — "your tests wouldn't have noticed.")

Closing the gap — where the two tools meet

Here's the part I think is actually interesting.

The thing that catches "declined charge reported as success" isn't a smarter output judge — the output looks fine. It's a structural check on the trace. Which is exactly what tracelint does.

So I wired tracelint in as one of muteval's evals. tracelint reads the agent's trace, and if the tool declares what failure looks like (failure_when: {"pointer": "/status", "in": ["declined"]}), it flags the declined charge deterministically — no judge, no key.

Add that one check and rerun:

verdicts on the declined-charge mutant:
  semantic eval (confirms "successful")  -> PASS   (misses it)
  tracelint (declared failure contract)  -> FAIL   (kills it)

mutation score:  semantic eval alone 0%  ->  + tracelint 100%

muteval didn't just tell me the gap existed — it let me prove the fix closes it: the mutant that survived the semantic suite dies the moment the structural check is in the suite, and the baseline stays green. That's the whole loop — a survivor names a missing eval, you add it, the score goes up.

A tool fault is a mutant; a trace-lint rule is an eval. Mutation testing on one side, deterministic trace checking on the other, and they compose.

What this does NOT tell you (the part I care about most)

A diagnostic you can't trust is worse than none, so:

• A survivor is a candidate, not a verdict. muteval tells you your eval missed an injected change. Whether that change could really happen, and would really be bad, is a human call. Some survivors won't survive that question.
• Mutation coverage is not validity. A suite can be highly sensitive to mutations and still be wrong — sensitivity isn't the same as agreeing with ground truth or a human. That needs labels; no label-free tool gets you there.
• The mutations are rule-based, synthetic edits. They model real regressions; they aren't identical to them. Whether mutants predict the failures your system actually experiences is the open question I'm still chasing.
• It's a per-suite diagnostic, not a universal flaw-finder. It tells you where your suite has a hole — not a new universal truth about evals.
• It needs a re-runnable system. muteval degrades the system and needs a fresh output per mutant. A frozen CSV of outputs can't be mutated.

muteval fails closed on all of this: a red baseline, too few mutants, or too many errored mutants means it refuses to emit a score rather than hand you a confident-but-meaningless number.

Try it

There's a demo that runs with no key — a mock model, so you can watch the mutate → survive → fix loop end to end in about a minute. (Running it on your own suite calls your model + a key, like any eval.)

git clone https://github.com/AshwinUgale/muteval && cd muteval
pip install -e ".[tracelint]"
python examples/agent_tool_fault/run_demo.py     # keyless, ~1s

Repo: https://github.com/AshwinUgale/muteval (Apache-2.0) · tracelint: https://github.com/AshwinUgale/tracelint (MIT).

And the genuine ask, same as last time: if you write evals for agents or RAG, I'd like to know what your suite would miss. What's the regression you're most afraid of shipping — and if you injected it, would it survive your evals? Issues, replies, and war stories all welcome.