I ran my scanner against 5 real CVEs. It missed 4. Then I reverted my own fix.

Why this post is different from the last one

The last write-up in this series announced four tools. This one is about what happened when I stopped writing tests for my own tools and started checking one of them against reality — and about the fix I built, tested, shipped, and then took back out, because it was wrong in a way that only showed up once I looked past the headline number.

The setup: inlet, and the claim it hadn't actually tested

inlet is a static scanner: point it at a Python codebase, it finds every call site that looks like SQL execution — raw DB API calls, Django's .raw()/.extra(), SQLAlchemy's text() — and classifies each as parameterized, concatenated, or uncertain. Every claim in its original README was backed by 7 hand-written fixtures, each proving one specific classification rule.

That's real, but it's not evidence of anything beyond "the mechanism works on cases designed to exercise it." So I built a real-world evaluation: 15 PyPI packages, split into two groups.

Group A (5 packages): each with a documented, independently verified historical SQL-injection CVE, with the fix commit or advisory located ahead of time so I could check inlet's output against ground truth, not against inlet's own opinion of itself.

Group B (10 packages): popular, no known SQLi history — a noise-floor check on how often inlet flags something that isn't actually a risk.

~772K lines of real code, scanned unmodified, in about 10.6 seconds combined.

The result: 0 for 5

Django (CVE-2022-28346), Apache Superset (CVE-2023-49736), Tortoise ORM (CVE-2020-11010), and Airflow's common-sql provider (CVE-2025-30473) were all complete misses. Archery's CVE-2023-30556 was a partial — the exact vulnerable line showed up in inlet's output, but classified uncertain instead of concatenated, because the f-string assignment sat inside a try: block, a name-resolution gap beyond the documented scope wall.

The pattern behind all four full misses was identical: the vulnerable code never literally calls something named .execute(), .raw(), .extra(), or text(). It goes through a framework's own abstraction — hook.get_records(), field.like(), a plain helper function — that eventually reaches SQL execution, several calls away from any name inlet recognizes.

I put that as the headline of EVALUATION.md, not a footnote. A 0/5 result, reported plainly, is worth more than a clean-looking demo — it's the first piece of evidence in this whole project that came from checking against something I didn't design.

The smaller, real bug the same evaluation found

Group B surfaced something fixable: one package's uncertain findings were 67% false positives, from a name collision. peewee's own query builder has an .execute(database) method — same method name as a real DB cursor's .execute(sql), completely different meaning. inlet's name-only matching couldn't tell them apart.

The fix that worked, and then didn't

I built a positive-evidence rule: only treat an .execute()-shaped call as a real DB-idiom candidate if there's actual evidence for it — either the argument is string-shaped, or the receiver chain shows a .cursor() call or a conventional cursor/connection name. Otherwise, exclude it.

It worked, exactly as intended, on peewee: 33 uncertain findings down to 9, a clean diff confirming all 24 removed were the exact false-positive shape, zero true positives lost.

Then I re-ran the other 9 Group B packages, and found the same rule had silently dropped 94 real database call sites — Django's SchemaEditor.execute(), SQLAlchemy's own Engine/Session internals, a dataset helper, SQLModel's super().execute(). All real DB calls, lost for one reason: their receiver was named something generic like self, which the new rule couldn't distinguish from peewee's unrelated Query.execute().

Why I reverted it instead of tuning it further

There's a version of this where I keep iterating the heuristic, trying to find a cleverer rule that keeps the peewee win without the 94-finding cost. I didn't do that, because the actual finding underneath both results is more important than either number:

When the argument isn't string-shaped and the receiver name is generic, there is no way to tell a real DB wrapper from an unrelated same-named method using local syntax alone. self.execute(x) is genuinely, irreducibly ambiguous from where inlet sits. That's not a heuristic to keep tuning — it's the same category of hard limit as the tool's existing cross-function-scope wall.

And there's an asymmetry that matters more than either number: a finding in uncertain is recoverable — a human can look at it and dismiss it. A finding that's silently excluded is not recoverable — it never existed for anyone to see. Trading visible noise for confident silence is a strictly worse failure mode, even when the summary metric (fewer uncertain findings!) looks like an improvement.

So I reverted it. Every .execute()-shaped candidate goes back to being reported, at whatever verdict the classifier can actually support — no silent exclusion, ever. The receiver-evidence detection code is still there, inert, available as a future upgrade-only signal (never a removal signal) if a principled way to use it that way ever turns up.

I wrote the whole thing up as its own section in EVALUATION.md — what broke, why it was reverted, and the actual finding — because a failed attempt with an honest postmortem is a better artifact than either the original bug or a fix that quietly traded one failure mode for a worse one.

Then, a fifth tool: escrow

Separately, I built escrow, which vets a Python package before a real pip install by actually installing and importing it in a sandbox first — built directly on two earlier tools in this series: husk (the hardened sandbox) and witness (the audit-hook behavior reporter). This exists because of slopsquatting: LLMs hallucinate plausible-but-nonexistent package names at meaningful rates, attackers register those exact names, and the next pip install executes whatever they put there — this is already a real, documented attack pattern, not a hypothetical.

Building it surfaced a real limitation in witness's own technique: witness observes behavior by prepending an audit-hook preamble to a script running in one interpreter process. That can't see into pip's own build-backend subprocess — exactly where install-time (setup.py) attacks actually run. escrow's hook ships instead as a real sitecustomize.py, auto-loaded by Python's own site module in every subprocess pip spawns, not just the top-level driver. Found and fixed empirically, including discovering that pip install silently swallows successful build-step subprocess output unless run with --verbose — which would have hidden a caught-and-ignored malicious write from the very report meant to catch it.

escrow keeps its own honest limit stated up front: installing a real package requires network access, so anything malicious that completes fast enough during that window can be detected and reported, but not prevented in real time. That's not a gap to be engineered around in v0.1.0 — it's a fundamental property of vetting something that needs network access to install at all.

The actual pattern across all five tools now

secfix refuses to say "fixed" without a fresh trace. husk backs every hardening claim with an adversarial test. witness turns its own blind spot into a loud signal instead of a silent gap. inlet measured itself against real CVEs, got a bad number, and reported it as the headline. And when a fix improved that number by making the tool quietly worse in a different way, it went back out — documented, not buried.

That's the thing I'm actually trying to build a track record of. Not five clever tools. Five tools that keep finding their own mistakes before anyone else has to.

Repos: github.com/balbaks/secfix · github.com/balbaks/husk · github.com/balbaks/witness · github.com/balbaks/inlet · github.com/balbaks/escrow