I Built a Leak Detector for API Keys. My Synthetic Tests Caught 8 Out of 20

The honest story of building Cerberus and why I need real traffic to finish it.


A few days ago I wrote here about a test that failed. It was a synthetic data generator that fell over on real-world noise. I got more engagement on that post than anything I'd written before, which taught me something: developers don't trust perfection. They trust someone who shows the mess.
So here's the mess I'm in now.
The Problem
I sell API access for a living. A few months ago a customer's key leaked—committed to a public repo, scraped by a bot, used from a hundred places. We found out when the bill landed. The gap between the leak and the discovery is where all the damage lives.
What I Built
Cerberus watches usage metadata per API key and looks for one signature: a key being used from many places at once, each doing very little work.
A customer scaling legitimately runs on more machines and does proportionally more work, so requests-per-machine stays constant. A leaked credential inverts that ratio. More origins, less work per origin.
The detection rule uses three signals:
• Origin count per key
• Work per origin
• Network spread (unrelated networks vs. one cloud provider)
When all three hold for three consecutive hours, it sends one Slack message. That's it. No dashboard. No daily report. No noise.
Where It Falls Apart
I ran 20 synthetic leak scenarios through it. It caught eight.
The misses are all on the most common key shape: low baseline traffic, suddenly distributed. The thresholds are guesses. Every number in the rule is a guess, and the code comments literally say so.
I need real traffic history from actual API platforms to calibrate it. Synthetic data doesn't have the weird edge cases that real customers create.
The Privacy Architecture
Before anyone asks: keys and IPs never reach my server.
The SDK computes an HMAC-SHA256 of each value under a secret salt the customer holds. I never receive that salt, so I cannot reverse a fingerprint—not for the customer, not for an attacker who breaches me, not for anyone who compels me legally.
Twelve metadata fields only. Timestamp, hashed key, route template, token counts, latency, status, three hashed network fingerprints, address family, optional cost. The ingest endpoint rejects anything else by name. There is no field for a prompt, a response, a key, an address, or a user identifier. They can't be sent even by accident.
What I'm Looking For
Three API companies willing to install it free forever in exchange for sharing two weeks of traffic history so I can tune the detection.
First installs are done by hand on a call, deliberately, so there's a person to ask when something looks wrong.
If you sell API access and have ever worried about a leaked key, hit reply