I put my cost router on a neutral benchmark. It ranked near the bottom, and that's the interesting part

A few weeks back I repriced three months of my own Claude Code usage. Real traffic, not a demo: 39.5 billion tokens across 139,835 requests. At API rates that's about $30,000, and 91% of it went to Opus because that's what the default reaches for. Route the fraction a smaller model handles as well, and the bill drops by roughly 50 to 58%. Five or six grand a month, on work that was already done and graded.

That's why OmnisRouter exists. It sits in front of your agents, reads each request, and sends it to the cheapest model that can answer it. The full write-up and the method are in the flagship post.

Saving your own money on your own data is one thing. I wanted an outside check on somebody else's turf, with somebody else's scoring. So I took the router to RouterArena.

What RouterArena is

RouterArena is an independent benchmark for LLM routers, out of an ICLR 2026 paper. It's a proper piece of work: 809 queries in the quick split, spread across 39 datasets, everything from MMLU-Pro and code generation to translation, chess and medical multiple-choice. You plug your router in, it picks a model per query, the harness runs that model and grades the answer, and you get an accuracy number, a cost number, and a combined Arena score. There's a hard rule I stuck to: you evaluate on their data, you never tune on it. No gaming the test.

I wired OmnisRouter in with the pool I care about: GPT-5, Opus-5, Claude Haiku, and GPT-5-nano. The models a team running coding agents genuinely uses.

First, two bugs, because honesty starts at home

The first scored run came back at 42% accuracy, which was obviously wrong. So before I say a word about the result, here's what I found when I opened it up.

Opus was scoring 0% on every single query. The scoring path was grabbing the model's extended-thinking block instead of its answer text, so the grader was marking a raw thinking object wrong 40 times out of 40. Second, my query file was missing the per-dataset answer-format instruction the grader keys on, so most models answered correctly in prose and got marked wrong because the grader couldn't find the answer in the shape it expected.

Both fixed. Opus went from 0% to 85%, the overall number went from 42% to 72.7%, and abnormal entries went from 558 to zero. If you run a benchmark and the number looks too bad to be true, it usually is, and it's usually your plumbing.

The honest result

On its own pool, OmnisRouter scores 72.7% accuracy at $3.71 per thousand queries, for an Arena score of 0.669.

That ranks it 16th out of 18. Near the bottom. On the easier split, even.

I could stop there and let you think I'm burying it, so I won't. Here's the top of the cost board next to us:

router              accuracy   cost/1K   Arena
Paix2-router          79.7%    $0.27     0.776
cross-router          78.2%    $0.29     0.762
sqwish-router         76.4%    $0.16     0.754
hybrid-router         71.4%    $0.04     0.721
omnisrouter           72.7%    $3.71     0.669

Look at the cost column. The leaders come in at four cents to twenty-nine cents per thousand. We come in at $3.71. That's the whole gap, and it isn't routing skill.

What the cost board rewards

I went and looked at what the top routers route to. Every one of them runs a cheap open-model pool:

• Paix2 sends 78% of its traffic to a single flash model.
• cross-router and sqwish run mostly DeepSeek-v4-flash and Gemini-3.1-flash-lite.
• hybrid-router puts 95% of everything through one Qwen model.

Not one of the leaders uses GPT-5, Claude, or any frontier model. And that makes sense, because the Arena cost axis rewards spend, so the way to win it is to route to the cheapest models that still clear the accuracy bar on an academic test set. Fair enough as a benchmark design. But it means the board measures how cheap your pool is at least as much as how well you route it, and a router built for the premium models people run their agents on starts the race carrying a piano.

I'm not knocking the benchmark, to be clear. It's careful and the accuracy scoring is sound. I'm saying the cost ranking answers a different question than the one I care about.

The warts, in full

"The board's unfair to premium pools" is a convenient thing for me to say, so I ran the check that could prove me wrong. RouterArena also computes optimality metrics: how close each router gets to the oracle, the best model per query, within its own pool. That strips out the cheap-pool advantage entirely and measures routing quality on its own.

Here's where we land:

router            Opt.Sel   Opt.Acc
Paix2-router       89.7%     100.0%
BARouter           64.4%      93.8%
agentforge         51.1%      98.7%
auto_router        37.7%      86.0%
r2-router          24.5%      99.8%
omnisrouter        23.7%      81.5%

Opt.Sel is how often the router picks the cheapest correct model. Opt.Acc is how much of its pool's achievable accuracy it captures. We're 8th of 13 on selection, and last of 13 on accuracy captured.

That last number is the honest one. 81.5% means OmnisRouter leaves more winnable accuracy on the table than anyone else on the board, and it does that on purpose. It's a cost-first router. Faced with a query a cheaper model can probably handle, it takes the cheaper model and the saving, even when the strongest model in the pool would have nudged the answer over the line. Tune it to chase accuracy and that number climbs, but then it stops being the thing I built. Some of the high selection scores above come from routers that barely route at all, by the way: park 95% of traffic on one cheap model that's usually the cheapest-correct and you "win" selection by barely choosing at all. I'd rather show you my mediocre 23.7% and tell you why than dress that up as skill.

The number that matters

A leaderboard rank isn't the point, and I'm not going to pretend a benchmark built around cheap open models is the scoreboard OmnisRouter should be judged on. The point is the bill.

On real Claude Code traffic, on the frontier models a working team runs every day, routing the cheaper-capable fraction cuts the cost by about half. The bill and the 91% Opus share are measured from three months of real requests. The saving is what comes off that bill once you route the work a smaller model handles, sized from the fraction OmnisBench measures as routable, so it's a reprice rather than a reroute I ran end to end. No academic test set moves it, and no cost board can rank it, because none of them run the models the work is happening on.

If your agent bill has a comma in it and you don't know where the money goes, that's the problem worth solving. The leaderboard was me checking my own work in public. Turns out the honest version is more useful than a good rank would've been.

OmnisRouter and the OmnisBench benchmark behind it are both open, Apache-2.0, and the numbers above are reproducible with RouterArena's own harness. Bring your own keys and check me.