If you run an agent workflow in production, you have almost certainly thought about building a model router.
Frontier models cost twenty to fifty times more than compact reasoning models. For three-quarters of your daily tasks, a smaller model running on commodity infrastructure produces an identical result. The obvious architecture is to place a lightweight classifier in front of your gateway: examine the incoming prompt, predict which model will win the task, and dispatch the request to the cheapest candidate capable of solving it.
Almost everyone implements this as a standard classification head. You take historical run logs, mark the model that succeeded with the lowest latency or lowest token cost as the winner, and train a model to output that label.
That setup looks clean on paper, yet these routers regularly degrade in production. They route delicate multi-step tool calls to brittle models that crash, and they fail to beat a simple fallback policy that always calls a single frontier model.
A comprehensive study from researchers at the National University of Singapore, Cardinal AI Lab, UC Berkeley, and HKUST explains why this happens. Their paper, titled "Route Anything: A Study of Model and Policy Routing for General-Purpose AI Systems" (arXiv:2611.05275), evaluates routing across 25 benchmarks, spanning 374,401 scored runs across 452 candidate model configurations.
Their central finding dismantles the standard way engineers build model routers: training a router to predict winners penalizes the very capabilities you want to preserve.
The winner-label trap
The flaw in standard model routing lies in how training targets are constructed.
In multi-model systems, tasks are rarely exclusive. On a large fraction of typical inputs, several different candidate models can solve the problem correctly. If Model A, Model B, and Model C all succeed, your logging pipeline needs a rule to pick a single winner label for training.
Most pipelines use an operational tie-breaker, such as execution latency or token consumption. If Model A finished in 1.8 seconds while Model B took 4.2 seconds, Model A is labeled the winner.
The router paper ran this comparison on robotic manipulation tasks where multiple control policies operated across thousands of scenes. A latency tie-break decided 94 percent of the training labels. Out of 31,408 successful training runs, 24,556 successful trajectories were turned into negative targets.
By training a classifier to predict a single winner, the router was penalized whenever it predicted a capable policy that took slightly longer to finish. The classifier learned to prioritize fast, aggressive candidates that failed when the scene became difficult. When evaluated on test scenes, the winner-label classifier achieved only 42.8 percent task success.
When the researchers replaced the winner classifier with per-candidate outcome regression, predicting the expected success rate of each candidate independently, task success jumped to 71.6 percent on the identical training scenes and architecture.
When you predict outcomes, a candidate that succeeds reliably remains a positive signal regardless of whether another model solved the task three hundred milliseconds faster.
The benchmark baseline illusion
Another persistent problem in the model routing literature is the claim that small routing models easily outperform frontier models on general reasoning.
The paper uncovers an evaluation artifact that inflates router gains across published papers. In many benchmark evaluations, a candidate model is admitted to the fixed baseline only if it has a recorded score on every single test input. If a frontier model like Opus missed eleven out of 1,122 evaluation items due to transient network timeouts or provider rate limits, the baseline drops that model entirely and falls back to a weaker candidate.
On an 11-benchmark evaluation suite, this missing-data rule caused the pooled baseline score to drop from 68.30 to 65.52. That drop turned a 0.40 point deficit for the router into an apparent 2.38 point gain.
When the researchers evaluated candidates on matched test inputs, giving every model an identical fallback for missing items, the gap closed. Across 16 general benchmarks spanning text, vision, audio, and code, learned routers beat the single best fixed candidate by 0.3 points (79.98 versus 79.68).
If your goal is to beat the accuracy of the best frontier model on text reasoning, an upstream prompt router will rarely provide a significant edge.
Where routing actually pays off
The study demonstrates that routing delivers massive returns when applied to cost efficiency and cross-environment adaptation rather than benchmark point-chasing.
Across four production-style testbeds, dynamic routing improved compute and token efficiency by an average factor of 8.82 over calling a reference frontier model. When the router was calibrated to maintain a strict quality floor, it successfully routed straightforward queries to cheap models while reserving expensive models for high-variance inputs.
A second major gain appears when candidate models have complementary physical or environmental strengths. In robotic manipulation tests across clean and randomized environments, the best single policy achieved 70.45 percent success. A scene-aware router directed clean scenes to one policy (PatchWAM) and cluttered, randomized scenes to another (ME-Dex). The combined system reached 71.55 percent success while reducing policy execution time by 24 percent.
The efficiency gain was real because the router was not trying to guess whether one text model understood a pun better than another. It was dispatching tasks based on structural scene conditions where different architectures had distinct operating boundaries.
The retry fallacy
Many agent frameworks combine routing with fallback retries. If the primary routed model fails to complete an agent step or generate valid JSON, the gateway automatically retries the task using a larger frontier model.
The paper examined 42 ordered model pairs to measure how models perform when called as secondary fallbacks. In every single pair, the secondary model succeeded far less often on retried inputs than its normal baseline score suggested. The median success ratio on retries was 0.49.
When an initial model fails on an input, that failure acts as an informational filter. The remaining tasks are not an average sample from your production distribution; they are the pathological edge cases, ambiguous instructions, and contradictory tool schemas.
If your frontier model has an 80 percent success rate on raw traffic, you should expect it to succeed on less than 40 percent of the requests that failed on your primary model. Designing gateway timeouts and budgets around raw baseline accuracy guarantees cascading latency spikes.
Practical rules for gateway routing
If you are building an AI gateway or an agent dispatcher, the data from the Route Anything study suggests four engineering adjustments:
First, stop training winner classifiers. Train per-model regressors that estimate the probability of success or expected task score for each candidate independently. Select the cheapest candidate whose expected score meets your application threshold.
Second, test routing logic against a strictly matched baseline. Run both your router and your candidate models on the exact same set of historical requests, with identical handling for missing items. If a router paper does not guarantee matched test sets, assume the headline gain is an artifact of dropped baseline candidates.
Third, route for cost reduction and domain handoffs, not for frontier text accuracy. Use routing to offload high-volume boilerplate tasks to compact models, or to switch between specialized policies like document parsers and code interpreters. Expecting a small classifier to consistently pick a smarter text model than GPT-4 or Claude on open-ended reasoning is a losing bet.
Fourth, discount your retry assumptions by half. When sizing execution budgets for secondary fallbacks, calculate costs assuming the fallback model will fail twice as often as its public benchmark suggests.
That keeps the gateway fast, prevents brittle candidate loops, and protects the token budget when things go sideways.
Top comments (5)
Good write up. I build GPTree, which has its own auto router that we built. Three things I’d add:
The retry finding is about bad answers, not outages. The paper replayed quality scores and doesn’t appear to cover timeouts or rate limits. If a provider is down, the request may be perfectly easy, so I wouldn’t assume the backup model does half as well.
The big winner label result (42.8% vs 71.6%) comes from robotics, and the paper doesn’t test the two approaches head to head outside it. I still think the logic holds though. We have a version of this ourselves that we see when a user branches a conversation, and Auto uses Claude on Branch A and GPT on Branch B. Both answers are good but the user happens to keep typing in Branch A, perhaps because that’s the idea they wanted to pursue, not because the GPT answer was bad. GPTree treats that as Claude winning and GPT losing, and lowers GPT’s learned score. That makes Auto a little less likely to pick GPT for that type of task later, even though nothing was wrong with its answer. It just wasn’t the branch the user continued.
The part I found most useful was the part about quality thresholds. A router can save money by using a cheap model whenever it expects no drop in quality but to decide when that’s safe, you need a rule from past tasks, such as “use the cheap model for short factual questions”. The rule can look safe on those tasks partly by luck, or because it was shaped to fit them. The test is whether it still holds on tasks it hasn’t seen. It’s like writing a practice exam and then taking that same exam. A good score doesn’t prove you know the subject. You need a test you haven’t seen for that. In GPTree, when we tune Auto we test routing changes on stored routing records, which contain no message text before trusting it.
The distinction between an outage and a bad answer matters a lot. When a provider returns a 429 or drops a connection, the underlying prompt difficulty has not changed. Assuming the fallback model will struggle at the same rate penalizes the run for basic network noise.
The branch continuation telemetry you described is a clean example of the unobserved counterfactual problem. A user picks a branch to follow a specific idea. If the router marks the untouched branch as a failure, it depresses scores on models that gave completely valid answers.
Testing threshold rules against stored routing records without message text is a solid sanity check. It catches policy overfitting without leaking prompt tokens into the routing model.
The winner-label trap explains something I keep hitting from the other side. Once you swap the classifier for per-candidate regressors, the target stops being a label and becomes an expected score, and that score is itself one sample. I sent the same scoring prompt six times in a row today at temperature 0 and got 85, 85, 90, 85, 85, 90. Same input, same model, nothing changed in between.
If each item in your training set is judged once, that five-point wobble goes in as if it were signal. It is also why a 0.3-point router gain can look real: one run per item. Averaging two or three judgements per item is the cheapest fix I know, and it is still cheaper than the frontier calls the router is supposed to replace.
That mechanism is testable from the outside, so I ran it: same question, same model, temperature 0, ten calls, and then ten more with the judge told to emit the bare integer and nothing else. Both arms gave only two values, 85 and 90, and both arms kept a 5-point spread. Forcing the number did not narrow it.
The part I did not expect is the token counts. Ten calls per arm, and the completion token count was different in all ten, in both arms, even when the visible answer was the same two characters and finish_reason was stop. So the divergence is not in how the score gets bucketed downstream, and it is not in the judge's choice of wording either, since there is no wording to choose. Something on the generation path before the final digit differs on every run.
That changes which fix is worth paying for. Median of three passes still helps, because it smooths whatever that path produces. But the cheap version of it, telling the judge to answer with a number only, buys nothing on its own. If the flips come from batching, the way to shrink them is fewer decisions per pass, not fewer words after them.
Boundary: one prompt, one model, n=10 per arm. I am not claiming your batch explanation is wrong, only that a bare-number judge is not a workaround for it.
The five-point wobble at temperature zero usually comes down to batching non-determinism on the provider side. Floating-point reduction order shifts across different batch sizes, so a tied logit flips and cascades into a different score bucket.
When a router trains on single-pass targets, it ends up fitting those batch-scheduler artifacts instead of actual routing boundaries. Averaging three judge passes or taking the median is clean because the router stops spending parameters memorizing which prompt happened to catch a lucky batch.