Model Routing by Task Type — When Smaller Models Are Good Enough
Splitting tasks across models by complexity cuts AI costs by up to 72 percent.
Most AI products are running an expensive model on tasks that don't need one. A team builds its first AI feature, reaches for the most capable model on the market because it's the safest choice at prototype stage, and ships it. Usage grows. More features get bolted on. Nobody goes back and asks whether the original model choice still makes sense for what the product has become. The model was never the problem. The habit of never revisiting that first choice is. A single conversational turn in most products actually hides several smaller jobs stitched together: figuring out what the user wants, checking the message for anything unsafe, pulling up relevant memory, writing a reply, and shaping that reply to match the product's voice. A team running all five of those jobs through one expensive model is paying frontier prices for four tasks that a much cheaper model handles just as well.
The price gap between frontier and small models makes routing the highest-leverage cost lever available
Once the task mix is split out like that, the pricing gap between model tiers turns from a footnote into the main event. As of July 2026, GPT-5.5 costs $5.00 per million input tokens against $0.20 for GPT-5.4 nano, a 96% drop. Claude Opus 4.8 runs $5.00 per million input tokens versus $1.00 for Claude Haiku 4.5, an 80% drop. Gemini 3.1 Pro costs $2.00 per million input tokens against $0.25 for Gemini 3.1 Flash-Lite, an 87% drop. Output tokens cost four to eight times more than input tokens across these APIs, so moving a verbose task down a tier cuts cost on both the input and the output side at once. Stacking the cheapest usable model against the most capable one widens the overall spread considerably beyond any single pair's gap. Inworld Research's Consumer AI Stack 2026 report names model routing as the highest-leverage cost optimization available to consumer AI applications, and the math above explains why: when the price gap is this wide, even modest shifts in traffic produce outsized savings.
Task types suited to smaller models
Classification work, like figuring out intent, sentiment, or tags, belongs on a smaller model. So does pulling structured fields or entities out of text, converting between formats, deduplicating records, answering short factual questions when the answer is already sitting in the provided context, generating from a template, filtering for safety, and producing short acknowledgments. Each of these has a narrow, checkable output, which is what smaller models are good at.
Frontier models earn their cost on reasoning that spans many steps, planning that has to adapt as it goes, instructions that are ambiguous enough to need judgment, writing where a mistake carries real cost, and code changes that involve real design decisions.
One wrinkle catches teams off guard: reasoning models bill their internal "thinking" tokens as output tokens, so pointing a reasoning model at a simple task doesn't just waste capability, it inflates the bill disproportionately. Small models tend to do better with tighter instructions and a single clear example, a point that matters again once routing decisions need testing.
Traffic that can move to smaller models, and the resulting savings
The task breakdown only matters if it reflects how traffic actually distributes in a live product, and in practice it skews heavily toward the simple end. Teams routinely route 60% to 80% of their traffic to smaller models without any measurable drop in quality on those tasks. A routing layer that sends 60% to 70% of traffic to small models and keeps 30% to 40% on frontier models tends to cut cost per query by roughly 37% to 46%. Pushing the split further, to 80% small and 20% frontier, moves the reduction toward 72%. These ranges depend on the specific mix of tasks a product runs, so the exact number any team lands on will vary, but the direction holds across reports.
The RouteLLM paper, presented at ICLR 2025, backs this up from the research side: a router that sent only 14% of queries to the strong model still matched 95% of GPT-4's quality on MT Bench while cutting costs by 85%. At consumer AI scale, Inworld Research estimates that the gap between running a router and not running one reaches hundreds of thousands of dollars a month once daily active users climb high enough. Past that point, routing stops functioning as a cost optimization: it becomes the line between an AI bill the business can sustain and one that can sink it. How much of that saving a team actually captures comes down to one thing: how accurately the router can tell an easy request from a hard one.
Three router designs, from an afternoon of work to a full ML system
Capturing most of the savings doesn't require a machine learning system. It starts with something much simpler, and teams should build up in complexity only as the simple version runs out of room.
The first design is rule-based routing, and it's the right place to start. When the task type is already known, this is a dictionary lookup that adds no latency and takes an afternoon to write. A rule set of around 20 lines is enough to get a real, working router in production.
The second design is a cascade that escalates on failure, sending every request to the small model first under a strict output contract. Only when that model fails validation, returns a low-confidence result, or gets retried by the user does the request escalate to the frontier model. Run this way, most requests never touch the expensive model. The latency worry that usually comes up here is often overstated: because small models respond faster, average latency across the whole system tends to drop even though the requests that do escalate pay for one extra round trip.
The third design is classifier routing, suited to streams of mixed, unpredictable input where the task type isn't known ahead of time. A cheap, nano-class model sits in front and judges whether each request is simple or hard, routing it accordingly, for a cost that runs to a fraction of a cent per call, small enough that it's usually negligible against the savings it unlocks. A router trained on one pairing of a strong model and a weak model, the RouteLLM authors found, kept its performance even when the underlying strong and weak models were swapped out at test time, a finding that makes all three designs more durable than they might seem. In a market where the model lineup changes monthly, that transfer property is what keeps a router from needing to be rebuilt every time a new model ships.
How to validate a routing decision before shipping it
The real risk in routing is that it fails quietly: cost metrics improve in dashboards the moment a route goes live, while any drop in quality appears days later, in customer tickets, long after the dashboards already looked like a win. Validating a route before shipping it is what keeps that gap from opening up.
The process is short enough to run in an afternoon. Run those same examples through a model one or two tiers below the one currently handling them, using the same prompt. Score the two sets side by side, either with a human reviewer or with an LLM judge, using the frontier model as the evaluator against a rubric written in advance. If the smaller model clears the quality bar, ship the route. A pre-merge eval gate running 50 to 500 cases is the standard mitigation for catching regressions before they reach production, and it's the practice that lets a team bank the savings without taking on the quiet risk described above.
Two details matter past the initial test. If any routed task needs tool calls, confirm that every candidate model supports the function-calling schema the task requires, since gaps between tiers appear there first.
Routing logic belongs in a gateway layer, not in application code
Once a team has a few validated routes, the next question is where that logic should live. Scattering it across application code, team by team, is where routing quietly falls apart. Routing can be built around cost, latency, capability, policy, or semantic intent, or some hybrid of these, and each approach works fine on its own. But these approaches only stay coherent when applied consistently across all of a product's traffic. The moment different teams implement routing with different criteria in their own corners of the codebase, the result is a patchwork nobody can see end to end: finance gets one combined bill with no way to tell which team or model drove it, and engineering has no way to trace a cost spike back to its source.
A gateway is also the natural home for the resilience work that routing depends on: automatic retries with backoff, fallback chains to other providers when one fails, and circuit breakers that isolate a failing provider so its outage doesn't take the whole product down with it. These are infrastructure concerns, not things that belong wired into individual application logic. Production AI workloads need capabilities that traditional API gateways were never built for, including token-based rate limiting, routing across multiple model providers, semantic caching, defenses against prompt injection, and support for the MCP protocol used by AI agents. Centralizing routing in a gateway with one unified API across providers also makes routing decisions reversible: swapping in a better small model for a given task becomes a configuration change instead of a code change spread across however many services touch that task.
What a gateway-first routing setup gives engineering and finance teams
Centralizing routing this way gives engineering and finance something ad-hoc code structurally can't: a single, real-time view of what's being spent and why. LangSmith's LLM Gateway, introduced in 2026, ships with spend limits set as hard caps at the organization, workspace, user, or API key level, returning a 402 response the moment a request would push spend past the cap, alongside real-time cost rollups broken down by workspace and user. The result, in LangSmith's own framing, is no more month-end surprise invoices.
A gateway is also where security and compliance enforcement actually becomes possible. Runtime enforcement can intercept model inputs and outputs as live traffic flows through, blocking, redacting, or flagging anything that violates policy, using fast classifiers and policy checks for PII, prompt injection, jailbreak attempts, toxicity, and brand risk. Writing plain-text prompts containing PII or PHI into an unencrypted log store creates a fresh compliance problem the moment it happens. Real-time guardrails that redact sensitive data at the gateway, or that store cryptographic hashes of prompts alongside separate, access-controlled archives, close that gap. These regulatory requirements already set the floor: SOC 2 requires evidence of change management and access control, HIPAA requires audit controls and six years of retention, GDPR requires lawful processing, data minimization, and the ability to erase data on request, and the EU AI Act requires automatic lifetime logging for high-risk systems under Article 12, with deployers required to retain logs for at least six months under Article 26.
Reliability is the third piece. Vercel's production data shows the scale of what a fallback layer can catch: it rescued more than one trillion tokens a month from errors through April 2026, out of tens of trillions of tokens processed in total across that period. Choosing a managed gateway over self-hosted gateway software moves the burden of running, securing, and patching that infrastructure off the engineering team. The risk of skipping that step became clear in March 2026, when malicious versions of a widely used self-hosted gateway were published to PyPI, a supply-chain incident that managed deployments are built to absorb before it ever reaches a team's own infrastructure.
Putting it together: a routing rollout sequence for a production team
None of this requires a big-bang rewrite. It works best as a sequence, run one task at a time. Start by mapping the full task inventory: list every distinct task the application performs and label each one as classification, extraction, formatting, routine Q&A, or complex reasoning, since this inventory is what every later routing decision depends on. From that list, find the single highest-volume simple task, the one generating the most traffic, and freeze 20 to 50 real examples of it along with the outputs that were already accepted as correct. Run the validation process described earlier against that task: test the smaller model, score it against the frozen examples, and ship the route the moment it clears the quality bar. Before adding any more routes, set up per-route token logging at the gateway layer, so cost attribution is visible in real time at the gateway instead of being reconstructed later from a combined bill. With that instrumentation in place, move to the next task type and repeat the process. Each route validated this way lowers the blended cost a little further and adds data that sharpens the next routing decision, making the whole rollout iterative. Once several routes are live, put spend limits and quality alerts in place: hard caps per team or per key catch runaway costs before they become a surprise invoice, and automated checks for quality regression catch silent degradation before a customer ever notices it. None of this depends on starting with the most sophisticated router. A rule-based router on the first task, built in an afternoon, is enough to begin, with room to grow into a cascade or a classifier only once the task inventory and the traffic actually call for it.
