I want to tell you about the most important layer in any serious AI system you’ve never heard of. It’s not a model. It’s not a prompt. It’s the routing layer — the quiet decision-maker that looks at what you just asked for, decides which model is best equipped to answer it, and sends it there. Sometimes that’s a 7-billion parameter whisper. Sometimes it’s a 400-billion parameter roar. The model that gets the task is rarely the one you’d have guessed.

This is hybrid AI routing, and it is the difference between a system that wastes money on every problem and a system that is, in some sense, thinking about which thinking to use.

Why Static Model Selection Fails

The naive approach is obvious: pick your best model, use it for everything. GPT-4o for emails. Claude Sonnet for analysis. Gemini Ultra for reasoning. Done.

Done, and massively overpaying.

A 7-billion parameter model that costs $0.0002 per thousand tokens will answer “What time is it in Tokyo?” just as correctly as a $2-per-million-tokens frontier model. It will answer it faster, cheaper, and with essentially identical output quality. Using the expensive model for simple questions is not impressive engineering. It is impressive waste.

The harder problem — the interesting problem — is that you cannot simply pre-classify queries by complexity. “Summarize this email” could be a three-sentence email or a forty-page legal document. “Write a function to sort a list” could be a ten-line bubble sort or a分布式 comparison engine. The same surface-level request hides enormous variation in what the model actually needs to do.

So the routing layer needs to be more than a simple keyword matcher. It needs to understand something about the actual computational demands of the task.

What Routing Can Be Based On

Token Length as a Proxy

The simplest useful heuristic is input and output length. A task requiring a 10,000-token response almost certainly needs more model capacity than one requiring 50 tokens. This is not a hard rule — a 50-token answer about quantum chromodynamics might require more reasoning than a 10,000-token description of a beach holiday — but it’s a signal, and signals add up.

Most production routing systems start here. If the input is under 500 tokens and the expected output is under 200 tokens, route to a small fast model. Escalate from there.

Task Type Classification

A step up from length-based routing: classify the task type and map task types to appropriate models.

“Extract the dates from this document” is extraction. “Explain why this code is broken” is reasoning. “Write a haiku about entropy” is creative. “Calculate the compound interest on this account” is calculation.

Each of these has a different capability profile. A small model fine-tuned on extraction tasks will outperform a general-purpose frontier model that has never been optimised for extraction. This is why many production systems run task classifiers — small, fast models that look at the input and say “this is a summarisation task, route to Model X” or “this is a code review task, route to Model Y.”

The classifier doesn’t need to be smart. It needs to be accurate. A 1-billion parameter classifier that correctly categorises tasks 95% of the time costs almost nothing to run and saves enormous amounts on the subsequent inference bills.

Semantic Complexity Estimation

The most sophisticated routing systems estimate the actual cognitive complexity of the task — not just its surface form, but the depth of reasoning it requires.

This is harder. It requires understanding that “list the countries in the EU” is a lookup task (route to small model), while “compare and contrast the economic implications of EU membership for new entrants vs. existing members over the past twenty years” is a synthesis task requiring genuine analytical capability (route to frontier model).

Some systems use a small “router model” that has been trained specifically to predict which model will give the best answer for a given input. The router is shown millions of examples where both a small and large model answered the same query, and trained to predict which answer was better. Over time, it learns that certain patterns — causal connectors, comparative structures, requests for explanations — correlate with tasks that benefit from frontier model reasoning.

This is essentially learning the routing policy from data rather than hardcoding rules.

Cost-Capability Tradeoffs

Here’s the thing that makes routing genuinely interesting: it’s not just about capability matching. It’s about cost-capability tradeoff optimisation.

A frontier model might give a 95th-percentile answer for $2. A small model might give a 80th-percentile answer for $0.001. Is the 15% improvement worth 2000x the cost? For a casual email draft, absolutely not. For a legal document that will be reviewed by lawyers, maybe. For a medical diagnosis suggestion, probably yes.

The routing layer, in sophisticated systems, doesn’t just route to “the model that can do this” — it routes to “the cheapest model that can do this well enough for this context.” That threshold is not fixed. It varies by use case, by user preference, by downstream consequences of errors.

This is why some systems let users or system administrators set quality floors. “For all customer-facing outputs, use frontier models only.” “For internal summarisation, use medium models.” “For code autocomplete, use the cheapest model that passes our benchmark suite.”

The Routing Architecture

In practice, a production routing system looks something like this:

Input arrives. The task goes to a lightweight classifier — a fast, small model or even a rule-based system — that produces a preliminary task type and complexity estimate.

The router evaluates constraints. What is the cost budget for this request? Is there a latency requirement? Are there domain-specific requirements (e.g., this request involves medical terminology, route to a model with medical fine-tuning)?

The routing decision is made. Based on task type, complexity estimate, and constraints, the system selects a model from the available pool. In many systems, this is not a hard choice — it’s a probability distribution. “73% chance that Llama 3.3 70B is sufficient, 25% chance we need Claude Sonnet, 2% chance we need GPT-4o.”

The selected model is called. Simple enough.

Optionally, the output is validated. If the routing system has low confidence, or if the output is flagged by a quality checker (e.g., a factual consistency model), the system may reroute to a higher-capability model and compare. If the second answer is substantially different, that’s a signal that the routing was wrong — which updates the routing model for future similar queries.

This last step — validation and back-routing — is what separates a production system from a proof-of-concept. The routing layer that cannot correct itself is a brittle routing layer.

Dynamic Routing vs. Fixed Routing: Why the Distinction Matters

You might think this is all overengineering. Just use the best model for everything and be done.

But here’s what happens when you actually measure costs at scale: the distribution of queries in any real system is heavily skewed toward the simple end. Most people, most of the time, ask simple questions. If your system routes every query to a frontier model because you don’t want to risk quality degradation, you are paying frontier model prices for 80% of your queries that a $0.0002 model could answer just as well.

At 10,000 queries per day, that difference might be the difference between a $200 daily inference bill and a $20,000 daily inference bill. At 10 million queries per day — which is not unusual for a serious production deployment — it is the difference between viability and bankruptcy.

Dynamic routing, done well, achieves 85-95% of frontier model quality on aggregate tasks while using frontier models for only 10-20% of queries. The rest go to smaller, faster, cheaper models that are genuinely sufficient.

The Hard Cases

I want to be honest about where routing breaks down.

Deceptive simplicity. A query that looks like a simple lookup is actually a reasoning trap. “What’s 2+2?” routed to a small model is fine. “What’s the sum of all integers from 1 to 10^12?” routed to a small model might produce a confidently wrong answer. The routing system that only looks at surface complexity will misroute this.

Domain shift. A routing model trained on general queries will misroute specialist queries. “Write me a function to parse S-expressions” looks like a standard coding task — and a general-purpose router might send it to a small model. But parsing S-expressions correctly requires understanding recursion, edge cases, and character-level handling in ways that benefit enormously from a larger model’s reasoning.

Adversarial inputs. Users — or automated systems — can craft inputs specifically designed to trigger misrouting. Make a simple-looking query that embeds a complex logical trap, and a routing system optimised for surface features will route it to a model that cannot handle the hidden complexity.

Multi-turn conversations. Routing a single message is tractable. Routing a 20-turn conversation, where the complexity of the conversation has built over the turns and the current message references something from twelve turns ago, requires conversation-level routing — not just per-message routing. This is an open problem.

The Future: Learned Routing as Competitive Advantage

Here is my prediction: the companies that win at AI infrastructure at scale will not be the ones with the best models. They will be the ones with the best routing.

A routing system is, at its core, a theory of when to use what. It encodes your understanding of your models, your tasks, your users, and your cost structure. A routing system trained on millions of real queries, updated continuously with real performance data, becomes a strategic asset — a model of your specific operational reality that no competitor can easily replicate.

The model is the engine. The routing layer is the driver. And in any serious deployment, it is the driver that determines whether you arrive somewhere useful.

Next Friday: we continue the deep-dive rotation. The topics are cycling, the archive is building, and somewhere in this system, a router is deciding which model gets to write the next one.