Model Routing

Model routing is the process of selecting which model or provider should handle a given AI request. The routing decision may be based on task type, expected quality, cost, latency, context length, availability, compliance policy, tool support, or user tier.

Model routing can happen in application code, an LLM Gateway, a cloud service, or a model provider. It is increasingly important because production systems often combine frontier reasoning models, cheaper Small Language Models (SLMs), embedding models, image models, and specialized fine-tuned models.

Common routing strategies include:

• static rules by feature or endpoint;
• intent classification;
• complexity estimation;
• fallback on provider errors or rate limits;
• cost-first or latency-first selection;
• quality prediction;
• user-configured model preferences; and
• cascades where a small model handles easy cases and escalates difficult cases.

Routing is not merely load balancing. It changes model behavior and therefore must be evaluated like any other model choice. A cheaper model may be sufficient for classification but inadequate for legal analysis or complex code changes.

For agentic workflows, routing can apply at each step. A planner may use a frontier model, while extraction, formatting, and validation steps use cheaper models or deterministic code.

Routing decisions should be logged with request metadata, selected model, fallback path, latency, cost, and outcome. Without this, regressions are hard to diagnose and token economics cannot be measured accurately.

Amazon Bedrock's intelligent prompt routing documentation describes routing prompts between foundation models to optimize response quality and cost.