Intent Routing with Small LLMs: Fast, Cheap and Right

Intent routing with small LLMs works when the model's job is narrow: read a message and return a small JSON object naming the action or tool and its arguments, constrained by a strict schema where every field is required and nullable. Put rules in front of it for obvious inputs, re-extract critical arguments (ticket numbers, dates, names) in code, use a short timeout with failover, and gate every change on an evaluation set. Done this way, a small model routes as accurately as a large one, faster and cheaper.

What routing is

A chat assistant that can do more than talk needs to decide, for each message, what kind of thing it is:

• A how-to question ("how do I add a teammate?") → answer from the product guide.
• A data question ("what's waiting on me today?") → call a read tool with the user's permissions.
• An action ("make a task for Jordan due Friday") → propose it on a confirmation card.
• Small talk → a short reply.

The router makes that decision and extracts the arguments. Everything downstream depends on it, so its accuracy and speed set the ceiling for the whole assistant.

Why a small model

Routing is classification plus extraction — a job small models do well, especially with structured output:

• Latency. In LetRelay's measurements, routing through Gemini Flash took 4–7 seconds; through a small model on Groq, well under a second once warm (0.5–1.3 s). Routing happens before the answer starts, so every second is felt.
• Cost and limits. On free tiers, tokens per minute is capacity. A small model's quota is separate from the one answering questions, so routing doesn't compete with answers.
• Accuracy. For a well-defined set of actions, a small model with a strict schema matched larger models on our evaluation.

The schema: flat, strict, all fields required

The single most important decision. LetRelay's first router used a nested schema with optional arguments. Reviewing 24 real calls showed the model picked the right action in 23 — but often returned no arguments: {"action":"tool","tool":"get_request"} without the ticket number. Nothing failed loudly; answers were quietly wrong.

The fix was a flat schema where every field is required and nullable when unknown:

{
  "action": "tool | write | howto | chat",
  "tool": "get_request | my_day | search | … | null",
  "tool2": "… | null",
  "ref": "RLY-142 | null",
  "query": "free text | null",
  "when": "friday | null",
  "person": "Jordan | null"
}

A model can omit an optional field; it can't omit a required one. It must fill it or write null. Combined with strict structured output (Groq's strict mode on supported models guarantees schema conformance, though not with streaming — which routing doesn't need), missing-argument bugs disappeared.

Rules before the model

Many messages don't need a model at all:

• A lone ticket reference → look up that request.
• Greetings and thanks → a fixed reply.
• The exact text of a suggestion button → its known intent.
• "Where is settings?" → the route from the feature registry.

LetRelay's quick-route layer handles these deterministically. It saves tokens and latency on common inputs and makes wrong routes impossible for them.

Re-extract critical arguments in code

Even with a strict schema, don't trust the model with values that must be exact:

• Ticket references — a regular expression finds RLY-\d+ in the message; if the model's ref disagrees, the regex wins.
• Dates and times — "friday", "tomorrow 3pm", "for 2 days" are resolved by a deterministic parser in the user's timezone, never by the model. See timezone bugs in web apps.
• People — names resolve against the user's organization; ambiguous names become a pick-list.
• Statuses and priorities — matched against known values.

The model decides what kind of request it is; code decides the exact values. Where the model returns something generic ("my stuff") for a search query, a guard replaces it with words from the original message.

Compound questions

"What's on my plate and is anyone on leave tomorrow?" needs two tools. Allowing the router to return up to two tools (tool and tool2) handles most compound questions with one routing call; the tools run in parallel with no extra model call. We capped it at two: open-ended multi-step agent loops cost another prompt per step and rarely help for workplace questions.

Timeouts and failover

Routing gets a short timeout — LetRelay uses 5 seconds — and a chain of models. When one model times out or returns HTTP 429, it's skipped for a cooldown period and the next model is tried. We learned why the timeout matters from a measurement: a hung free model once pushed a single routing call to 19–21 seconds (two attempts of about 9 seconds each) before failing over. The gateway is described in a free AI gateway with Gemini and Groq failover.

When every model fails, the assistant still answers — from the product guide or with a clear note that it can't look things up or take actions right now — rather than guessing. See the case for AI that degrades gracefully.

Multilingual input

Real users mix languages and scripts. LetRelay's users write in English, Nepali and Hindi — often romanized ("bholi ko meeting cancel gara"). The router prompt includes examples in those forms, and the evaluation set includes them. Date words in romanized Nepali ("bholi" for tomorrow) are handled by the deterministic parser, not left to the model.

Evaluate every change

LetRelay's router evaluation has 51 cases: tools, arguments, actions, compound questions and romanized Nepali. The rule is 51/51 before a routing change ships. Each case checks the action, the tool(s) and the key arguments exactly. Running it costs tokens — about 3,100 tokens per case, so roughly 155,000 for the set, most of a day's free budget on one model — so it runs on prompt or model changes, paced to stay under per-minute limits. The method is in how to evaluate an LLM feature with a golden set.

A checklist

• Flat schema, every field required and nullable
• Strict structured output where the provider supports it
• Rules for obvious inputs before the model
• Critical values re-extracted in code
• At most two tools per message
• A short timeout, cooldowns and failover
• A non-AI fallback
• An evaluation set that gates changes

FAQ

Can a small LLM do intent routing well?

Yes, for a well-defined set of actions. With a strict schema, rules for obvious inputs and code that re-extracts exact values, small models route accurately and much faster than large ones.

Why make every schema field required?

Models tend to omit optional fields. Making every field required — and nullable when unknown — forces the model to fill or explicitly null each one, which stopped missing-argument bugs in our router.

Should the model parse dates?

No. Let the model identify the date phrase and resolve it with a deterministic, tested parser in the user's timezone.

How long should a routing call be allowed to take?

A few seconds at most — LetRelay uses a 5-second timeout — then fail over to another model.

Sources

• Groq — Structured outputs
• Google — Gemini structured output
• Groq — Rate limits
• Anthropic — Building effective agents
• Nielsen Norman Group — Response times: the 3 important limits