AI Request Classification with Structured Output

AI request classification works reliably when you ask the model for structured output — a JSON object that must match a schema with an enum of your real categories, a priority and a confidence score — then validate it again in your code, apply it automatically only above a confidence threshold, and send everything else to a human. Save the request before calling the model, so a timeout or nonsense answer only means a person triages it. That's how LetRelay runs on free Gemini and Groq keys.

What classification needs to decide

For an internal request desk, the model reads a free-text request ("my laptop keeps disconnecting from the office Wi-Fi since Monday") and returns:

• Category — one of the organization's existing categories, e.g. Network.
• Priority — low, normal, high or urgent.
• Confidence — a number between 0 and 1.
• Optionally a short summary or a suggested first reply.

It doesn't decide the team if the requester already chose one. In LetRelay the requester picks the team; AI refines the category within it, and only routes on its own when a request arrives with no team at all (from a webhook, for example). The trade-offs are in support ticket routing.

Why structured output, not "reply in JSON please"

Asking a model in plain words to "reply with JSON" works most of the time, which is the problem: at volume, "most of the time" means some malformed responses every day — extra prose, a trailing comma, a category spelled differently. Both major free APIs now support schema-constrained output:

• Gemini accepts a response schema and a JSON MIME type, and constrains generation to it.
• Groq supports JSON-schema structured outputs; with strict mode on supported models, responses are guaranteed to match the schema. Strict mode isn't available with streaming, which is fine for classification — it doesn't need to stream.

In our testing this was the difference between occasionally missing fields and never missing fields. LetRelay's assistant router had a real bug where a model returned {"action":"tool","tool":"get_request"} with no arguments; a flat schema with every field required (and nullable when unknown) fixed it, because the model can no longer omit a field — it must fill it or set it to null.

Designing the schema

Keep it flat and small:

const Classification = z.object({
  category: z.enum(categoryNames),          // this organization's real categories
  priority: z.enum(["low", "normal", "high", "urgent"]),
  confidence: z.number().min(0).max(1),
  summary: z.string().max(200).nullable(),
});

Rules that matter:

• Enums from your data. Build the category enum from the organization's actual categories at request time. The model can't invent "Wi-Fi Stuff" if the schema only allows Network, Hardware, Access, Software.
• Every field required, nullable when unknown. Required-and-nullable is more reliable than optional; models drop optional fields.
• Short strings with limits. A summary capped at 200 characters can't become an essay.
• No nesting unless necessary. Deep objects increase failure rates and token use.

Validate again in your own code

Schema-constrained output reduces errors; it doesn't remove your responsibility. Parse the response with the same Zod schema before using it. If parsing fails, treat it exactly like a model outage: log it, and leave the request for human triage. Never "fix up" a bad response with string manipulation.

The prompt

A good classification prompt is short and concrete:

1. One sentence of role: "You classify internal help desk requests."
2. The categories, each with a one-line description and one or two example requests. Descriptions matter more than names: Access — permissions to systems, shared drives or SaaS tools.
3. The priority rubric in plain words: urgent = someone cannot work at all; high = work is blocked but there's a workaround.
4. The request text, clearly delimited.
5. "If unsure, lower your confidence rather than guessing."

Keep temperature low (0–0.2). Classification wants consistency, not creativity.

Confidence thresholds

Model-reported confidence isn't a calibrated probability, but it's a useful ranking signal. LetRelay's default: apply the suggestion automatically at confidence ≥ 0.8, show it as a one-click hint below that, and let each organization adjust the threshold. Track how often agents change an auto-applied category; if more than a small share are corrected, raise the threshold or improve the category descriptions.

Free-tier limits shape the design

On free keys, capacity is a budget, not an afterthought. Groq's documented free limits for openai/gpt-oss-20b are 30 requests per minute, 1,000 per day, 8,000 tokens per minute and 200,000 tokens per day; past them it returns HTTP 429 with a retry-after header. Gemini's free limits are per account, visible in AI Studio, with daily request quotas resetting at midnight Pacific time.

Practical consequences:

• Keep prompts small. At 8,000 tokens per minute, a 2,000-token prompt allows about four calls a minute per model. Trim category descriptions and examples to what actually improves accuracy.
• Fail over between providers. When one model returns 429, try another rather than retrying the same one. See a free AI gateway with Gemini and Groq failover.
• Don't retry interactively. A retry on a rate-limited model makes the user wait and burns quota.
• Batch the backlog. Requests that missed classification can be reprocessed later by a scheduled job, a few at a time.
• Meter per organization. One tenant shouldn't be able to spend the whole shared quota.

The fallback path is the product

Write the no-AI path first. When the model is unavailable:

1. The request is already saved (the AI call happens after the insert).
2. It stays unclassified and appears in the human triage queue.
3. A scheduled job retries classification later if the AI comes back.
4. Nothing on the requester's side changes.

If your intake can fail because the AI failed, the AI is a dependency, not a helper. The case for AI that degrades gracefully goes further.

Measuring accuracy

Build a small labelled set — 50 to 100 real requests with the category a human chose — and run it whenever you change the prompt, model or categories. Report:

• Top-1 accuracy — how often the suggested category is the human's.
• Accuracy above threshold — how often auto-applied suggestions are right. This is the number that matters, since it's what users see.
• Coverage — the share of requests above threshold.

Raising the threshold trades coverage for accuracy. For LetRelay's assistant router we keep a 51-case evaluation and require every case to pass before switching models or prompts — a change that looks better on five examples can be worse on fifty.

Similar requests and suggested answers

Classification pairs well with retrieval: embed the request text, find similar past requests or knowledge-base articles with vector search, and show them to the agent (or to the requester before they submit). LetRelay stores 768-dimension embeddings in Postgres with pgvector and only surfaces matches above a similarity threshold. The setup is in semantic search with Supabase pgvector.

FAQ

Which model is best for classifying help desk tickets?

A small, fast model is usually enough for a well-described category list. Accuracy depends more on clear category descriptions and a labelled test set than on model size. Measure on your own requests before choosing.

What confidence threshold should auto-classification use?

Start high — around 0.8 — and adjust based on how often agents correct auto-applied categories. Below the threshold, show the suggestion as a hint instead of applying it.

What happens if the AI returns an invalid category?

With an enum in the schema it can't; if a response still fails validation, treat it like an outage and leave the request for human triage.

Can I do this on free API tiers?

Yes, for modest volume, if prompts are small, providers fail over on 429s, and the backlog is reprocessed in the background. Plan for the daily limits rather than hoping not to hit them.

Sources

• Google — Gemini structured output
• Groq — Structured outputs
• Groq — Rate limits
• Google — Gemini API rate limits
• OpenAI — Structured outputs guide
• Zod documentation