How to Reduce LLM Token Usage Without Worse Answers

To reduce LLM token usage without worse answers, cut in this order: skip the model when a rule can handle the input, send less context (fewer, more relevant documents, a compact system prompt), route simple tasks to small models, cache answers to repeated questions that don't depend on private data, use your provider's prompt caching for long shared prefixes, and cap output length. Measure tokens per request and quality on a fixed test set before and after each change, so savings never come from silently worse answers.

Why tokens matter even when they're cheap

Per-token prices keep falling, but tokens still govern three things:

• Cost at volume.
• Rate limits. On free and low tiers, tokens per minute is the real capacity. Groq's free plan documents 8,000 tokens per minute and 200,000 per day for openai/gpt-oss-20b; a 2,000-token prompt means four calls a minute.
• Latency. Longer prompts take longer to process; longer outputs take longer to generate.

In LetRelay's assistant, a routing prompt of about 3,100 tokens meant roughly 65 routed questions a day on one free model. Every token cut was capacity gained.

1. Don't call the model

The cheapest call is the one you don't make. Before any model call, check whether a deterministic rule can answer:

• A lone ticket reference (RLY-142) → look it up.
• A greeting or "thanks" → a fixed reply.
• The exact text of a suggestion button → its known intent.
• "Where is settings?" → the route from your feature registry.

LetRelay's quick-route layer handles these with no model call at all. It also removes any chance of a wrong route for the most common inputs.

2. Send less context

Most prompts carry more than the model needs.

• Fewer retrieved documents. Retrieve ten candidates, send the best three to five. Research on long contexts ("Lost in the Middle") found models use information in the middle of long inputs less well — more context isn't automatically better.
• Compact indexes. A list of every feature with a paragraph each is expensive; a list of names with one line each is often enough for the model to know what exists.
• Strip formatting and boilerplate from retrieved text: navigation, repeated headers, long URLs.
• Trim history. Send the last several turns, and summarize or drop older ones. LetRelay sends the last 12 messages, with earlier action cards reduced to one-line summaries.

Measured: LetRelay's how-to prompt went from 2,200–2,600 tokens to about 1,900 by sending four guide pages instead of six and a compact feature index, with no drop on its answer evaluation.

3. Route to smaller models

Not every step needs the strongest model:

Task	Model size that usually suffices
Intent routing, classification, extraction	Small, fast model with structured output
Short factual answers from provided documents	Small to medium
Long synthesis, nuanced writing	Larger model

Small models are cheaper and faster, and with schema-constrained output they're reliable for structured tasks. LetRelay routes with a small model and only uses larger ones for streamed answers. The routing gateway is described in a free AI gateway with Gemini and Groq failover.

Some reasoning models also let you set how much they "think" (for example a reasoning_effort setting). For routing and short data answers, low effort cut tokens and latency without hurting LetRelay's routing accuracy.

4. Cache answers you've already given

Many questions repeat: "how do I reset my password?", "how do I add a teammate?". If the answer doesn't depend on private data, cache it:

• Key by a normalized form of the question plus a version of your source content, so a docs change invalidates the cache.
• Only cache product help, never answers built from someone's workspace data.
• Expire entries (LetRelay uses 7 days) and store them where only the server can read them.
• Serve cached answers only for fresh questions, not for follow-ups whose meaning depends on the conversation.

The main pitfall is scope: a cache that ever stores an answer containing one person's data can serve it to someone else. Keep caching to answers that are identical for everyone.

5. Use provider prompt caching

Several providers can cache a long, repeated prompt prefix — a system prompt or reference document — so subsequent calls that start with the same prefix are processed more cheaply. Anthropic and Google both document this (Anthropic as prompt caching, Gemini as context caching). To benefit:

• Put the stable part (instructions, reference material) first and the variable part (the user's question) last.
• Keep the stable part byte-identical between calls; a timestamp at the top breaks the cache.
• Check each provider's minimum cacheable length and pricing; small prompts may not qualify.

6. Cap and shape output

Output tokens are usually more expensive than input and slower to produce.

• Set a sensible max_tokens per use case.
• Ask for the format you need: "three bullet points", "one sentence", a JSON object with specific fields.
• For structured tasks, strict schemas prevent the model from adding prose around the JSON.

7. Don't retry blindly

Retries double token use. On a rate limit (HTTP 429), fail over to another model rather than retrying the same one; on invalid output, validate and fall back rather than asking again and again. For interactive use, zero retries per model with a failover chain is usually right.

Measure before and after

Track per use case: calls per day, average input tokens, average output tokens, and the share served without a model (rules and cache). Pair every change with your evaluation set — how to evaluate an LLM feature with a golden set — so a token saving that hurts quality is caught. Count tokens with the provider's own tokenizer or token-counting endpoint; different models tokenize differently.

A quick audit

• Which inputs could be answered by a rule? Route them before the model.
• How many documents go into each prompt? Could it be fewer?
• Is the system prompt the smallest version that passes your evals?
• Are small models used for routing and classification?
• Are repeated non-private answers cached?
• Is the stable prompt prefix first, for provider caching?
• Are output lengths capped?
• Are retries replaced with failover?

FAQ

What's the fastest way to reduce LLM token usage?

Skip the model for inputs a rule can handle, and send fewer, more relevant documents in each prompt. Both usually cut tokens substantially with no loss in quality.

Does sending more context improve answers?

Not necessarily. Beyond the relevant passages, extra context adds cost and latency, and research shows models can use information in the middle of long contexts less reliably.

Is caching LLM responses safe?

For generic product help, yes, with versioned keys and expiry. Never cache answers built from a specific user's private data and serve them to others.

What is prompt caching?

A provider feature that caches a repeated prompt prefix — such as a long system prompt — so later calls starting with the same prefix cost less and run faster.

Sources

• Anthropic — Prompt caching
• Google — Gemini context caching
• Google — Gemini token counting
• Groq — Rate limits
• Liu et al. — Lost in the Middle