Caching LLM Responses Safely: What to Cache and What Never To

Caching LLM responses is safe when the cached answer would be identical for every user — generic product help, public documentation answers — and the cache key includes a version of the source content, so edits invalidate old answers. Never cache answers built from private data, earlier turns or tool results — a hit would serve them to someone else. Normalize questions, expire entries, keep the cache server-only, and treat it as optional: a miss just means a normal model call.

Why cache at all

Repeated questions are common in any assistant: "how do I reset my password?", "how do I invite someone?", "where are the reports?". Each answer costs tokens and seconds. On free AI tiers, it also costs daily capacity — in LetRelay's case, a how-to answer is roughly 2,000–3,000 tokens of a limited daily quota. Serving a repeat question from a cache costs one database read.

A cache also improves consistency: the same question gets the same answer, instead of slightly different phrasing each time.

The danger: cache poisoning and data leaks

Caches serve stored output to whoever sends a matching key. If the stored output contains anything specific to the original asker, the next asker gets it:

• "What's on my plate today?" answered for Priya, then served to Sam.
• An answer that mentions a ticket number or colleague from the first asker's organization.
• An answer that depended on a previous turn ("and what about the second one?") served without that context.
• An answer that was wrong because a model was having a bad day — now wrong for everyone, for as long as the entry lives.

The design has to make these impossible, not unlikely.

Rule 1: cache only what is the same for everyone

Divide your assistant's answers by source:

LetRelay caches exactly one category: answers from the product guide, to the first question of a conversation, with no history. The route decides this before looking up the cache, so a data question can't reach it.

Rule 2: key on the normalized question plus a content version

Two keys matter:

• The question, normalized: lowercased, punctuation removed, whitespace collapsed, typographic apostrophes unified. "How do I invite someone?" and "how do i invite someone" should match. Keep the words; don't stem aggressively, or different questions will collide.
• A version of the source content: a hash of the guide or documentation the answer was generated from. When the docs change, the version changes, and every old answer becomes unreachable — no manual purge needed.

const GUIDE_VERSION = hashOfGuideContent();          // changes whenever the guide changes
 
export function normalizeQuestion(q: string) {
  return q.toLowerCase()
    .replace(/[’‘]/g, "'")
    .replace(/[^\p{L}\p{N}' ]+/gu, " ")               // keep letters and numbers in any script
    .replace(/\s+/g, " ")
    .trim();
}
 
export const cacheKey = (q: string) =>
  sha256(`${GUIDE_VERSION}|${normalizeQuestion(q)}`);

Use Unicode-aware character classes so non-English questions normalize correctly rather than being stripped to nothing.

Rule 3: expire entries

Even with versioned keys, entries should expire — LetRelay uses 7 days. Expiry bounds the damage of a bad answer, keeps the table small, and lets improved prompts or models take effect. A scheduled database job deletes expired rows.

Rule 4: only cache good answers

Don't store:

• Empty or very short outputs (a truncated stream, an error message).
• Answers generated by a fallback path while the primary model was unavailable, if they're lower quality.
• Answers you can't attribute to a specific content version.

Store after the answer completes and before the response closes; on serverless platforms, work after the response may never run.

Rule 5: lock the cache down

The cache table holds text generated by your system — treat it as internal:

• Row-level security on, no policies for client roles: nothing reads it through the public API.
• Server-only access through a narrow function (LetRelay's assistant_cache_get) or the service role.
• Cap stored size so a runaway answer can't bloat storage.

LetRelay's security suite includes a check that the cache table and its function can't be reached by anonymous or signed-in client roles. See Supabase security definer functions for why that check exists.

Rule 6: the cache is optional

Every cache operation should fail soft: a read error is a miss; a write error is ignored. The feature must work identically with the cache empty or unavailable. Test it that way.

A worked example

Three messages arrive in LetRelay's assistant on the same day:

1. Priya, first message of a new chat: "How do I invite someone?" — a product how-to, no history. The route checks the cache: a miss. The model answers from the guide; the answer is stored under the key for "how do i invite someone" plus the current guide version.
2. Sam, first message: "how do I invite someone" — normalizes to the same text. A hit: the stored answer is served in milliseconds with no model call.
3. Sam again, in the same chat: "and can I make them an admin straight away?" — a follow-up whose meaning depends on the previous turn. The cache isn't consulted at all; the model answers with the conversation as context.

The next morning, the guide page for inviting people is updated with a new step. Its content hash changes, so the guide version changes, and the next "how do I invite someone?" is a miss — answered fresh from the updated guide. No one had to remember to purge anything.

Exact-match vs semantic caching

Some systems cache by meaning: embed the question and reuse an answer if a previous question is similar enough. It raises hit rates and adds risk: "how do I add an admin?" and "how do I remove an admin?" are close in embedding space and need opposite answers. If you try it, use a very high similarity threshold, restrict it to generic content, and evaluate it against pairs of similar-but-different questions. Exact-match on normalized text is the safe default.

Provider prompt caching is different

Some providers offer prompt caching: they cache the processing of a repeated prompt prefix (a long system prompt or document), so later calls with the same prefix are cheaper and faster. The model still generates a fresh answer every time, so there's no cross-user answer leak. It complements response caching rather than replacing it; see how to reduce LLM token usage.

Measuring it

Track hit rate (hits ÷ eligible questions), tokens saved (hits × average tokens per answer) and, importantly, whether users rephrase or complain after cached answers — a sign a stale or poor answer is being served. Include cached answers in your periodic quality review. The evaluation approach is in how to evaluate an LLM feature with a golden set.

FAQ

Is it safe to cache LLM responses?

Yes, for answers that are identical for everyone — like generic product help — with versioned keys, expiry and server-only access. Never cache answers that use a user's private data or conversation history.

What should an LLM cache key include?

A normalized form of the question and a version identifier of the content the answer came from, so content changes invalidate old answers automatically.

How long should cached LLM answers live?

Days rather than months — LetRelay uses seven — so improvements take effect and mistakes don't persist.

What's the difference between response caching and prompt caching?

Response caching stores and reuses a whole generated answer. Prompt caching, offered by some providers, reuses the processing of a repeated prompt prefix while still generating a fresh answer each time.

Sources

• Anthropic — Prompt caching
• Google — Gemini context caching
• Supabase — Row Level Security
• Supabase — pg_cron
• OWASP — Top 10 for LLM Applications