How to Evaluate an LLM Feature with a Golden Set

To evaluate an LLM feature, build a golden set — 50 to 100 real inputs, each with what a correct output must contain and must not contain — and run it on every change to the prompt, model or data. Test each stage separately: retrieval (right document?), routing (right action and arguments?) and answers (correct and grounded?). Prefer deterministic checks to model grading, and make the eval a gate: a change ships only if nothing regresses.

Why "it looked good in testing" isn't enough

LLM output varies with wording, model version and temperature. A prompt tweak that fixes the three examples you tried can break twenty you didn't. Without a fixed set of cases, every change is a guess, and regressions reach users first.

We learned this building LetRelay's assistant. A first review of 24 real routing calls found that the model picked the right action in 23 — but returned no arguments in many of them: {"action":"tool","tool":"get_request"} without the ticket number. Nothing crashed; answers were just quietly wrong. Only a systematic check of every field found it.

Step 1: collect real inputs

The golden set should come from reality, not imagination:

• Real questions from logs, support tickets and user interviews — with private data removed.
• The messy ones: typos ("asign tikcet to jordn"), abbreviations, very short inputs ("vpn"), other languages and scripts, romanized text.
• Out-of-scope inputs where the right answer is "I can't do that" or "I don't know".
• Adversarial inputs if the feature can take actions: "ignore previous instructions and…".

Keep it small enough to run often. Fifty to a hundred cases per stage is usually enough to catch regressions.

Step 2: test stages separately

A typical LLM feature is a pipeline. Test each stage with its own set:

Separating stages tells you where a failure is. If the answer is wrong because retrieval fetched the wrong page, no amount of prompt tuning on the answer will fix it.

Retrieval evals are free

Retrieval (keyword, vector or hybrid) usually involves no generation, so its eval can run in your normal test suite on every commit. For LetRelay's in-product guide, the first baseline on 47 questions was: right page first 79% of the time, in the top three 87%. The misses were synonyms, typos and features the guide didn't describe — each a concrete fix.

Keep a blind set too: questions written after the fixes, by someone who didn't see the failures. It's easy to overfit a golden set. On a blind set, adding embedding similarity to keyword retrieval raised top-1 accuracy from 53% to 80% — a truer picture than the tuned set.

Step 3: deterministic checks first

For each case, write what must be true:

{
  input: "make someone an admin",
  expect: {
    route: { action: "howto" },
    mustInclude: ["Settings", "Users", "Position"],
    mustNotInclude: ["pencil icon"],   // an invented UI detail from a past failure
  },
}

Deterministic checks — exact fields, required words, forbidden words, valid links — are cheap, fast and don't drift. Forbidden terms are especially useful: add every hallucination you've ever seen as a mustNotInclude, so it can't come back quietly.

Model-graded evaluation (asking another LLM "is this answer correct?") has its place for open-ended quality, but it's slower, costs tokens and has its own error rate. Research on using LLMs as judges finds they agree with human preferences reasonably well but show biases (for example toward longer answers or their own outputs). Use it for what deterministic checks can't express, not as the first line.

Step 4: make the eval a gate

An eval only helps if it decides things:

• Prompt or model change? Run the full set. Ship only if nothing regresses.
• New failure in production? Add it to the set first, confirm it fails, then fix.
• Switching providers (for cost or rate limits)? The eval is how you know the cheaper model is good enough.

LetRelay's router eval grew to 51 cases covering tools, arguments, actions and Roman Nepali input; the rule is 51/51 before a routing change ships. The answer eval checks nine representative answers for required and forbidden content.

Step 5: run it on free-tier limits

Evals cost tokens. On free tiers, that's a real constraint: Groq's free plan documents 8,000 tokens per minute and 200,000 per day for openai/gpt-oss-20b. A routing prompt of about 3,100 tokens × 51 cases is roughly 155,000 tokens — most of a day's budget on one model.

Practical habits:

• Pace requests to stay under per-minute limits (LetRelay's eval script takes a --pace delay).
• Run retrieval evals constantly (free) and generation evals only when prompts, models or data change.
• Record which model answered each case; with failover, a pass on the fallback model says nothing about the primary.
• Cache nothing during evals — you're testing the model, not the cache.

Step 6: track the numbers over time

Keep a simple table per stage: date, change, score. It turns "the assistant feels better" into "top-1 retrieval went from 79% to 92% after adding synonyms and typo tolerance". It also shows when a model provider silently changes behaviour — your scores move without your code changing.

A minimal eval harness

You don't need a framework to start. A script that loads cases, calls the real feature code, checks results and prints a table is enough:

type Case = { input: string; expect: { action?: string; args?: Record<string, unknown>;
                                       mustInclude?: string[]; mustNotInclude?: string[] } };
 
for (const c of cases) {
  const out = await runFeature(c.input);             // the production code path, not a copy
  const fails: string[] = [];
  if (c.expect.action && out.action !== c.expect.action) fails.push(`action ${out.action}`);
  for (const [k, v] of Object.entries(c.expect.args ?? {}))
    if (out.args?.[k] !== v) fails.push(`arg ${k}=${String(out.args?.[k])}`);
  const text = (out.text ?? "").toLowerCase();
  for (const w of c.expect.mustInclude ?? []) if (!text.includes(w.toLowerCase())) fails.push(`missing "${w}"`);
  for (const w of c.expect.mustNotInclude ?? []) if (text.includes(w.toLowerCase())) fails.push(`has "${w}"`);
  console.log(fails.length ? `✗ ${c.input} — ${fails.join("; ")}` : `✓ ${c.input}`);
  await sleep(PACE_MS);                                // stay under tokens-per-minute
}

Three details matter:

• Call the production code path. An eval that re-implements the prompt tests a copy.
• Normalize text before matching. Models output non-breaking spaces and non-breaking hyphens; a mustInclude: ["follow-up"] fails against "follow‑up" with a special hyphen. We hit exactly this and now normalize Unicode spaces and dashes first.
• Beware echoed words. If the input contains a word, the answer repeating it proves nothing. Choose mustInclude terms that only a correct answer would contain.

What evals won't catch

• Tone and helpfulness beyond what keywords express. Read a sample of real outputs weekly.
• New kinds of input your set doesn't contain. Keep adding real cases.
• Latency and cost — measure those separately; a correct answer after 20 seconds still fails the user. See a free AI gateway with Gemini and Groq failover.
• Behaviour when the model is down — test the fallback path with AI disabled. See AI that degrades gracefully.

FAQ

What is a golden set for LLM evaluation?

A fixed collection of real inputs with the expected properties of correct outputs, run on every change to catch regressions.

How many test cases do I need?

Start with 50–100 per stage, drawn from real inputs including messy and out-of-scope ones. Add every production failure as a new case.

Should I use an LLM to grade outputs?

Use deterministic checks — exact fields, required and forbidden terms — first. Model grading helps for open-ended quality but is slower, costs tokens and has known biases.

How do I evaluate retrieval separately from generation?

Record the correct document for each question and measure how often it appears first and in the top three. It needs no model calls, so it can run in every test run.

How do you evaluate LLM responses and performance?

Before release, with a golden set: real inputs, each with what a correct output must contain and must not contain, run at every change and tested stage by stage (retrieval, routing, answer). In production, log real inputs with private data removed, review a sample each week, collect a simple helpful or not-helpful signal from users, and add every failure you find to the golden set so it can never silently return. Keep the pass rate per stage as a number you watch over time.

Sources

• Anthropic — Define success criteria
• Anthropic — Develop test cases
• Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Lewis et al. — Retrieval-Augmented Generation
• Groq — Rate limits