The Case for AI That Degrades Gracefully

AI that degrades gracefully is AI your product doesn't depend on: when the model is slow, rate-limited, down or wrong, the core task still completes and the user is told plainly what's unavailable. In practice that means saving the user's work before calling a model, letting AI suggest rather than decide, stopping calls to a failing provider (a circuit breaker), and having a non-AI answer for every AI feature. Build the fallback path first; the AI is the enhancement.

AI fails more often than the rest of your stack

A database query fails rarely. A call to a language model fails in more ways, more often:

• Rate limits. Free and paid tiers both cap requests and tokens per minute and per day. Groq's free tier, for example, documents 8,000 tokens per minute and 200,000 per day for openai/gpt-oss-20b, returning HTTP 429 beyond that.
• Latency spikes and hangs. A model can respond in half a second one minute and hang the next. We measured a free model that hung long enough to push a single routing call to 19–21 seconds before our gateway learned to skip it.
• Outages of a provider or region.
• Quota exhaustion. Gemini's daily free request quotas reset at midnight Pacific time; if you exhaust them at 10 am, that's most of a day.
• Wrong answers. Confidently wrong, well-formatted, plausible.

Designing as if any of these is rare is designing for a demo.

Principle 1: save first, enrich later

The most important rule: the user's action must not wait on the model. In LetRelay's request desk, a submitted request is inserted into the database, then classified. If classification fails, the request is already safe in the human triage queue. A scheduled job retries classification later if the AI comes back.

Compare the fragile version: call the model to classify, then save with the category. Now an AI outage is an intake outage. Every AI feature should be checked for this ordering bug.

Principle 2: suggest, don't decide

AI should produce suggestions a human (or a deterministic rule) accepts:

• A suggested category, applied automatically only above a confidence threshold — LetRelay defaults to 0.8 — and otherwise shown as a hint. The mechanics are in AI request classification with structured output.
• A drafted reply that an agent edits and sends.
• A proposed action — "create this task for Jordan, due Friday" — shown as a card with editable fields that the user confirms. LetRelay's assistant never performs a write without that confirmation, and the confirmed action goes through the same server code and database permissions as the button on the screen.

Suggestions fail softly: a wrong suggestion is ignored; a wrong decision has to be undone.

Principle 3: fail fast and stop calling a failing dependency

A slow failure is worse than a fast one: the user waits, then fails anyway. Three mechanisms:

• Short timeouts per use case. LetRelay's routing call times out at 5 seconds.
• Cooldowns (a simple circuit breaker). After a 429, skip that model for 60 seconds; after a timeout or 5xx, skip it for 30. Martin Fowler's description of the circuit breaker pattern captures the idea: stop calling something that's failing, and check again later, instead of letting every request pay for the failure.
• Failover to another model or provider rather than retrying the same one. Details in a free AI gateway with Gemini and Groq failover.

For a whole organization, an admin-controlled switch and an automatic "pause AI for N minutes" breaker let you turn AI off without a deploy when a provider misbehaves.

Principle 4: every AI feature has a non-AI answer

List your AI features and write the fallback beside each:

Feature	With AI	Without AI
Request classification	Suggested category + priority	Unclassified → human triage
Similar requests	Vector search over embeddings	Keyword search
Assistant: "how do I…?"	Model writes a grounded answer	The best-matching page of the built-in guide, with steps and a link
Assistant: "what's on my plate?"	Model summarizes tool results	The same tool results, formatted by a template
Assistant: repeated how-to questions	—	Cached earlier answer (product help only, never organization data)
Drafted replies	Draft appears	Templates

LetRelay tests this literally: an end-to-end run disables every model and checks that nine representative assistant conversations still get useful answers. If a fallback isn't tested, assume it's broken.

Principle 5: skip the model when a rule is certain

Some inputs don't need a model at all. A message that's just a ticket reference (RLY-42), a greeting, the fixed text of a suggestion button, or "where is settings" can be handled by deterministic code. That saves quota, removes latency and makes wrong answers impossible for those cases. Reach for the model when the input is genuinely open-ended.

Principle 6: be honest in the interface

When AI is unavailable, say so — specifically and early:

• "Suggestions are paused right now; your request was saved and the IT team will triage it."
• "I can't look things up or make changes right now, but here's the guide page for that."

We found this matters in a subtle way. When routing failed, LetRelay's assistant sometimes told users "I can't create tasks for you" — in organizations where it could. The model was guessing about its own abilities. The fix was deterministic: when routing fails, the app prepends a fixed sentence saying it can't take actions right now, before any model text. Never let the model improvise statements about what the product can do.

Principle 7: ground the model, and let it say "I don't know"

Wrong answers are the hardest failure because they look like success. Reduce them by:

• Retrieval grounding. Give the model only the facts it may use — guide pages, tool results — and tell it to answer only from them. Retrieval-augmented generation (Lewis et al., 2020) is the standard approach.
• Scoped knowledge. LetRelay's how-to answerer is told it cannot see workspace data; before that rule, it invented a colleague's availability from nothing.
• Evaluation sets. Keep a set of real questions with required and forbidden content ("mentions the Position field", "doesn't mention a pencil icon") and run it on every prompt or model change.

The business case

Graceful degradation isn't only about reliability. It also makes AI affordable: when caching, rules and fallbacks handle part of the traffic, free or cheap tiers cover more users. And it makes AI trustworthy: users who have seen the product behave sensibly when AI is down are more willing to rely on it when it's up. On an internal request desk, the requests people file are the product; the AI is how those requests get triaged faster.

A checklist before shipping any AI feature

• The user's data is saved before any model call
• The model's output is a suggestion or a confirmable proposal
• Timeouts are short; failing models are skipped for a while
• There's a tested non-AI path
• The UI says clearly when AI is unavailable
• The model is grounded in data it's allowed to use
• An evaluation set runs on every change
• Per-user and per-organization limits protect the shared quota

FAQ

What does "degrade gracefully" mean for AI features?

The product keeps doing its core job with reduced capability when the AI fails, instead of breaking. Users may lose suggestions or summaries, but they don't lose their work or get stuck.

Is a circuit breaker necessary for LLM calls?

Some form of it, yes. Even a simple cooldown — skip a model for a minute after a 429 or timeout — prevents every request from waiting on a dependency that's already failing.

Should AI actions require user confirmation?

For anything that changes data, yes. Show the proposed action with editable fields and execute it only when the user confirms, through the same permissions as the regular UI.

How do you test AI fallbacks?

Disable the AI (remove keys or force errors) in an end-to-end test and assert that each feature still completes its core task. Run it regularly, not once.

Sources

• Martin Fowler — CircuitBreaker
• Google SRE Book — Addressing cascading failures
• AWS Builders' Library — Timeouts, retries and backoff with jitter
• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Groq — Rate limits
• Anthropic — Building effective agents