A Free AI Gateway: Gemini + Groq Failover That Stays Fast

A free AI gateway with failover is one function your app calls — chat(useCase, prompt) — that tries an ordered list of free models across providers (for example Groq, then Gemini, then OpenRouter), skips any model that recently rate-limited or timed out, and gives up quickly instead of retrying. The rules: short timeouts, cooldowns on HTTP 429 and on slow failures, no retries while a user waits, and one model chain per use case. Here's LetRelay's gateway and the measurements behind each rule.

Why one provider isn't enough on free tiers

Free tiers are generous per call and stingy per day. Groq documents, for openai/gpt-oss-20b on the free plan: 30 requests per minute, 1,000 requests per day, 8,000 tokens per minute and 200,000 tokens per day. Gemini's free limits are set per account (visible in AI Studio) and daily request quotas reset at midnight Pacific time. OpenRouter offers free variants of some models with its own limits.

Each limit is fine alone. But a single assistant question in LetRelay can involve a routing call of about 3,100 tokens plus an answer call of about 1,900 — so 8,000 tokens per minute means only a couple of questions per minute per Groq model, and 200,000 tokens per day means roughly 65 routed questions a day before that model is exhausted. Spread across several models and providers, the same free keys go much further.

The shape of the gateway

type Spec = { provider: "groq" | "gemini" | "openrouter"; model: string; strictJson?: boolean };
 
const CHAINS: Record<UseCase, Spec[]> = {
  route:    [groq("openai/gpt-oss-20b"), groq("openai/gpt-oss-120b"), gemini("flash-lite"), gemini("flash")],
  classify: [gemini("flash"), groq("openai/gpt-oss-20b")],
  answer:   [groq("openai/gpt-oss-120b"), gemini("flash"), openrouter("…:free")],
};
 
export async function chatJson(useCase: UseCase, input: Input) {
  for (const spec of CHAINS[useCase]) {
    if (coolingDown(spec)) continue;
    try {
      return await call(spec, input, { timeoutMs: TIMEOUTS[useCase] });
    } catch (e) {
      cool(spec, e);           // decides how long to skip this model
    }
  }
  throw new AllModelsFailed(); // the caller has a non-AI fallback
}

The OpenAI-compatible APIs (Groq, OpenRouter, Cerebras) share one client with a different base URL; Gemini uses its own SDK. Everything else is policy.

Rule 1: a chain per use case

Different jobs want different models:

• Routing (deciding what a message is asking for) must be fast and return strict JSON. Small models with schema-guaranteed output are ideal.
• Classification runs in the background; latency matters less than consistency.
• Answers stream to a user and benefit from a stronger model.

Separate chains also spread load: routing and answering don't compete for the same model's tokens-per-minute.

Rule 2: cool down on 429 — and on slowness

When a provider returns 429, it usually includes a retry-after header. Calling it again immediately is pointless. LetRelay puts that model in a 60-second cooldown and moves to the next in the chain.

The less obvious rule came from a measurement. Our router evaluation once showed 19–21 seconds for a single routing call. The cause: one free model wasn't rate-limiting — it was hanging. With a 9-second timeout and two attempts, each call waited 18 seconds before failing over. Only 429s triggered a cooldown, so every request paid the full wait again.

The fix: treat timeouts and 5xx errors as a signal too, with a 30-second cooldown. After that change, the first call that hit the hung model took about 7 seconds (one timeout, then failover), and subsequent calls took 0.5–1.3 seconds on the next model, because the hung one was skipped.

Rule 3: no retries when a user is waiting

Retries are right for background jobs and wrong for interactive ones. A retry on a rate-limited or hung model doubles the wait and spends quota for a low chance of success. LetRelay's interactive use cases (routing, chat answers) get zero retries per model; the chain is the retry. Background classification can retry once, later, from a scheduled job.

AWS's Builders' Library article on timeouts, retries and backoff makes the general point: retries amplify load on a struggling dependency, so they need limits, backoff and jitter. In a free-tier gateway, "limits" means "move to another provider".

Rule 4: short, per-use-case timeouts

LetRelay's routing call has a 5-second timeout. Measured on free keys, routing through Gemini Flash took 4–7 seconds; through Groq's small model, well under a second. A 5-second ceiling means a slow model costs at most 5 seconds before failover rather than 9 or 18.

Nielsen Norman Group's classic response-time limits explain why this matters: about 1 second keeps a user's flow of thought uninterrupted, and about 10 seconds is the limit of keeping attention. A chat that pauses for 20 seconds before starting to answer feels broken, however good the answer is.

Rule 5: guaranteed JSON where it counts

For routing and classification, require schema-constrained output. Groq's strict structured outputs guarantee the response matches your JSON schema on supported models (not with streaming); Gemini accepts a response schema too. Mark every field required and nullable when unknown — a model can't drop a field it must fill. Then validate again with Zod. Details in AI request classification with structured output.

Rule 6: the gateway can fail; the feature can't

When every model in a chain is cooling down, the gateway throws — and the caller must have a non-AI answer:

• Request intake saves the request and leaves it for human triage.
• The assistant answers how-to questions from the best-matching page of its built-in guide, or formats the data it already fetched, and says plainly that it can't take actions right now.
• Common questions come from a cache of earlier answers (product how-to only, never organization data), so repeated questions cost no tokens.

LetRelay's no-AI end-to-end test runs nine assistant conversations with every model disabled, and all nine must still get a useful answer. The philosophy is in the case for AI that degrades gracefully.

Rule 7: skip the model when a rule is certain

The cheapest AI call is the one you don't make. Before routing, a deterministic check handles messages whose intent is obvious: a lone ticket reference like RLY-42, a greeting, a suggestion button's fixed text, "where is settings". Those never reach a model, which saves quota and removes any chance of a wrong route.

Observability you need

Log per call: use case, provider, model, latency, outcome (ok, 429, timeout, invalid JSON), tokens if reported. Two views pay for themselves:

• Failover rate per model — a model that fails over constantly should move down its chain.
• Daily token use against each limit — so "the assistant is slow today" has an explanation.

Never log prompts containing user data unless you need them, and never log API keys.

Where the keys live

All provider keys are server-only environment variables, used from Route Handlers or Server Actions. None go in NEXT_PUBLIC_ variables. The browser talks to your route; your route talks to the providers. Add a per-user rate limit in front of AI routes so one person can't spend the day's quota — see rate limiting without Redis.

FAQ

Is it allowed to use several free AI tiers together?

Each provider's terms govern its own free tier; using several providers in one app is common. Read each provider's terms for commercial use and data handling before relying on them in production.

How long should an AI API timeout be?

For interactive routing or classification, a few seconds — LetRelay uses 5. For streamed answers, time out on the first token rather than the whole response.

Should I retry on HTTP 429?

Not on the same model while a user waits. Respect retry-after, put that model in a cooldown, and fail over to another model or provider.

What happens when every free model is exhausted?

The feature falls back to a non-AI path: save and queue for humans, answer from static guides or cached answers, and tell the user plainly what isn't available.

Sources

• Groq — Rate limits
• Google — Gemini API rate limits
• OpenRouter — API limits
• AWS Builders' Library — Timeouts, retries and backoff with jitter
• Nielsen Norman Group — Response times: the 3 important limits
• MDN — 429 Too Many Requests