Streaming AI Responses in Next.js: Plain Streams vs SSE

Streaming AI responses in Next.js means a Route Handler returns a Response whose body is a ReadableStream, enqueues each chunk of model output as it arrives, and the browser reads it with fetch so words appear as they're generated. Format it as plain text or as Server-Sent Events (SSE): plain text is simpler for one chat answer; SSE helps with several event types. Either way, save before closing the stream, and have an answer ready if the model fails partway.

Why stream at all

A model answer of a few hundred words can take several seconds to generate. Without streaming, the user stares at a spinner for the whole time. With streaming, the first words appear in well under a second on a fast model, and people start reading. Nielsen Norman Group's response-time limits explain the difference: around 1 second keeps a user's flow of thought; around 10 seconds is the limit for keeping attention. Streaming turns a 6-second wait into a 0.5-second one, as far as perception goes.

Option A: a plain text stream

This is what LetRelay's assistant uses. The Route Handler returns a stream of UTF-8 text:

// app/api/assistant/route.ts
export async function POST(req: Request) {
  const { message } = await parse(req);            // Zod-validated
  const encoder = new TextEncoder();
  const chunks = chatStream("answer", { user: message }); // async iterable of strings
 
  const stream = new ReadableStream<Uint8Array>({
    async start(controller) {
      let full = "";
      try {
        for await (const chunk of chunks) {
          full += chunk;
          controller.enqueue(encoder.encode(chunk));
        }
      } catch {
        /* the model failed mid-answer — handled below */
      }
      if (!full.trim()) {
        const fallback = answerWithoutAi(message);
        full = fallback;
        controller.enqueue(encoder.encode(fallback));
      }
      await saveAnswer(full);                       // before close — see below
      controller.close();
    },
  });
 
  return new Response(stream, {
    headers: { "Content-Type": "text/plain; charset=utf-8", "Cache-Control": "no-store" },
  });
}

And the client:

const res = await fetch("/api/assistant", { method: "POST", body: JSON.stringify({ message }), signal });
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let text = "";
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  text += decoder.decode(value, { stream: true });
  setAnswer(text);
}

Pass { stream: true } to decode so multi-byte characters split across chunks (Nepali, Hindi, emoji) decode correctly.

Option B: Server-Sent Events

SSE is a text format — lines like event: … and data: … separated by blank lines — served as text/event-stream. The browser's EventSource API parses it and reconnects automatically. MDN's guide describes the format and API.

SSE is worth it when:

• You need several kinds of messages on one stream: text deltas, a "tool started" status, a final JSON payload with sources.
• The stream is long-lived and should reconnect automatically (notifications, progress feeds).

Its drawbacks for chat: EventSource only does GET requests without a body, so sending a chat message means either putting it in the URL (don't — messages can contain private information) or using fetch and parsing SSE yourself. Many AI SDKs do the latter.

For a single answer to a single POST, a plain text stream is less code and has nothing to parse. If you need structure later, you can send a small JSON header line first, or switch to SSE.

Rule 1: say the important thing first

The first bytes of a stream set expectations. In LetRelay, when the routing step fails (so the assistant can't look anything up or take actions this time), a fixed sentence goes out before any model text: that it can't do that right now. We added this after seeing the model, left to itself, tell users "I can't create tasks for you" in organizations where it could. Deterministic statements about the product's abilities shouldn't come from the model.

Rule 2: save before you close

On serverless platforms, work scheduled after the response finishes may be cut off. If you log the answer or cache it "after the stream", it may never be written. Do it inside start(), before controller.close(), and wrap it in try so a failed write never leaves the stream open.

Rule 3: handle failure mid-stream

A model can fail after some tokens have been sent. You can't take back sent text, so:

• If nothing was sent, send a complete non-AI answer (a guide page, formatted data, or a clear message). LetRelay's assistant has one for every question type — see AI that degrades gracefully.
• If some text was sent, append a short, honest note ("The answer was cut off — try again") rather than silently stopping.
• Time out on the first token, not the whole answer. A model that hasn't started after a few seconds should be abandoned for another; see a free AI gateway with failover.

Rule 4: let the user stop it

Give the fetch an AbortController and a Stop button. When the user aborts, the browser closes the connection. On the server, the stream's cancel() callback (or request.signal) tells you the client left, so you can stop pulling tokens from the model — which also stops spending quota.

Rule 5: don't cache, don't buffer

• Cache-Control: no-store: answers are per-user and must never be cached by a CDN.
• Some proxies buffer responses, which defeats streaming. Platforms that support streaming Route Handlers handle this; if you add your own proxy, disable response buffering for the route.
• Compression can delay small chunks on some setups. If the first words arrive in one lump, check buffering and compression before blaming the model.

Rule 6: render incrementally, safely

Rendering Markdown on every chunk is fine for short answers but re-parses the whole text each time. Throttle updates to every animation frame. Render model output as text or through a sanitizing Markdown renderer — never dangerouslySetInnerHTML on raw model output. And only turn links into anchors when they point to real routes in your app or to https:// URLs; models invent plausible paths. LetRelay's widget renders a link only if its route exists in the app's own feature registry.

When the answer isn't only text

Chat answers often need more than words: the sources used, a link to a record, or a proposed action the user can confirm. Three approaches, from simplest:

1. Two requests. Stream the text; fetch structured extras with a separate JSON call. Simple, and the text isn't delayed by the extras.
2. Structure before the stream. Decide the structured part first (it usually comes from a fast routing step), send it as a single JSON line, then stream the text. LetRelay's assistant works this way for proposed actions: the routing step decides "this is a create-task request", the server builds the confirmation card without any model call, and the card is returned as JSON rather than streamed prose.
3. SSE with event types. event: text for deltas, event: sources for citations, event: done for the final payload. The most flexible, and the most parsing.

Whichever you choose, keep proposed actions out of the model's free text. A card with fields a user can edit and confirm is safer than prose the user has to trust.

Streaming and rate limits

Streaming doesn't change token costs, but it changes what "too slow" means. With a streamed answer, users tolerate a long answer if the first words arrive quickly, so optimize time to first token: route with a small fast model, keep the answer prompt short, and skip models that are slow to start. Some providers' strict JSON modes don't work with streaming — Groq's strict structured outputs, for one — which is another reason to keep routing (non-streamed, strict JSON) separate from answering (streamed text).

Testing streaming

• Unit-test the non-AI fallback directly.
• In an end-to-end test, disable the AI keys and assert the stream still returns a useful answer.
• Measure time to first byte and total time separately. Users feel the first; your quota pays for the second.

FAQ

Should I use Server-Sent Events or a plain stream for a chatbot?

For one answer per request, a plain text stream read with fetch is simpler. Use SSE if you need multiple event types on one stream or automatic reconnection for long-lived feeds.

Why does my Next.js stream arrive all at once?

Usually buffering by a proxy or compression layer, or code that awaits the full model response before enqueueing. Enqueue each chunk as it arrives and check your hosting's streaming support.

How do I stop a streaming response?

Abort the fetch with an AbortController. On the server, handle cancellation to stop reading from the model.

Can I save the answer after the stream ends?

Save it before closing the stream. On serverless platforms, work after the response completes may not run.

Sources

• MDN — Using server-sent events
• MDN — Streams API
• Next.js — Route Handlers (route.js)
• Nielsen Norman Group — Response times: the 3 important limits