Streaming an LLM Response Through a Worker

A Worker proxying a model API is a few lines, and the first version I wrote streamed perfectly from the provider and delivered nothing to the browser until it was finished.

The cause was one await:

const text = await upstream.text();
return new Response(text, { headers });

return new Response(upstream.body, { headers });

upstream.body is a ReadableStream. Hand it straight to the Response and the runtime pipes it through as chunks arrive. Anything that consumes the body first, whether .text(), .json(), or a helper that logs the response, collapses the stream into a buffer.

The same trap hides in pipeTo. It returns a promise that resolves when the stream is finished, so awaiting it before returning the response waits for the whole generation. Don’t await it, or pass it to ctx.waitUntil() if you need the completion for logging.

Chunk boundaries are not event boundaries

The second bug was subtler and only showed up under load. I was transforming the provider’s SSE into my own format, and parsing looked reasonable:

for (const line of chunk.split('\n')) { … }

Network chunks split wherever the network feels like it. An SSE event can arrive in two pieces, and a multi-byte character can be cut down the middle. Occasionally a token came through mangled, or an event vanished.

Both problems need a buffer that survives across chunks:

let buffer = '';

new TransformStream<string, string>({
    transform(chunk, controller) {
        buffer += chunk;
        const lines = buffer.split('\n');
        buffer = lines.pop() ?? '';

        for (const line of lines) {
            if (line.startsWith('data: ')) controller.enqueue(line.slice(6));
        }
    },
});

Keeping the trailing partial line in buffer handles split events. Decoding with TextDecoderStream before this step handles split characters, because it holds incomplete sequences internally instead of emitting replacement characters.

One thing in your favour

Workers bill CPU time, not wall clock. A request that spends forty seconds waiting on a slow model costs almost nothing, because your code is idle for nearly all of it. Streaming is the cheap path here as well as the responsive one.