Streaming output

A user asks a question and the screen stays blank; three seconds later, a whole answer appears at once. Same three seconds, but if the first character appears after half a second and the rest follow one by one, it feels much better: the user can see the program is working, and can read while waiting.

This is streaming. As lesson 2 of module 01 showed, the model generates one token at a time anyway. Streaming just sends you each small piece as soon as it's generated, instead of collecting everything and sending it together.

Turning streaming on

Add stream=True to the request, and instead of a complete answer you get back a data stream you can iterate over with a for loop:

stream = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "用大约 200 字介绍 httpx 和 requests 的主要区别。"}],
    stream=True,
    extra_body={"thinking": {"type": "disabled"}},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

Each chunk is a small piece of data, and the newly generated text is in chunk.choices[0].delta.content. delta means "increment": it contains only the new content in this small piece, not everything so far. So you have to join them together yourself.

print's end="" makes each piece of text follow on without a line break, and flush=True shows it on screen immediately instead of waiting for the buffer to fill. Leave out flush=True and you'll see the text appear in batches, losing the streaming effect.

Measuring how much faster

code/03-llm-apps/streaming.py calls the same question once with streaming and once without, recording when the first character appears and when everything is done, both with thinking off and with it on:

def streaming(thinking, show=False):
    start = time.time()
    first_content = None
    stream = client.chat.completions.create(
        model=MODEL, messages=QUESTION, stream=True,
        stream_options={"include_usage": True},  # 让最后一个数据块带上 usage
        extra_body={"thinking": {"type": "enabled" if thinking else "disabled"}},
    )
    usage = None
    for chunk in stream:
        if chunk.usage:
            usage = chunk.usage
        if not chunk.choices:
            continue
        delta = chunk.choices[0].delta
        if delta.content:
            if first_content is None:
                first_content = time.time() - start
            if show:
                print(delta.content, end="", flush=True)
    if show:
        print()
    return first_content, time.time() - start, usage.completion_tokens

What I got (it prints one streamed answer first to show the effect; only the timing is kept here):

不思考 非流式：第一个字 1.82 秒，全部完成 1.82 秒，输出 153 词元
不思考 流式：  第一个字 0.67 秒，全部完成 1.71 秒，输出 202 词元
开思考 非流式：第一个字 2.48 秒，全部完成 2.48 秒，输出 330 词元
开思考 流式：  第一个字 1.82 秒，全部完成 2.75 秒，输出 379 词元

With thinking off, the first streamed character appeared at 0.67 seconds, while without streaming you see nothing for 1.82 seconds.

Note that the "all done" times are about the same for both (1.71 against 1.82 seconds, and the two answers weren't the same length either). Streaming doesn't make the model generate any faster; the total time is unchanged. What it changes is when the user starts seeing content. The longer the answer, the bigger the difference: for a long answer that takes 20 seconds to generate, no streaming means the user stares at a blank screen for 20 seconds.

With thinking on, the first streamed character took 1.82 seconds to appear, because the model thinks first and only starts writing the formal answer when it's done. The thinking is streamed too, in delta.reasoning_content. If you want users to see "thinking…", you can display it, or just show a placeholder.

Getting usage while streaming

For a non-streaming call, response.usage tells you directly how many tokens were used. A streaming call has no such information by default.

Add stream_options={"include_usage": True} and the server sends one extra data chunk at the end of the stream that contains usage, but whose choices is an empty list. That's why the code above has if not chunk.choices: continue; without that line, accessing chunk.choices[0] raises an IndexError.

Things to watch when streaming

finish_reason is in the last chunk. When streaming, finish_reason is None in the earlier chunks; only the last chunk with content has a value like stop or length. To check whether the answer was cut off, record it inside the loop.

Errors can happen partway through. A non-streaming call either succeeds or fails. A streaming call can break off after half the output, when the user has already seen half an answer. Retrying at that point generates a fresh answer from the start, which may not match the first half the user already saw. The usual approach: errors while establishing the connection can be retried; errors after output has started should tell the user "the answer was interrupted" and let them decide whether to ask again. RepoBot in lesson 5 does exactly this.

You have to assemble the complete answer yourself. In a multi-turn chat, the model's answer has to be stored in the history as an assistant message. When streaming there's no ready-made complete answer, so you join all the delta.content pieces together.

Sending the stream to a browser

In a terminal, print is enough. For a web page, you need your backend to forward the model's output to the browser in real time. The most common way is SSE (Server-Sent Events): a mechanism browsers support natively for a server to keep pushing messages to the browser. Its format is very simple: each message is a line data: content followed by a blank line.

A minimal example with FastAPI (code/03-llm-apps/streaming_web.py):

import json
import os

from fastapi import FastAPI
from fastapi.responses import HTMLResponse, StreamingResponse
from openai import AsyncOpenAI

client = AsyncOpenAI(  # 网页服务要同时应付很多请求，用异步客户端
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.deepseek.com"),
)
MODEL = os.environ.get("LLM_MODEL", "deepseek-flash")
app = FastAPI()


@app.get("/chat")
async def chat(q: str):
    async def events():
        stream = await client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": q}],
            stream=True,
            extra_body={"thinking": {"type": "disabled"}},
        )
        async for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                # SSE 的格式：每条消息以 "data: " 开头，以空行结尾
                yield f"data: {json.dumps(chunk.choices[0].delta.content, ensure_ascii=False)}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(events(), media_type="text/event-stream")

A few key points:

• Use AsyncOpenAI, not OpenAI. A web service handles many users at once, and a synchronous client blocks the whole service while it waits for the model's answer; an asynchronous client can handle other requests while it waits.
• StreamingResponse takes a generator, and each yield from the generator sends a piece of data to the browser.
• Each piece of content is encoded with json.dumps. The model's output may contain newlines, and SSE separates messages with newlines, so putting it in raw would break the format; encoding it as a JSON string avoids the problem.
• A final [DONE] tells the browser it's finished.

On the browser side, receive it with EventSource:

const source = new EventSource("/chat?q=" + encodeURIComponent(question));
source.onmessage = (e) => {
  if (e.data === "[DONE]") { source.close(); return; }
  out.textContent += JSON.parse(e.data);
};

The complete page code is in streaming_web.py. Install the dependencies and start it:

uv add fastapi uvicorn
uvicorn streaming_web:app --port 8000

Open http://127.0.0.1:8000 in a browser to try it. You can also look at the raw SSE data with curl; the -N flag makes curl display each piece as soon as it arrives:

curl -N "http://127.0.0.1:8000/chat?q=用一句话介绍httpx"

The first few messages I saw:

data: "HTTP"

data: "X"

data: " "

data: "是一个"

data: "功能"

Each message is one or two tokens. Every time the browser receives one, it appends it to the page.

EventSource can only send GET requests, so the question has to go in the URL, which limits its length and makes it awkward to include the conversation history. Real projects usually send a POST request with fetch and read the returned data stream. RepoBot's deployment in lesson 6 of module 06 uses that approach.

When not to stream

• The result goes to a program. For the JSON extraction in lesson 4 of module 02, for example, the program needs the complete JSON before it can parse it, so streaming is pointless.
• Background batch jobs. Nobody is waiting at a screen, and streaming only complicates the code.

Anywhere a person is waiting at a screen for an answer, you should stream.

Exercises

1. In streaming.py, change the question to "写一篇 800 字的文章介绍 httpx" (write an 800-character article introducing httpx) and compare the time to first character with and without streaming again. Is the gap bigger?
2. Change the streaming function so that with thinking on it also prints delta.reasoning_content, in grey or with some other marker, so users can see what the model is "thinking".
3. Add a feature to streaming_web.py: when the stream ends, send one more message telling the browser how many tokens this answer used (remember stream_options).

Self-check

1. Does streaming make the model generate the complete answer faster?

No. The total time to generate the complete answer is essentially unchanged. What streaming changes is when the user sees the first character: content is shown as it's generated, so the user doesn't have to wait for all of it before seeing anything.

2. A streaming call with stream_options={"include_usage": True} raises an IndexError on the line chunk.choices[0]. Why?

With include_usage on, the server sends a final chunk that contains only usage, and its choices is an empty list. Check whether chunk.choices is empty before accessing choices[0].

3. Why should a web backend use AsyncOpenAI rather than OpenAI?

A synchronous client blocks while it waits for the model's answer, and during that time the server can't handle other users' requests. An asynchronous client can switch to other requests while waiting, so one process can serve many users at once.