> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Streaming responses
> Token-by-token responses with the OpenAI SDK, over server-sent events.
On a model whose page lists streaming, stream tokens as they are generated so a user sees the answer appear instead of waiting for all of it. Any chat UI wants this. So does anything long enough that silence looks like a hang. This guide uses the OpenAI-compatible stream. See [Anthropic Messages](/docs/api-reference/anthropic-messages#streaming) for the event-named Messages grammar.
## Minimal code
```python Python theme={"dark"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.runinfra.ai/v1",
api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)
stream = client.chat.completions.create(
model="nemotron-3-5-lightning-30b",
messages=[{"role": "user", "content": "Tell me a short story"}],
max_tokens=16384,
stream=True,
)
for chunk in stream:
if not chunk.choices:
continue # the usage frame has no choices
delta = chunk.choices[0].delta.content or ""
print(delta, end="", flush=True)
```
```typescript TypeScript theme={"dark"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.runinfra.ai/v1",
apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});
const stream = await client.chat.completions.create({
model: "nemotron-3-5-lightning-30b",
messages: [{ role: "user", content: "Tell me a short story" }],
max_tokens: 16384,
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```
```bash cURL theme={"dark"}
curl -N https://api.runinfra.ai/v1/chat/completions \
-H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "nemotron-3-5-lightning-30b",
"messages": [{"role":"user","content":"Tell me a short story"}],
"max_tokens": 16384,
"stream": true
}'
```
## What your loop will see
## What to tune
| Parameter | Effect |
| - | - |
| `stream_options.include_usage` | Adds the final usage frame with token counts. Off by default. |
| `max_tokens` | Hard cap on generated length. Reasoning models spend part of this budget before answering, so keep it generous. |
| `temperature` | Higher is more surprise per token. |
| `stop` | Up to four stop sequences. The stream ends early on a match. |
## Common mistakes
* **Forgetting `flush=True` in Python.** Without it stdout buffers and the tokens arrive in clumps.
* **Treating an error frame as a network failure.** A stream that fails mid-generation ends with one final frame carrying the standard error envelope, on the same HTTP 200. Handle it inside your frame loop.
* **Breaking the loop too early.** The final content frame carries `finish_reason` and an empty `delta.content`. Keep reading until the stream closes.
* **Streaming through a buffering proxy.** Some edges buffer server-sent events. Stream direct, or disable buffering on that layer.
* **Mixing `n > 1` with streaming.** Allowed, but every delta carries its own `choices[].index`. Route by index or the outputs interleave.
## Next steps
Exact frame shapes, and the usage frame.
Let the model act, then answer.
Get JSON that matches your schema.