Streaming

Set "stream": true on any chat request to receive the response as Server-Sent Events (Content-Type: text/event-stream). The gateway forwards tokens to your client as they arrive from the upstream provider.

SSE passthrough

The gateway serves a request only when your SDK’s wire format matches the provider’s (see Which SDK to call a provider with), so the provider’s events are already in the shape your SDK expects. The gateway forwards the SSE bytes as they arrive, without re-serializing them, and reads token usage, model, finish reason, and content from the stream as it passes. Apart from error frames (see Mid-stream errors), it changes a stream only to add something the provider’s own format requires and the provider left out:

a closing data: [DONE] line, when the provider ends the stream without one;
the index of a streamed tool_calls entry, when the provider omits it.

Usage chunk on `/v1/chat/completions`

On a streaming /v1/chat/completions request, the gateway sets stream_options.include_usage to true so the provider reports the request’s token usage, even when your request sets it to false. Cohere is the exception and is not sent the option. The stream then includes a chunk whose usage carries the token counts and whose choices array is empty, so check choices before indexing it, as the Python example below does.

Requesting a stream

curl -N https://gw.to11.ai/v1/chat/completions \
  -H "x-to11-authorization: Bearer $TO11_API_KEY" \
  -H "x-to11-project-id: $TO11_PROJECT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai::gpt-4o",
    "messages": [{"role": "user", "content": "Count to 3"}],
    "stream": true
  }'

from openai import OpenAI

client = OpenAI(base_url="https://gw.to11.ai/v1")
stream = client.chat.completions.create(
    model="openai::gpt-4o",
    messages=[{"role": "user", "content": "Count to 3"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")

Guardrails during streaming

When output guardrails are enabled, the gateway accumulates response text (up to 64 KB) to inspect it. The enforcement point differs between streaming and non-streaming because streamed chunks have already left the gateway by the time the full response is known.

Mode	Behavior
Non-streaming	The full response is inspected before delivery. A violation returns `422 Unprocessable Entity` and the response is withheld.
Streaming	Chunks are delivered as they arrive. After the stream completes, accumulated text is checked; because the content has already been sent, a violation is recorded for monitoring rather than blocking the response.

The 64 KB accumulation cap bounds only the text inspected for guardrails — it never truncates the response streamed to your client, which continues unbounded.

Mid-stream errors

If an upstream fails after the stream has already started (HTTP 200 with text/event-stream headers sent), the gateway delivers the error as an SSE error frame in your SDK’s shape and stops the stream — it does not emit a false success terminator. See Error handling for the frame format per SDK.

Telemetry

Streaming and non-streaming requests produce the same GenAI span — model, token counts, finish reason, and timing — and the same metrics. Streaming additionally records time-to-first-token as gen_ai.server.time_to_first_token, measured from the start of the response to the first content token. See the Telemetry reference for the full attribute and metric catalog.

SDKs

CLI

Gateway

SSE passthrough

Usage chunk on `/v1/chat/completions`

Requesting a stream

Guardrails during streaming

Mid-stream errors

Telemetry

​SSE passthrough

​Usage chunk on /v1/chat/completions

​Requesting a stream

​Guardrails during streaming

​Mid-stream errors

​Telemetry

SSE passthrough

Usage chunk on `/v1/chat/completions`

Requesting a stream

Guardrails during streaming

Mid-stream errors

Telemetry