Concepts / Streaming

# Streaming

Deltas as they arrive, a terminal usage frame with the charge, and clear rules for fallback and disconnects.

## Server-sent events

Set `stream: true` and read `data:` frames as they arrive — each is a `chat.completion.chunk`. The last frame before `data: [DONE]` carries `usage` and the same `yelno` object as a non-streaming response; its `cost.state` is `final` once the provider reported usage and `provisional` until then.

```
stream = client.chat.completions.create(model="openai/gpt-4o", messages=msgs, stream=True)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:                                   # last frame before [DONE]
        meta = chunk.model_extra["yelno"]
        print(meta["provider"], meta["cost"]["customer_nanos"], meta["cost"]["state"])  # openai 6400000 final
```

## Frame types

| Type | Meaning |
| --- | --- |
| delta.content | A chunk of text (the first frame carries only delta.role: "assistant") |
| delta.tool_calls | A tool call's index, id and name, then partial function.arguments strings to concatenate |
| finish_reason | stop, length, tool_calls or content_filter, with an empty delta |
| usage + yelno | Last frame before [DONE]: prompt, completion and total tokens, and the same yelno object a non-streaming response carries |
| error | A failed attempt as the error envelope (type, code, retriable); a retriable one is followed by the next attempt's frames |
| [DONE] | End of stream — always sent, after the terminal frame |

## Fallback mid-stream

If an attempt fails with a retriable error (`rate_limit`, `overloaded`, `provider_error`), the gateway moves to the next planned attempt — up to your policy's `max_attempts` — and you see an `error` frame followed by the next attempt's frames; every attempt and its `state` is listed in the terminal `yelno.attempts`. A non-retriable error ends the stream: the error frame is followed by the terminal frame with `request_state: "failed"`, then `[DONE]`.

Disconnects Closing the connection cancels the upstream request as soon as the gateway observes the disconnect. Usage seen so far is recorded with `client_disconnected: true`; if the provider never reported final usage, `generation_may_have_continued` is set and the cost stays provisional until reconciliation.

Source: https://yelno.ai/docs/streaming