yelno.ai
ModelsGet API key
Concepts / Streaming

Streaming

Deltas as they arrive, a terminal usage frame with the charge, and clear rules for fallback and disconnects.

Server-sent events

Set stream: true and read data: frames as they arrive — each is a chat.completion.chunk. The last frame before data: [DONE] carries usage and the same yelno object as a non-streaming response; its cost.state is final once the provider reported usage and provisional until then.

stream = client.chat.completions.create(model="openai/gpt-4o", messages=msgs, stream=True)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:                                   # last frame before [DONE]
        meta = chunk.model_extra["yelno"]
        print(meta["provider"], meta["cost"]["customer_nanos"], meta["cost"]["state"])  # openai 6400000 final

Frame types

TypeMeaning
delta.contentA chunk of text (the first frame carries only delta.role: "assistant")
delta.tool_callsA tool call's index, id and name, then partial function.arguments strings to concatenate
finish_reasonstop, length, tool_calls or content_filter, with an empty delta
usage + yelnoLast frame before [DONE]: prompt, completion and total tokens, and the same yelno object a non-streaming response carries
errorA failed attempt as the error envelope (type, code, retriable); a retriable one is followed by the next attempt's frames
[DONE]End of stream — always sent, after the terminal frame

Fallback mid-stream

If an attempt fails with a retriable error (rate_limit, overloaded, provider_error), the gateway moves to the next planned attempt — up to your policy's max_attempts — and you see an error frame followed by the next attempt's frames; every attempt and its state is listed in the terminal yelno.attempts. A non-retriable error ends the stream: the error frame is followed by the terminal frame with request_state: "failed", then [DONE].

Disconnects Closing the connection cancels the upstream request as soon as the gateway observes the disconnect. Usage seen so far is recorded with client_disconnected: true; if the provider never reported final usage, generation_may_have_continued is set and the cost stays provisional until reconciliation.