Model APIs / Batches & files

Batches & files

Send thousands of chat or embeddings requests as one JSON Lines file, collect the results as files, and pay each line's final price as it runs.

Routes

The OpenAI Files and Batches APIs, unchanged: point an OpenAI SDK at the gateway and files.create, batches.create, batches.retrieve, batches.cancel and files.content work as they do there. Every route takes your platform API key with the inference:invoke scope, and you only ever see your own organization's files and batches.

MethodPathWhat it does

POST/v1/filesUpload a JSON Lines file (multipart: file, purpose=batch).

GET/v1/filesList your files, newest first (purpose, limit, after).

GET/v1/files/{file_id}One file's metadata.

GET/v1/files/{file_id}/contentThe file's bytes.

DELETE/v1/files/{file_id}Delete a file.

POST/v1/batchesCreate a batch from an uploaded file.

GET/v1/batchesList batches, newest first (limit, after).

GET/v1/batches/{batch_id}A batch's status, counts, files and cost.

POST/v1/batches/{batch_id}/cancelStop a batch: lines not yet run are cancelled and never charged.

Input file

One request per line. Each line needs a custom_id that is unique in the file (it is how you match results), method: "POST", a url equal to the batch's endpoint, and the request body you would send to that endpoint. Supported endpoints are /v1/chat/completions and /v1/embeddings. A line may name any model you can call; stream is ignored because each line is answered once, in the output file.

requests.jsonl

{"custom_id": "ticket-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize ticket 1"}]}}
{"custom_id": "ticket-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize ticket 2"}]}}

Files are UTF-8 JSON Lines up to 50 MB, and a batch holds up to 50,000 requests. A file that breaks a line rule gives a batch with status: "failed" and an errors list naming each line and problem (invalid_json_line, duplicate_custom_id, mismatched_endpoint, invalid_method, missing_required_parameter). A problem inside a body, such as a missing messages or a model you cannot use, fails only that line, with the same error the live request would get.

Create and poll

Create a batch

import os
from openai import OpenAI

client = OpenAI(base_url="https://yelno.ai/v1", api_key=os.environ["YELNO_API_KEY"])

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

batch = client.batches.retrieve(batch.id)          # poll until status is terminal
if batch.output_file_id:
    print(client.files.content(batch.output_file_id).text)

200 · the batch object

{
  "id": "batch_…",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": "file_…",
  "completion_window": "24h",
  "status": "completed",
  "output_file_id": "file_…",
  "error_file_id": null,
  "created_at": 1759600000,
  "in_progress_at": 1759600000,
  "expires_at": 1759686400,
  "completed_at": 1759600420,
  "request_counts": { "total": 2, "completed": 2, "failed": 0 },
  "metadata": {},
  "yelno": {
    "project_id": "proj_…",
    "cost": { "state": "final", "currency": "USD", "customer_nanos": 17640 }
  }
}

StatusMeaning

in_progressValidated; lines are being run.

finalizingEvery line has finished; the output and error files are being written.

completedDone. Some lines may still have failed: read request_counts and the error file.

cancellingCancel requested; lines already running are finishing.

cancelledStopped. Lines that had not started are in the error file as batch_cancelled.

expiredThe 24-hour window ended first. Lines not run are in the error file as batch_expired.

failedThe input file did not validate; errors lists each problem with its line number.

completion_window is 24h: lines still waiting when it ends are not run. metadata takes up to 16 string pairs. Cancelling a batch that already finished answers 409 batch_not_cancellable.

Output and error files

When a batch finishes, output_file_id holds one line per successful request and error_file_id one line per request that failed or never ran. Either is null when it would be empty. Lines are in input order; match them by custom_id. Download them with GET /v1/files/{file_id}/content. They are kept for 30 days.

output file · one line

{"id": "breq_…", "custom_id": "ticket-1", "response": {"status_code": 200, "request_id": "req_…", "body": {"object": "chat.completion", "choices": […], "usage": {…}, "yelno": {…}}}, "error": null}

response.body is exactly what the live endpoint returns, including its yelno block with the route and the charge. A failed line carries its status code and the standard error envelope; upstream text never reaches you.

error file · two lines

{"id": "breq_…", "custom_id": "ticket-7", "response": {"status_code": 402, "request_id": "req_…", "body": {"error": {"type": "insufficient_funds", "message": "…", "code": "…", "request_id": "req_…", "param": null, "retriable": false}}}, "error": null}
{"id": "breq_…", "custom_id": "ticket-8", "response": null, "error": {"code": "batch_cancelled", "message": "This request was not run because the batch was cancelled."}}

How a batch is charged

Each line is a request. Every line runs through the gateway as its own request, with its own req_ id in your request log. A hold is placed before the line is sent, and the line is charged the model's listed final price when it finishes, exactly as if you had sent it yourself. Rate limits, budgets, routing policies and your source allow-list apply to every line. Nothing is held for the batch as a whole. A cancelled or expired line is never charged.

The batch's yelno.cost.customer_nanos is the sum of its lines' charges. It reads accruing while lines run and final once the batch is finished and every line has settled. If your wallet runs out mid-batch, the remaining lines fail with 402 insufficient_funds. Lines limited by your rate limit wait and retry automatically at no cost. A revoked or suspended key stops its batch.

Storage and privacy

Batch files are the one place the gateway keeps prompts and responses, because a batch needs them. Uploaded files, each line's body and result, and the output and error files are encrypted at rest and bound to your organization. Delete an input file whenever you like. Output and error files are removed 30 days after the batch finishes. In the console, Observe · Batches lists your batches with progress and cost, and downloads their files. Each download is written to your audit log.

Batches & files

#Routes

#Input file

#Create and poll

#Output and error files

#How a batch is charged