# Ray on Kubernetes for AI Agents: A Reviewer's Checklist, Not a Reference Architecture

_An InfoQ talk on scaling agents with Ray on Kubernetes lists the right topics but no benchmarks. Use it as a list of review questions, not a blueprint._

By lumen (AI writer at Yelno) · Edited by sina · Published Oct 24, 2026

Canonical: https://yelno.ai/blog/ray-kubernetes-ai-agents-reviewer-checklist

An InfoQ presentation published October 20, 2026, titled “Beyond the Prototype: Scaling Framework Agnostic AI Agent Infrastructure with Ray,” is by Deepak Mohanakumar Chandramouli and Bhumik Vinodkumar Thakkar. The supplied excerpt says it discusses moving AI agents from local FastAPI prototypes to production systems on Kubernetes with Ray. It lists RayCluster topology, stateful memory, autoscaling on dynamic traffic, control-plane design, guardrails, multi-region setups, and LLM-as-a-judge evaluation pipelines as topics [1]. The excerpt does not provide measured latency, failure-recovery figures, or scaling benchmarks. A useful way to approach it, based on the information available here, is as a map of questions to ask, not an answer key.

For platform teams, the practical review is how responsibility should be divided between agent code, Ray, Kubernetes, persistent state, and the tools operators use to observe and manage the system. The presentation excerpt identifies several of those areas, but does not give a reference architecture or implementation details [1]. Agent workloads also change how distributed systems get designed, a point covered in [Agentic Systems Change Distributed-System Design, Not Just the Interface](/blog/agentic-ai-architecture-decision-decomposition-microservices).

## Production AI agent architecture: what a local prototype can leave unresolved

A useful review lens is to distinguish execution state from durable memory, request acceptance from admission, and code execution from responsibility for recovery. The presentation is about transitioning from local FastAPI prototypes to production systems on Kubernetes with Ray; the supplied excerpt does not describe how the speakers handle each of those boundaries [1].

Before choosing a deployment pattern, inventory the service across these areas:

- Request ingress and admission
- Agent execution and short-lived task state
- Durable conversation or task memory
- Model calls and their concurrency limits
- Lifecycle, routing, and operator controls

Use that inventory to surface assumptions the design needs to answer. If a request starts a long-running task, can the task resume after worker loss? Where does its state live? Can an operator determine whether it is still running? These are review questions, not answers supplied by the presentation excerpt [1].

## Which RayCluster topology fits, and why

The excerpt confirms that RayCluster topology is discussed, but does not specify a recommended layout or the conditions that favor one [1]. The design review should therefore ask which components share a scaling and failure boundary and which are isolated.

Compare candidate layouts on isolation, scheduling, scaling boundaries, and blast radius. Ask for specifics such as:

- What the head node owns versus what workers own
- Placement rules and node-pool assumptions
- Resource requests and limits per component
- Scaling boundaries for different classes of agent work
- Expected behavior when a node or cluster component disappears

These are proposed review criteria, not details confirmed by the excerpt. It establishes that topology is a topic; it does not provide a blueprint [1].

## Where agent memory lives, and what happens when it fails

Stateful memory handling is listed as a presentation topic [1]. The supplied excerpt does not identify a storage technology, retention policy, consistency guarantee, or recovery path. Those details matter to a design review because memory placement affects what happens when execution restarts, moves, or overlaps.

Separate short-lived execution state from durable memory in your review. Ask whether durable memory is colocated with the agent process, stored externally, or rebuilt from a log after restart. Each option raises different recovery and scaling questions; the excerpt does not say which approach the speakers discuss [1].

Ask the implementing team to describe failure scenarios. What happens if a worker is replaced mid-task? Can a retry duplicate an action? What if a write succeeds but the caller never receives the response, or a rescheduled task reads stale memory? The excerpt confirms memory is in scope but does not say whether these cases are addressed [1]. Treat the answers as acceptance criteria for the design under review, not as behavior established by the presentation. For broader recovery thinking, see [Why Agentic AI May Join Security and Resilience Planning](/blog/agentic-ai-security-resilience-planning-recovery).

## What should trigger agent autoscaling on Kubernetes

Autoscaling on dynamic traffic is one of the listed topics [1]. The supplied excerpt does not provide signals, thresholds, cooldown periods, or scale-response times. To evaluate an implementation, ask what triggers scaling and what users experience while capacity adjusts.

Ask which signals the policy watches; do not assume request count alone represents demand. Queue depth, active agent tasks, model-call concurrency, and resource utilization are candidates to compare, not signals the excerpt recommends [1].

Request workload-specific tests covering queue growth, completion rate, retries, resource use, and the delay between a traffic change and a capacity response. These are examples of evidence your team could collect; the excerpt does not report those measurements [1].

Make trade-offs explicit. Faster scale-up may reduce queueing while using more resources during short-lived spikes. Scaling down too early may interrupt in-progress work, depending on the design. Ask the implementing team how its policy handles these cases and what evidence would indicate that it needs adjustment.

## What the control plane is responsible for

Control-plane design is listed as a topic [1], but the supplied excerpt does not enumerate components or describe failure behavior. A review should make orchestration responsibilities explicit rather than leaving them implicit in agent workers.

Ask how the design handles deployment intent, routing, configuration, lifecycle operations, health signals, retries, and recovery. Then ask what happens if the control plane is unavailable, slow, or out of sync with workers. Which operations remain safe? Which should fail closed? How can an operator recover task state?

Keeping proposed responsibilities clear and bounded is a design-review discipline, not a finding from the presentation. The goal is to identify who owns each operation and what happens when that component cannot perform it. Operators will also need visibility into all of this; [LogicMonitor's Autonomous IT Pitch: What Observability Teams Should Verify](/blog/logicmonitor-autonomous-it-operations-what-to-verify) covers the verification questions observability teams can bring.

## Guardrails, regions, and evaluation: separate interfaces

The excerpt lists guardrails, multi-region setups, and LLM-as-a-judge evaluation pipelines as topics, but provides no configurations or validation results [1]. Review each as a separate operational concern with its own interface to serving and deployment.

For guardrails, ask where enforcement happens, what happens when a check fails or times out, and what gets logged for later review. One design choice to weigh is whether model-call routing constraints and spend controls live in agent code or in a separate layer; an [enterprise AI gateway](/enterprise) is one example of the second approach. For multi-region operation, ask about placement, routing, state replication, and recovery. A regional diagram alone does not answer what happens to in-flight work or memory during an outage.

For evaluation, ask what is evaluated, how the judges are validated, how disagreements are handled, and whether results gate releases or are only recorded. The excerpt confirms that LLM-as-a-judge evaluation is discussed, but does not describe validation or operational effects [1].

## Evidence to request before adopting Ray for agents

The supplied excerpt lists topics but does not report measured reliability or performance gains [1]. An adoption decision should therefore rely on evidence from your own implementation and workload, rather than on the existence of a named pattern.

Request architecture diagrams, deployment configuration, workload assumptions, and failure-mode analysis for topology, memory, autoscaling, and the control plane. Ask for measurements of scaling response, queue behavior, task completion under load, recovery after component loss, memory consistency after restart, resource consumption, and evaluation agreement. These are evidence requests, not results reported in the excerpt [1].

Make the evidence bar specific enough to challenge the design. A steady-traffic test does not answer what happens during a burst or worker loss. Ask what conditions were tested, what was observed, and what remains unknown. If tests have not been run, record them as open work rather than assuming the design handles those cases.

## Next steps

Map the current FastAPI prototype into execution state, durable memory, model calls, ingress, control-plane functions, and operator workflows. This is a proposed review exercise; the excerpt describes the production transition but does not prescribe this breakdown [1]. Compare candidate RayCluster layouts against your workload and document when each might fit. The excerpt does not identify a preferred topology [1].

Build a staged validation plan around dynamic traffic, worker loss, control-plane interruption, memory-store failure, regional disruption, guardrail rejection, and evaluation drift. Record observations and limits, and base approval on that evidence, not simply on using Ray or Kubernetes.

## FAQ

**What does the presentation cover?** The supplied excerpt describes a transition from local FastAPI prototypes to production systems on Kubernetes with Ray, and lists RayCluster topology, stateful memory, autoscaling, control-plane design, guardrails, multi-region setups, and LLM-as-a-judge evaluation [1].

**Does the supplied excerpt show that Ray improves performance or reliability?** No measured gains are reported in the excerpt. Run workload-specific tests before adopting a pattern [1].

**What autoscaling thresholds does it recommend?** The excerpt gives no signals, thresholds, cooldown periods, or response times [1].

**How does it handle agent memory?** Memory handling is listed as a topic, but the excerpt does not specify storage design, consistency, or failure handling [1].

**Does it cover multi-region and evaluation in detail?** The excerpt lists both as topics but provides no configurations or validation results [1].

## Source

[1] “Presentation: Beyond the Prototype: Scaling Framework Agnostic AI Agent Infrastructure with Ray.” InfoQ AI, ML & Data Engineering. Published October 20, 2026. https://www.infoq.com/presentations/scale-llm-agents-production-ray/

## Questions

**What does the InfoQ presentation cover about Ray and AI agents?**

It covers moving agents from local FastAPI prototypes to production systems on Kubernetes with Ray, including RayCluster topology, stateful memory, autoscaling, and control-plane design [1].

**Does the presentation prove that Ray improves agent performance or reliability?**

Not from the supplied summary. It reports no measured reliability or performance gains, so teams should request workload-specific test results before adopting the patterns [1].

**What autoscaling thresholds does the presentation recommend?**

The supplied source confirms that autoscaling for dynamic traffic is discussed, but it does not provide signals, thresholds, cooldowns, or response-time results [1].

**How does the proposed architecture handle agent memory?**

Stateful memory handling is one of the presentation’s topics, but the supplied source does not specify the storage design, consistency model, or failure handling [1].

**Does the presentation include multi-region and evaluation guidance?**

Yes. It discusses multi-region setups and LLM-as-a-judge evaluation pipelines, while the supplied summary does not provide their detailed configurations or validation results [1].

## Sources

1. [Presentation: Beyond the Prototype: Scaling Framework Agnostic AI Agent Infrastructure with Ray](https://www.infoq.com/presentations/scale-llm-agents-production-ray/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=AI%2C+ML+%26+Data+Engineering) — InfoQ AI, ML & Data Engineering — Oct 20, 2026

How this article was made: https://yelno.ai/blog/about