InfraDev Tests Coding Agents on Multi-GPU Infrastructure. The Best Score Is 38.9%.

InfraDev is a coding agent benchmark of multi-file tasks that must run across multiple GPUs. Across six frontier models, the best resolved 38.9% of its 36 tasks. That makes it a stress test of a specific kind of infrastructure work—not, by itself, a buying decision. ¹

A coding agent benchmark for AI infrastructure development

The benchmark authors say existing benchmarks have focused on single kernels or single-GPU tasks. InfraDev's 36 tasks come from real pull requests to Megatron-LM, vLLM and verl, covering training, inference and reinforcement learning. Each asks an agent to reimplement a change spanning many files and make it run across multiple GPUs. ¹

That is a demanding test of integration and distributed execution. Whether it represents your team's work is a separate question: the reported results concern tasks from those three projects, so applying them to another codebase is an extrapolation. ¹ For a different slice of infrastructure work, see this look at whether agents can make custom kernels for Ascend NPUs.

Evaluation dimension	What InfraDev does	Why it matters to a buyer
Code scope	Tasks require changes spanning many files 1	Tests integration, not an isolated edit
Execution scope	Changes must run correctly across multiple GPUs 1	Includes distributed execution in the task requirements
Workload areas	Covers training, inference and reinforcement learning 1	Includes more than one kind of infrastructure task
Repository coverage	Uses Megatron-LM, vLLM and verl 1	Results come from these codebases; applying them elsewhere requires caution

Hidden probes: coding agent evaluation beyond official tests

InfraDev combines official tests with hidden probes designed to enforce requirements those tests leave unchecked. The benchmark also supplies a prompt that states the interfaces the probes call without revealing the solution. ¹ The probes rejected 32% of submissions that passed every official test. ¹ So passing the official tests alone did not establish that a submission met all the benchmark's requirements.

The authors report that structural and state errors dominate failures, and that agents' own tests rarely check memory, time or traffic behavior. ¹ This points to a practical lesson: test integration and resource behavior, not only the visible functional requirements.

What the scores actually show

Across six frontier models, the best resolved 38.9% of the tasks. ¹ When given the original pull-request descriptions as prompts, none of the six models resolved a task. ¹ That contrast shows that the prompt condition was associated with a sharp change in results; the abstract alone does not establish why.

The authors also report that turning their findings into an agent skill raised a frontier model's resolved rate on Megatron-LM tasks by 10.5%. ¹ Treat that reported improvement as a result to test in your own setting, rather than assuming it will transfer to other codebases or workflows.

What the benchmark doesn't establish

InfraDev evaluates 36 tasks drawn from three open-source repositories. ¹ Its results do not, on their own, establish how an agent will perform on proprietary code, other repositories or production deployment. Applying the benchmark's findings to those settings is an inference, not a result reported by this evaluation.

Multi-GPU execution is a demanding technical task, but a benchmark score is not evidence of operational readiness. A strong result would not, by itself, justify unsupervised access to production infrastructure. Related reading: more AI-generated code without a matching increase in completed work.

How to use InfraDev in your own coding agent evaluation

Treat InfraDev as a prompt for better internal testing, not a final score.

• Check task fit first. Compare InfraDev's multi-file, multi-GPU pull-request tasks with the work your team needs agents to do. ¹
• Review the evaluation setup. Before comparing results, establish which models, agent configurations, prompts and execution budgets were used. These are useful details to request when assessing any benchmark result.
• Build your own hidden checks. InfraDev's probes rejected 32% of submissions that passed every official test. ¹ Use that finding as a reason to test your own repository beyond visible functional checks—not as a prediction of the gap you will see.
• Score structural and resource behavior separately. The authors report structural and state errors as dominant failure modes, while agents' own tests rarely checked memory, time or traffic behavior. ¹
• Test the skill improvement locally. The reported 10.5% increase applies to a frontier model's resolved rate on Megatron-LM tasks. ¹ Measure whether similar guidance helps on your own tasks before relying on it.

The useful internal measure is not InfraDev's 38.9% score alone. It is how your agent performs on visible tests and on hidden checks designed around your systems' failure modes. That is a more relevant basis for deciding what work to trust it with.

If you run the same private tasks across several models, a single model catalog behind one API can reduce the setup work. For agents that interact with people, testing tool-using agents against users who leave things unsaid covers a complementary evaluation gap.

Frequently asked questions

What is InfraDev?

InfraDev is a benchmark with 36 tasks based on real-world pull requests to Megatron-LM, vLLM and verl. The tasks cover training, inference and reinforcement-learning infrastructure and require multi-file changes that run across multiple GPUs. ¹

How does InfraDev grade coding agents?

It uses hidden probes alongside official tests. The probes check requirements not covered by the original tests, while the prompt describes the probe interfaces without revealing the solution. ¹

How well did the evaluated models perform?

Across six frontier models, the best resolved 38.9% of tasks. Hidden probes rejected 32% of submissions that passed every official test. ¹

Does InfraDev predict performance on production infrastructure?

Not by itself. The benchmark covers 36 tasks from three repositories, so teams still need private evaluations on their own code, interfaces, distributed environments and operational requirements. ¹

Why did prompt design matter in the results?

When models received the original pull-request descriptions as prompts, none resolved a task. ¹ The prompt condition was associated with a sharp change in results, but the abstract alone does not establish why.

Source

¹ “InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?” arXiv, published 2026-10-28. https://arxiv.org/abs/2610.36589