Can a Research Agent Tell a Good Plan from a Plausible One?

In Sci$\Pi$vot, the highest accuracy among the tested models in the project-record condition was 63.5%. Fresh researchers scored 73.4%. At medium reasoning effort with the project record, four of five frontier models performed below chance on the selected cases. ¹

Those results raise a practical question for teams considering research agents: can an agent choose between its original plan and a human-corrected alternative using the information available at the time? Sci$\Pi$vot, a scientific research agent benchmark, tests that kind of choice. Its results indicate difficulty on the benchmark's selected cases, not a general verdict on agent reasoning. ¹

What this scientific research agent benchmark measures

Sci$\Pi$vot presents a research record and asks an agent to choose between the original LLM plan and a human-corrected alternative. The agent receives the project record as it existed before the correction. ¹

That is a different question from whether an agent can execute a plan. An agent could, in principle, carry out a weak plan competently; a successful run would not by itself show that the plan was the right one. This distinction is a useful way to think about what a decision benchmark measures, though Sci$\Pi$vot's abstract does not compare it with other evaluation methods.

The benchmark's paired-choice format probes one aspect of scientific judgment. It does not establish whether an agent could execute its choice well or whether the resulting science would hold up. ¹

How the test cases were built

Sci$\Pi$vot contains 248 cases drawn from six months of research. Senior researchers ran LLM agents on 26 projects across six scientific fields and intervened to correct research judgments. ¹

For each case, the evaluated agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction. ¹ This design asks whether an agent can choose the alternative that a researcher corrected, based on the record available at the time. It does not test the subsequent execution or scientific validity of that choice. ¹

What the abstract leaves open

The abstract does not report a breakdown of decision types or of cases across projects and fields. It also does not specify how representative the 248 cases are of decisions in a typical R&D group. ¹

The cases come from research in which senior researchers intervened to correct agents' judgments. ¹ The abstract does not describe criteria for validating those corrections, so readers cannot assess the ground-truth process from the abstract alone. ¹

Before relying on the headline scores, it would be useful to ask for a taxonomy of decision types, case counts by project and field, selection criteria, a validation rubric for corrections, and sample sizes and uncertainty bounds for the reported accuracy figures. The abstract reports the accuracy figures but not those details. ¹

Reading the results: AI research agent evaluation at medium effort

At medium reasoning effort with the project record, models systematically preferred the original LLM plan over the human-corrected alternative. Four of five frontier models performed below chance on the selected cases. The highest accuracy across the tested models and reasoning-effort settings in the project-record condition was 63.5%, compared with 73.4% for fresh researchers. ¹

More context did not improve accuracy in the reported medium-effort comparison: every model scored lower with the project record than without it. Reconsideration did not consistently improve overall accuracy. ¹

These results concern this benchmark setup. The abstract does not explain why the project record lowered accuracy or establish whether the findings predict performance on new projects or in operational workflows. ¹

One possible interpretation is that relevant signals are difficult to extract from the record; another is that the additional context makes it harder for models to change their initial choice. These are hypotheses, not explanations established by the reported results.

Why completion and judgment are different questions

A completion test asks whether an agent carried out assigned work. Sci$\Pi$vot asks whether it selected the human-corrected alternative over its original plan. ¹ Those are different evaluation questions: an agent could execute a plan successfully without having selected the stronger plan in the first place.

Treat execution and decision evaluation as complementary. Neither alone establishes that an agent is suitable for a particular research program.

An evidence checklist for autonomous science agents

A benchmark score is a starting point for evaluation, not a deployment decision. The abstract reports headline accuracies but does not provide sample sizes, confidence intervals, or definitions of the chance and fresh-researcher baselines. ¹ Before relying on an aggregate score for autonomous science agents, consider:

• Reproducibility. Can an independent team rebuild the cases and reproduce the scoring from the stated method?
• Transfer. Does performance hold on your projects, fields, and decision types? The abstract does not establish that benchmark accuracy predicts operational performance. ¹
• Decision-specific error rates. A false continuation, a false rejection, and a failure to flag a need for more data can have different costs. An aggregate score does not show how those errors are distributed.
• Workflow comparison. Compare the agent alone, the agent with project context, and human review. Since reconsideration did not consistently improve overall accuracy in the benchmark, test whether your own review step helps rather than assuming that it will. ¹ Once an agent is running, drift checks, not just launch tests, help show whether results hold over time.
• Downstream outcomes. Can the agent execute the plan it picked, and does the resulting work hold up scientifically? Sci$\Pi$vot's paired-choice format does not test those outcomes. ¹

Set the authorization bar according to the stakes of the decision. A benchmark result can justify further evaluation; it cannot, by itself, justify delegating consequential research decisions without supervision. That is a cautious recommendation, not a result established by Sci$\Pi$vot.

A practical plan for evaluating AI experiment decision making

Map your research process into explicit decision points, then identify which ones can be tested using a paired-choice format and which need locally built cases. Sci$\Pi$vot's design offers one possible template: compare an agent's choice between its original plan and a human-corrected alternative using information available before the correction. ¹

Build held-out cases from your own projects and compare agent choices with researchers who receive the same information. Track whether adding context or asking the agent to reconsider changes its answer. Sci$\Pi$vot examines reasoning effort, access to the project record, and reconsideration, making those reasonable dimensions to test locally. ¹ If you want to run the same cases across several models, a model catalog is a place to see which ones are available.

Decision quality is only one axis. Other benchmarks look at different failure modes, such as whether agents act before the world changes, and step-level safety controls address what an agent is allowed to do along the way.

Set authorization limits based on your own evaluation. Sci$\Pi$vot's value is not a pass/fail grade for every agent; it poses a specific question for research teams: can an agent make the kind of judgment represented by these cases? Test that question on your own projects before relying on the benchmark alone.