Breaking01 / AI
AI agents' experiment ideas clear review in 194 of 415 cases
Evidence that agents' own rankings help decide which experiments to run first is still limited.

AI research agents' experiment proposals were judged sound, valuable and new in 194 of 415 cases, according to the rTaste paper by Art Kanke, published on 29 Oct 2026.
Of the 194, 73 were also judged non-obvious and beyond the paper the agents started from. Kanke, of the University of Minnesota, gave seven recent natural language processing papers to six frontier models. Each model, alone or as a team, proposed five small experiments and recorded a probability of success and a ranking before anything ran.
A separate executor model attempted every proposal, including those ranked last. Claude Opus 5.5, GPT-6.1 Sol and Grok 4.7 acted as judges.
The author wrote in the abstract that evidence of the agents' rankings improving the choice of experiments remains limited. The agents' first choices matched the top-rated experiment more often than a random pick, but there was no evidence they beat taking the experiment written first. Their forecasts were more accurate than a base rate, but not measurably better than those of the same model given only the paper and the proposals.
By model. In author-published figures in the repository, GPT-6.1 Sol averages 3.00 passes per five proposals alone and 3.71 as a team. Muse Spark 1.3, a smaller model, scores 0.14 alone and 1.00 as a team. Counted separately from passes, non-obvious proposals number 21 each for GPT-6 Astra alone and as a team, and 20 and 22 for GPT-6.1 Sol.
Team cost. In the author's published figures, teams pass 0.71 more proposals on average than single agents, with a 95% confidence interval of −0.10 to 1.52. Per $100, single agents produce 13 passing proposals and teams 11.
A METR risk assessment cited in the paper found no evidence that any of four frontier labs had handed research agendas or budget decisions to agents. The evaluation tools and the dataset of 415 proposals are publicly available on GitHub. The paper's abstract and the repository description give no results against human researchers.