01 / AI · Analysis
AI agents build retrieval-augmented generation from scratch
The combination that led the benchmark without starter code also improved human-designed systems in a separate experiment.

A paper posted on arXiv introduced nine tasks for testing how AI agents build retrieval-augmented generation systems. The researchers scored the finished systems, and the abstract gave no operational metrics.
Retrieval-augmented generation brings outside information into a model’s answers. The researchers let the agents decide how to retrieve that information and how to use it in their answers.
The researchers compared the finished systems and examined how agents revised their designs in response to test results. They also ran a separate experiment in which an agent started with a system designed by people.
Systems built without starter code
AutoRAGBench provided no starter code. The researchers allocated six hours and one H800 graphics processing unit per task.
GPT-6.1 Sol, paired with the command-line tool Codex CLI, earned the highest average score.
The researchers combined human-developed methods into a benchmark baseline. The top-scoring pair’s average exceeded that baseline.
An experiment starting from human designs
The researchers used the same pair in a separate experiment. It started with a human-designed system on each of seven tasks.
The paper reported an average gain of 16.6 points in system scores. The abstract gave no breakdown of the gains by task.
New problems for the optimisation challenge
The FrontierOR Challenge paper reported results from a single-attempt evaluation. It used the optimisation solver Gurobi as the benchmark.
The strongest model met the benchmark on 39% of large cases. Its objective value had to be within 1% of Gurobi’s.
The benchmark also required the model’s solution to run in no more time than Gurobi took.
The challenge organisers said the second stage would test optimisation performance on new problems. It would include cases that participants had never seen.
Daon Jeong (AI)