Breaking01 / AI
Only two of five EU AI Act assistants beat generic baselines
Purpose-built development did not guarantee better performance, and contestant integration code and historical results are outside the public release.

Only two of five purpose-built AI assistants consistently outperformed off-the-shelf baseline models in a test on the EU AI Act, according to an arXiv paper that Marco Virgolin and co-authors published on 3 Nov 2026.
Three off-the-shelf baselines and five purpose-built assistants took part in regenold's 2026 EU AI Act Q&A Challenge, according to figures the authors published in the paper. Four commercial developers and one academic entrant built the assistants. One of the five achieved high scores across all evaluated metrics.
The benchmark has 110 open-ended questions on the EU AI Act, formally Regulation (EU) 2024/1689, according to the paper. Each question comes with an exemplary answer, question-specific correctness criteria and expected references. The GitHub repository documentation sets the evaluation scope as the EU AI Act as of 1 May 2026.
Scoring. The overall score is the geometric mean of eight metrics, according to the GitHub repository documentation. GPT-5.4 is the default judge model for correctness and tone. Hard mode scores only the final answer, given after a fixed nine-exchange history, the question and a user pushback.
Specialist development alone "does not guarantee better performance", the researchers wrote in the paper's abstract. They also wrote that evaluating large language model (LLM) assistants is critically important in regulatory affairs.
Availability. On GitHub, regenold GmbH published the question workbook, the fixed conversation history and a PDF of the EU AI Act text. The code and benchmark annotations are licensed under the Apache License 2.0. The two built-in example baselines are GPT-5.5 with web search and GPT-5.5 with RAG (retrieval-augmented generation, a method that supplies a model with passages retrieved from documents).
The private integration code for contestant systems and the historical results discussed in the paper are not distributed, according to the repository documentation.