Breaking01 / AI
AI manufacturing models can run yet give wrong answers
Comparison with reference models exposed errors in the researchers' evaluation; the abstract reports no independent replication.

The SimuLean researchers reported an 86% valid-model rate (93 successes in 108 attempts across 18 manufacturing modelling tasks) in their paper, Agentic discrete-event simulation for manufacturing system design: an open 3D simulator, a benchmark and an industrial case, published on 10 Nov and not yet peer-reviewed, using reference-model comparisons; the abstract reports no independent replication.
The researchers present SimuLean as an open-source discrete-event simulation tool, in which a system's state changes when events occur. Built on the Unity engine, it is designed to let AI agents construct and analyse manufacturing models through commands describing manufacturing operations.
For the benchmark interface, the researchers report that models built through domain commands alone also retain a record of how they were constructed. This provides a trace of the modelling process.
The researchers report that every remaining failure involved a model that could be built and run but returned incorrect answers. Comparison with reference models was needed to reveal all these errors. The abstract reports the team's comparison results, but no replication by independent researchers.
The authors argue that having AI write simulation code from scratch creates new, unverified software each time. SimuLean instead verifies the rules governing model behaviour within the platform, then lets agents assemble models on top of those rules.
The abstract does not name the agents used in the evaluation.