Breaking01 / AI
PERFARENA evaluates the cost of running AI-generated code
The open-source tool assesses code correctness alongside runtime, memory use and energy consumption.
Researchers announced PERFARENA on 16 Nov as an open-source tool for evaluating code performance across 7 programming languages, helping developers choose models by how their generated code runs.
The paper's abstract reported measurements of 9,455 correct code solutions from 13 large language models (LLMs). These researcher-published figures accompany an evaluation framework covering correctness, runtime, peak memory use and energy consumption.
The researchers designed PERFARENA to run generated code in Unix-like environments. Its structure allows tools for judging correctness and measuring performance to be replaced without changing the evaluation process.
The researchers also released rankings and measured code solutions to support model selection by language and performance metric. These provide an additional basis for choosing models alongside correctness rates and pricing.
According to the researchers, existing code-generation evaluations mostly assess correctness alone. Performance-focused evaluations typically cover few languages and rarely measure energy consumption.
The abstract does not disclose absolute runtime or energy consumption values for individual models.