Breaking02 / IT
CWE-bench v2 opens 100 security tasks to AI model developers
Even the leading model failed 19% of the tasks on every attempt, and the announcement gives no licence or commercial terms.

Collinear AI released the security benchmark CWE-bench v2 on 3 Nov 2026, opening 100 public tasks so that model developers can run their own evaluations, the company said.
According to the Hugging Face blog post, an agent has to fix a security weakness in an open-source repository. Existing functionality has to keep working. The tasks span 91 types in CWE, the Common Weakness Enumeration, a catalogue of software weakness types. The post states that "Recognising a familiar pattern is not enough."
The company reported results for 11 models, with four attempts per task. The pass rate on one attempt was 67.8% for Gemini 4 Argon and 67.3% for Claude Opus 5.5. At least one model solved 95 of the tasks. Even the leading model failed 19% of the set on every attempt.
Compatibility. The blog post said the evaluation setup is unchanged. An agent still receives a repository and a short instruction. The dataset name in the Harbor command is [email protected].
According to the post, using the dataset requires accepting access conditions on its page. With a Hugging Face token saved to git's credentials, the benchmark runs without cloning the repository by hand. Evaluation on the private task set requires contacting Collinear AI.
Running environment. The post said each task contains a Docker environment, an instruction and a verifier. Scoring is programmatic, so the same results can be reproduced. With -k 4, each task gets four attempts, and the score is the mean reward over all 400 trials.
The blog post does not state the dataset's licence or commercial terms, or whether v2 scores can be compared with earlier ones.