CWE-bench v2 Releases 100 Open Tasks to Evaluate AI on Real Security Fixes
Collinear AI made CWE-bench v2 public with 100 reproducible tasks that test whether AI models can find and patch vulnerabilities in actual open-source code without breaking…
- The full set of 100 tasks, grader, and Harbor harness are open on Hugging Face so anyone can run deterministic evaluations on their own models at no extra cost.
- Gemini 4 Argon leads with 67.8 percent pass@1, followed closely by Claude Opus 5.5; 95 tasks are solvable by at least one model yet the long tail remains hard.
- Tasks were designed for realism and fairness with programmatic checks, manual review, and a held-out private set for continued tracking.
What Changed With CWE-bench v2
Collinear AI published a public set of 100 tasks on November 3, 2026. Each task gives an AI agent a checkout of a real open-source repository and a short instruction to locate and fix a security weakness.
The goal is to produce a working patch that does not break existing functionality. Scoring is fully programmatic and deterministic, allowing reproducible results without manual grading or extra cost. A held-out set of unseen tasks remains private; researchers can request evaluation by contacting [email protected].
Task Quality and Coverage
The 100 tasks span 91 different CWEs, covering a wide range of vulnerability types. For each task the team verified that the instruction reads like a realistic developer request, all graded behavior can be inferred from the instruction, the verifier checks actual behavior rather than one specific fix structure, the starter code fails while the reference fix passes, original functionality survives the patch, and verifiers reject memorized upstream patches.
Every task also received a manual review. The public release, open grader, and contribution process are intended to improve quality through community feedback.
Current Model Performance
Of the 11 models evaluated, Gemini 4 Argon achieves the highest scores at 67.8 percent pass@1 and 81.0 percent pass@4. Claude Opus 5.5 follows with 67.3 percent and 78.0 percent. DeepSeek V4.1 Flash reaches 57.0 percent pass@1 at a low cost of $0.04 per rollout, placing the three models on the Pareto frontier for performance versus price.
95 of the 100 tasks have been solved by at least one model, confirming the benchmark is solvable. However, even the leading model fails 19 percent of tasks on every attempt, showing that complex edge cases are not yet reliably handled.
How to Run the Benchmark
The dataset is gated on Hugging Face; users must first accept the access conditions. Tasks ship as self-contained Harbor environments with a registry.json, so the entire benchmark can be executed directly from the repo using a single Harbor command that spins up Dockerized setups and deterministic verifiers.
Open-weight model results update automatically on the dataset leaderboard via GitHub PRs. A separate leaderboard Space tracks API-only models such as Gemini, Claude, and GPT. The same environments can supply rewards for training defensive agents.
Why it matters for uncensored AI
Our analysis starts from the source fact that Collinear AI has open-sourced the full task set, grader, and harness while inviting community contributions and feedback on GitHub.
This transparency gives developers a practical way to test and compare how different models handle realistic code-security work. Greater visibility into these capabilities can inform choices about which models to adopt or fine-tune for technical projects.
When evaluation benchmarks become more open and reproducible, it supports the broader push for user-controlled tools over closed corporate systems.
