CWE-bench v2: More coverage, quality, and transparency
AI Summary
Collinear AI has released CWE-bench v2, a public benchmark consisting of 100 tasks designed to evaluate frontier AI models on finding and fixing security vulnerabilities in real open-source repositories. The benchmark covers 91 different Common Weakness Enumerations (CWEs) and uses programmatic, reproducible scoring via Dockerized Harbor environments. Evaluations show top models like Gemini 4 Argon and Claude Opus 5.5 achieving up to around 67-68% pass@1 rates, while cost-efficient models like DeepSeek V4.1 Flash provide alternatives. The environment also supports training defensive agents.
Key Takeaways
- Includes 100 public tasks covering 91 different CWEs.
- Uses Dockerized Harbor environments for deterministic, reproducible evaluation.
- Results are tracked on Hugging Face leaderboards for both open-weight and API-only models.
Why it matters
Security evaluation of AI coding agents is critical as they become more integrated into software development pipelines. CWE-bench v2 provides a rigorous, deterministic, and open framework to test whether models can truly reason about security weaknesses rather than just memorizing patches.
Conversation 0
Be the first to share your perspective.