Security spotlight
Fewest successful attacks first.
ASR is a failure metric, not a capability score. Smaller is safer; it does not establish task usefulness.
Gauntlet / agent evaluation
Five tracks.
A closer look.
Explore safety, code quality, generative builds, full projects, and bug fixes — from the headline plots down to individual cases and generated artifacts.
Historical results, with their original sources and limitations. No pooled score across unrelated experiments.
Five ways to inspect an agent
Follow the evidence.
Security & Safety
Attacks, defenses, and the cost of refusing too much.
—Attack success ·
Explore plots & dataset →Code Quality & Output Security
Requirement coverage, code health, and output security.
—Checks passed ·
Explore plots & dataset →Generative Capability
Feature delivery, build reliability, and generated app galleries.
—Feature completeness ·
Explore plots & dataset →Full-Repo Project Build
Whole-repository builds, signal vectors, and visual targets.
—Build composite ·
Explore plots & dataset →Bugfix · Issue Resolution (SWE-bench-style)
Issue resolution, held-out checks, and patch inspection.
—Issues resolved (shown) ·
Explore plots & dataset →The complete record
99
Experiments, not just highlights.
Both historical collections, all retained runs, source provenance, and downloadable measurements. Nothing is averaged into a new overall score.
Open the Results archive →