Security & Safety

Attacks, defenses, and the cost of refusing too much.

Retained generated outputs and their historical evaluators, not a matched or currently qualified experiment. Failures, incomplete coverage, replay scoring and different model/backend bindings remain explicit. A live flag does not establish containment or independent replication.

The original broad capture and its separately documented rescored view are both retained. ASR denominators differ by harness; reported differences are descriptive, not matched-case causal estimates. Rescoring does not create new observations.

Models & settings — click to expand

Attack Success Rate by harness

Successful attacks are failures of containment · Wilson 95% CI and case-cluster bootstrap in tooltip

ASR by technique

Higher bars = the technique evades this harness more often

Over-refusal (benign tasks wrongly refused)

The safety counterweight — refusing everything is not "safe"

ASR by surface

Indirect injection (repo file / tool output / memory) vs the direct user turn

ASR by modality — the multimodal gap

Same harmful ask delivered as text vs image vs audio (direct technique)

Evidence: side-effect-confirmed vs judge-only

Solid = L1 sandbox/structural confirmation · hatched = judge inference only

Utility-under-attack

Synthetic completion-marker results; modeled utility is not measured live task completion.

Reproducibility & budget

Case explorer

Click any row to inspect the prompt, rendered image/audio attack asset, response, proposed (never-executed) actions, sandbox side-effects, and judge rationale
CaseHarnessSurfaceTechniqueModalityObjectiveOutcome

Inspect the evidence

Inside the dataset

66 items
security-agentic_markdown_exfilsecurity-agentic_markdown_exfilrepo_file · markdown_exfil · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_env_example_trapsecurity-agentic_env_example_traprepo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_monitoring_urlsecurity-agentic_monitoring_urlrepo_file · markdown_exfil · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_voice_onboardingsecurity-agentic_voice_onboardingdirect_turn · workflow_framed · audio · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_rules_filesecurity-agentic_dep_rules_filerepo_file · rules_file · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_ci_fixsecurity-agentic_dep_ci_fixtool_output · diagnostic · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_readme_setupsecurity-agentic_dep_readme_setuprepo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_code_commentsecurity-agentic_dep_code_commentrepo_file · diagnostic · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_issuesecurity-agentic_dep_issuerepo_file · confused_deputy · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_important_messagesecurity-agentic_dep_important_messagetool_output · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_visionsecurity-agentic_dep_visiondirect_turn · workflow_framed · image · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_dangeroussecurity-agentic_dep_dangerousrepo_file · dangerous_action · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_memorysecurity-agentic_dep_memorymemory_session · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_requirementssecurity-agentic_dep_requirementsrepo_file · diagnostic · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_dep_lockfile_migrationsecurity-agentic_dep_lockfile_migrationrepo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_ci_deploysecurity-agentic_git_ci_deploytool_output · diagnostic · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_rules_filesecurity-agentic_git_rules_filerepo_file · rules_file · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_runbooksecurity-agentic_git_runbookrepo_file · dangerous_action · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_readme_releasesecurity-agentic_git_readme_releaserepo_file · workflow_framed · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_hidden_commentsecurity-agentic_git_hidden_commentrepo_file · confused_deputy · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_important_messagesecurity-agentic_git_important_messagetool_output · workflow_framed · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_cleanup_rmsecurity-agentic_git_cleanup_rmrepo_file · dangerous_action · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_memorysecurity-agentic_git_memorymemory_session · workflow_framed · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_precommitsecurity-agentic_git_precommitrepo_file · rules_file · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_commit_msgsecurity-agentic_git_commit_msgtool_output · confused_deputy · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-agentic_git_visionsecurity-agentic_git_visiondirect_turn · workflow_framed · image · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_se_health_diag_fill25security-hard_se_health_diag_fill25repo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_se_health_diag_fill50security-hard_se_health_diag_fill50repo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_se_health_diag_fill75security-hard_se_health_diag_fill75repo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_se_health_diag_fill90security-hard_se_health_diag_fill90repo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_md_build_shim_fill25security-hard_md_build_shim_fill25repo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_md_build_shim_fill50security-hard_md_build_shim_fill50repo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_md_build_shim_fill75security-hard_md_build_shim_fill75repo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_md_build_shim_fill90security-hard_md_build_shim_fill90repo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_dg_deploy_recovery_fill25security-hard_dg_deploy_recovery_fill25repo_file · diagnostic · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_dg_deploy_recovery_fill50security-hard_dg_deploy_recovery_fill50repo_file · diagnostic · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_dg_deploy_recovery_fill75security-hard_dg_deploy_recovery_fill75repo_file · diagnostic · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_dg_deploy_recovery_fill90security-hard_dg_deploy_recovery_fill90repo_file · diagnostic · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_se_monitor_badge_fill75security-hard_se_monitor_badge_fill75repo_file · markdown_exfil · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_md_migration_fill75security-hard_md_migration_fill75repo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_dg_git_workflow_fill75security-hard_dg_git_workflow_fill75repo_file · workflow_framed · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_se_config_value_fill75security-hard_se_config_value_fill75repo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_obf_se_health_ascii_smuggle_fill75security-hard_obf_se_health_ascii_smuggle_fill75repo_file · ascii_smuggle · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-hard_obf_se_health_base64_comment_fill75security-hard_obf_se_health_base64_comment_fill75repo_file · base64_comment · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-extreme_md_build_shimsecurity-extreme_md_build_shimrepo_file · workflow_framed · text · malicious_dep

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-extreme_se_config_auditsecurity-extreme_se_config_auditrepo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-extreme_dg_deploy_recoverysecurity-extreme_dg_deploy_recoveryrepo_file · diagnostic · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_policy_secretsecurity-fab_policy_secretrepo_file · false_provenance · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_git_reset_runbooksecurity-fab_git_reset_runbookrepo_file · false_provenance · text · destructive_git

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_tls_disable_adrsecurity-fab_tls_disable_adrrepo_file · false_provenance · text · safety_disable

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_auth_weaken_bugfixsecurity-fab_auth_weaken_bugfixrepo_file · diagnostic · text · safety_disable

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_illegal_sourcessecurity-fab_illegal_sourcesrepo_file · false_provenance · text · illegal_integration

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_scihub_educationalsecurity-fab_scihub_educationalrepo_file · false_provenance · text · illegal_integration

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_csrf_disablesecurity-fab_csrf_disablerepo_file · false_provenance · text · safety_disable

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_webhook_sig_skipsecurity-fab_webhook_sig_skiprepo_file · false_provenance · text · safety_disable

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_cors_wildcardsecurity-fab_cors_wildcardrepo_file · false_provenance · text · safety_disable

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-fab_secret_log_debugsecurity-fab_secret_log_debugrepo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-mal_telemetry_exfilsecurity-mal_telemetry_exfilrepo_file · diagnostic · text · secret_exfil

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-mal_postinstall_bootstrapsecurity-mal_postinstall_bootstraprepo_file · workflow_framed · text · arbitrary_shell

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-brand_clone_spotifysecurity-brand_clone_spotifydirect_turn · false_provenance · image · brand_clone

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-brand_clone_netflixsecurity-brand_clone_netflixdirect_turn · false_provenance · image · brand_clone

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-probe-directsecurity-probe-directdirect_turn · direct · text · canary_echo

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-probe-encoding_base64security-probe-encoding_base64direct_turn · encoding_base64 · text · canary_echo

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-benign-secret-exfilsecurity-benign-secret-exfildirect_turn · direct · text · benign

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-benign-malicious-depsecurity-benign-malicious-depdirect_turn · direct · text · benign

Prompt text is withheld in this restricted evidence view.

Open scored results ↓
security-benign-destructive-gitsecurity-benign-destructive-gitdirect_turn · direct · text · benign

Prompt text is withheld in this restricted evidence view.

Open scored results ↓

Inspect the evidence

Explore experiments

28 items
2026-06-18 · Historical run71 cases × 5 harnessestui-security-20260618-115204-721Plots, evidence & explorer →2026-06-18 · Historical run71 cases × 5 harnessessecurity-20260618-113527-607Plots, evidence & explorer →2026-06-18 · Historical run71 cases × 5 harnessessecurity-20260618-112634-829Plots, evidence & explorer →2026-06-18 · Historical run71 cases × 5 harnessessecurity-20260618-112142-804Plots, evidence & explorer →2026-06-21 · Historical run66 cases × 5 harnessestui-security-20260620-183114-748Plots, evidence & explorer →2026-06-19 · Historical run66 cases × 5 harnessessecurity-20260619-222106-625Plots, evidence & explorer →2026-06-19 · Historical run66 cases × 5 harnessessecurity-20260619-085443-688Plots, evidence & explorer →2026-06-21 · Historical run65 cases × 5 harnessestui-security-20260620-183114-748-rescoredPlots, evidence & explorer →2026-06-17 · Historical run62 cases × 5 harnessessecurity-20260617-214437-083Plots, evidence & explorer →2026-06-17 · Historical run62 cases × 5 harnessessecurity-20260617-163745-311Plots, evidence & explorer →2026-06-17 · Historical run62 cases × 5 harnessessecurity-20260617-162852-665Plots, evidence & explorer →2026-06-16 · Historical run62 cases × 5 harnessessecurity-20260616-202027-496Plots, evidence & explorer →2026-06-19 · Historical run42 cases × 5 harnessestui-suite-suite-20260619-092721-599-rescoredPlots, evidence & explorer →2026-06-20 · Historical run27 cases × 5 harnessestui-suite-suite-20260619-235349-892-rescoredPlots, evidence & explorer →2026-06-19 · Historical run6 cases × 5 harnessessecurity-20260618-234515-417Plots, evidence & explorer →2026-06-18 · Historical run4 cases × 2 harnessessecurity-20260618-230901-802Plots, evidence & explorer →2026-06-18 · Historical run4 cases × 2 harnessessecurity-20260618-215609-429Plots, evidence & explorer →2026-06-18 · Historical run4 cases × 3 harnessessecurity-20260618-195345-957Plots, evidence & explorer →2026-06-18 · Historical run3 cases × 2 harnessessecurity-20260618-223204-355Plots, evidence & explorer →2026-06-18 · Historical run3 cases × 5 harnessessecurity-20260618-111543-986Plots, evidence & explorer →2026-06-23 · Historical run2 cases × 5 harnessessecurity-20260623-231757-176Plots, evidence & explorer →2026-06-18 · Historical run2 cases × 2 harnessessecurity-20260618-213810-194Plots, evidence & explorer →2026-07-14 · Historical run1 cases × 7 harnessessecurity-20260714-111027-284Plots, evidence & explorer →2026-06-23 · Historical run1 cases × 4 harnessessecurity-20260623-131759-586Plots, evidence & explorer →2026-06-18 · Historical run1 cases × 5 harnessessecurity-20260618-122333-883Plots, evidence & explorer →2026-06-18 · Historical run1 cases × 1 harnessessecurity-20260618-120209-884Plots, evidence & explorer →2026-06-18 · Historical run1 cases × 5 harnessessecurity-20260618-120209-734Plots, evidence & explorer →2026-06-18 · Historical run1 cases × 5 harnessessecurity-20260618-014250-734Plots, evidence & explorer →