Data annex

Everything on this site traces back to raw run logs, and this page is the index. Preregistrations carry their lock dates; briefs report the campaigns as run, including the arms that went against expectation.

Sample questions

Sample questions, per map family — verbatim from the launch manifests, including the full 20-gate corridor ramp.

Results briefs

E1a paired campaign — 13-model cohort, paired control/wiped arms, pre-registered falsifier.
E1a control baseline — the full-cohort unaided baseline table.
The --look-gate result — instruction vs. enforcement of observe-before-answer (the flag's origin): telling did ~1 gate, forcing did ~23.
A3 ceiling-arm efficiency — the four ceiling models' wiped arms, from a registered follow-up: one met the efficiency falsifier, two resolved against the overlay.
Cohort onboarding record — how the cohort was assembled and gated, including the exclusions. Lands here at release.

Pre-registrations

Cohort campaign + external anchor — locked 2026-07-06, before the campaign ran; verbatim copy with the approval line and Addenda A1/A2.
Context-strategy axes — locked 2026-07-15. Verbatim copy lands here at release.

Figures

Strip plot of paired control vs wiped run depths per model — E1a paired arms — every run, both arms, reversals plotted identically to hits.

Strip plot of control-arm run depths for the full cohort — Control baseline — the full cohort, unaided.

Graph rendering of a gpt-oss:120b run through the alpha-2 maze: blue route with numbered steps, two dead-end detours, exit reached at step 15 — A single run, drawn: gpt-oss:120b traversing alpha-2 — dead-end detours, backtracks, exit at step 15. The renderer that drew this (`scripts/render_trace.py`) was written by **qwen3:14b** (14.8B, Q4_K_M, running locally) inside this benchmark's own sandbox harness, wiped-curated arm: task spec and an 11-point automated checker were frozen first; qwen took 3 attempts (0/11, 10/11, then 11/11 with zero human edits to any of its output); the full per-turn transcripts of all three attempts ship in the raw logs (`results/renderer-cell/`).

Raw data

repo results/e1a-table1/ — the E1a aggregate (e1a_table1.json) and campaign table sources.
/entries/ — the leaderboard entry files served by this site, one JSON per entry, CI bounds included (regenerate with cli/seed_entries.py).
Raw per-run JSONL logs (zip, ~11 MB) — one line per run, per-turn traces included; 15 control + 13 wiped arms (9 sub-ceiling per Addendum A2, 4 ceiling per Addendum A3) with a manifest. Model-host URLs in errored-attempt records are redacted to [lan-host]; every other byte is as-run.