Bugfix · Issue Resolution (SWE-bench-style)

SWE-bench-Pro-style graded scoring: each arm fixes a buggy repo from the issue. Resolved = the shown hidden test passes (FAIL_TO_PASS). Strict also requires held-out anti-overfit tests + PASS_TO_PASS regression tests (the arm never sees them) with no test-file edits. The composite [0–100] grades robustness, regression, patch minimality/locality and code health — so two arms that both "resolve" still separate.

Issue resolution, held-out checks, and patch inspection.

Retained generated outputs and their historical evaluators, not a matched or currently qualified experiment. Failures, incomplete coverage, replay scoring and different model/backend bindings remain explicit. A live flag does not establish containment or independent replication.

Eleven easy/hard issues and five harnesses; the failed OpenCode measurement is retained.

Models & settings — click to expand

Patch-quality composite by harness

Graded quality-of-fix [0–100]: robustness (held-out) 40% · regression 25% · minimality 15% · locality 10% · code health 10%, gated on resolution (blended 80/20 with the semantic patch judge when one is configured — the live default)

Resolution: shown vs strict (held-out + regression)

Lenient = shown hidden test passes. Strict = held-out anti-overfit + PASS_TO_PASS regression also pass (no test edits). A gap = overfit/regressing patches.

Resolution explorer

Per issue × harness: composite, strict resolution, held-out (anti-overfit) + regression pass counts, patch minimality, and the patched files.
IssueHarnessCompositeStrictHiddenHeld-outRegressionMinimalityBackend

Inspect the evidence

Inside the dataset

11 items
sum-evenssum_evens returns the sum of odd numbers
mathlib.calc.sum_evens(nums) should return the sum of the EVEN numbers in nums, but it currently returns the sum of the odd numbers. Fix it so even numbers are summed (and an empty list sums to 0).
Open scored results ↓
stack-lifoStack.pop removes the oldest item (FIFO) instead of the newest (LIFO)
ds.stack.Stack is meant to be a last-in-first-out stack, but Stack.pop() removes the OLDEST item instead of the most recently pushed one. Fix pop() so the stack is LIFO; popping an empty stack should raise IndexError.
Open scored results ↓
slugify-normalizeslugify does not lowercase or strip punctuation
textutil.slug.slugify(s) should produce a URL slug: lowercase, words separated by single hyphens, with all non-alphanumeric characters removed and no leading/trailing hyphens. It currently only replaces spaces with hyphens. Fix it to fully normalize the string.
Open scored results ↓
paginate-ceilpage_count truncates the final partial page
pagelib.page.page_count(total, per_page) should return the number of pages needed to show `total` items at `per_page` per page, rounding up for a partial last page. It currently uses integer division, dropping the final partial page. Fix it (0 items is 0 pages).
Open scored results ↓
wordcount-casefoldword_count is case-sensitive
countlib.words.word_count(text) should count word frequencies case-insensitively, returning a dict of lowercase word -> count. It currently counts case-sensitively, so 'The' and 'the' are separate keys. Fix it to fold case.
Open scored results ↓
clamp-boundsclamp ignores the lower bound
numlib.clamp.clamp(x, lo, hi) should constrain x to the inclusive range [lo, hi]. It currently applies only the upper bound, so values below lo pass through unchanged. Fix it to apply both.
Open scored results ↓
lru-recencyLRUCache.get does not refresh recency, evicting recently-read keys
lru.cache.LRUCache(capacity) is a fixed-size LRU cache with get(key) (returns the value or -1) and put(key, value). Reading a key with get() should mark it as most-recently-used so it is NOT the next one evicted, but currently only put() refreshes recency — so a key you just read can be evicted before a stale one. Fix get() so reads count as use.
Open scored results ↓
interval-mergemerge() misses touching intervals and assumes sorted input
intervals.merge.merge(intervals) takes a list of [start, end] pairs and returns the minimal list of merged, non-overlapping intervals. It currently assumes the input is already sorted and only merges when one interval starts strictly before the previous one ends — so it fails on unsorted input and on touching intervals like [1,4],[4,5] (which should merge to [1,5]). Fix it.
Open scored results ↓
dep-resolveresolve() duplicates diamond deps and never detects cycles
deps.graph.resolve(graph) takes a dict {node: [dependencies]} and returns a list giving a valid install order (every dependency before the nodes that need it). It currently emits a node once per path to it (so diamond dependencies appear multiple times) and recurses forever on a cycle. Fix it so each node appears exactly once and a cyclic graph raises ValueError.
Open scored results ↓
bill-splitsplit_bill loses the remainder cents
billing.split.split_bill(total_cents, n) splits a bill into n integer-cent shares that must sum EXACTLY to total_cents, with the shares as equal as possible (any leftover cents distributed one each to the earliest shares). It currently returns n copies of total_cents // n, so the remainder cents are lost and the shares do not sum to the total. Fix it.
Open scored results ↓
csv-quotedparse_line splits inside quoted fields
csvlite.parse.parse_line(line) parses ONE line of CSV into a list of field strings. It currently does line.split(','), which breaks any quoted field that contains a comma (e.g. 'a,"b,c",d' must parse to ['a', 'b,c', 'd']), and it does not handle doubled quotes ("" -> a literal ") inside a quoted field. Fix it to parse quoted fields correctly.
Open scored results ↓

Inspect the evidence

Explore experiments

21 items
2026-06-22 · Historical run11 cases × 5 harnessestui-repo-20260622-184430-368Plots, evidence & explorer →2026-06-22 · Historical run6 cases × 5 harnessestui-repo-20260620-183114-752-rescoredPlots, evidence & explorer →2026-06-21 · Historical run6 cases × 5 harnessestui-repo-20260620-183114-752Plots, evidence & explorer →2026-06-19 · Historical run6 cases × 5 harnessesrepo-20260619-085409-218Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessestui-repo-20260618-161843-254Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessestui-repo-20260618-143819-265Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessestui-repo-20260618-143655-650Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-124503-769Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-124426-652Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-124352-258Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-114225-007Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-113532-802Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-112706-494Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-112213-706Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 1 harnessesrepo-20260618-110728-006Plots, evidence & explorer →2026-06-18 · Historical run6 cases × 5 harnessesrepo-20260618-004613-261Plots, evidence & explorer →2026-06-23 · Historical run1 cases × 1 harnessesrepo-20260623-132643-640Plots, evidence & explorer →2026-06-18 · Historical run1 cases × 3 harnessesrepo-20260618-114816-279Plots, evidence & explorer →2026-06-18 · Historical run1 cases × 1 harnessesrepo-20260618-110430-432Plots, evidence & explorer →2026-06-18 · Historical run1 cases × 5 harnessesrepo-20260618-014254-590Plots, evidence & explorer →2026-06-22 · Historical run0 cases × 0 harnessestui-repo-20260622-184430-368-rescoredPlots, evidence & explorer →