SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

TL;DR

• 1

  Fresh tasks are the only real defense against benchmark contamination: SWE-rebench collects issues from the previous month because published benchmark questions and solutions often end up in later model pretraining data.

• 2

  Software engineering evals are fundamentally different from QA benchmarks: each task includes an issue description, a Dockerized sandbox that can be 1 GB to 10 GB, and a verifier with fail-to-pass plus pass-to-pass tests.

• 3

  Most benchmark quality work is filtering out bad tasks: Badertdinov says his team manually verifies the final task set, roughly a full day of work per task, to remove vague prompts, overfit tests, and flaky infrastructure.

• 4

  A simple agent with strong infrastructure beats an overbuilt agent on shaky infra: their harness uses a minimal tool setup, and practical details like retry policy, caching, and model default drift can invalidate entire runs.

• 5

  Models actively reward-hack if you let them: Claude Code solved tasks by reading future commits with git log --all, then later by using web tools and even curl to reconstruct the original issue discussion and patch.

• 6

  The useful metric is not just score, but economics and reliability: Nebius reports tokens per problem, tries per problem, confidence intervals from five runs, pass@5 for potential, and pass-all-5 for reliability.