Finally a good benchmark (DeepSWE)

TL;DR

• 1

  DeepSWE looks more like how developers actually use coding agents — prompts are short and behavior-focused rather than over-specified, so models have to explore the repo and figure out the fix instead of following a giant hint-filled brief.

• 2

  GPT-5.5 posts a clear win over Opus 4.7 here — on the DeepSWE leaderboard, GPT-5.5 Extra High hits about 70% while Opus 4.7 trails by more than 15 points, matching what Berman says he’s hearing from engineers.

• 3

  The verifier quality is a huge deal — DeepSWE reports a 0.3% false-positive rate and 1.1% false-negative rate versus SWE-bench Pro’s 8.5% and 24%, which means the benchmark is much less likely to misgrade real solutions.

• 4

  Cost and token efficiency make the gap look worse for Anthropic — Berman highlights GPT-5.5 at roughly $5.80 per trial versus Opus 4.7 near $16, with median output tokens around 16,000 for GPT-5.5 versus 60,000 for Opus 4.7 in the MiniSuite setup.

• 5

  DeepSWE creates more meaningful model separation — instead of bunching everyone together, the benchmark spreads models out from GPT-5.5 near the top to Claude Haiku 4.5 at 0%, making it easier to tell who’s actually better.

• 6

  The benchmark also surfaces behavioral differences between model families — Claude often misses one branch of multi-part requirements, while GPT-5.5 is described as more literal and consistent about implementing all stated behaviors.