Coding suite 1.0
OfficialFrozenWritten by BenchClubFive small Python repos. The model works as an agent; hidden tests grade each try.
These questions were written by BenchClub. No public agent-coding benchmark runs locally without Docker yet, so this suite stays until one does. The model never sees the grading tests during a run, but they are public on GitHub. Results here are a signal, not a leaderboard.
What it asks
Agent coding
5 · 3 times each · Not regradedRead a small Python repo, edit files and run tests, as an agent, within 15 turns.
- Source
- Written by BenchClub
- How chosen
- Five tasks from a one-file fix to a parser rewrite, written by BenchClub.
- Grading
- Hidden tests run after each try, in a sandbox.
Our reference runs
In calibration on Oct 27, Qwen3.6-35B-A3B solved four of the five tasks at least once; MiniCPM5-2B and Qwen3.5-2B solved one each. This suite shows a matrix of tasks, never a total score. Per-task results appear here when it goes on the site in mid-December.