All test suites

Coding suite 1.0

OfficialFrozenWritten by BenchClub

Five small Python repos. The model works as an agent; hidden tests grade each try.

[email protected]

Run it

These questions were written by BenchClub. No public agent-coding benchmark runs locally without Docker yet, so this suite stays until one does. The model never sees the grading tests during a run, but they are public on GitHub. Results here are a signal, not a leaderboard.

What it asks

  • Agent coding

    5 · 3 times each · Not regraded

    Read a small Python repo, edit files and run tests, as an agent, within 15 turns.

    Source
    Written by BenchClub
    How chosen
    Five tasks from a one-file fix to a parser rewrite, written by BenchClub.
    Grading
    Hidden tests run after each try, in a sandbox.

Our reference runs

In calibration on Oct 27, Qwen3.6-35B-A3B solved four of the five tasks at least once; MiniCPM5-2B and Qwen3.5-2B solved one each. This suite shows a matrix of tasks, never a total score. Per-task results appear here when it goes on the site in mid-December.