All test suites

Standard suite 1.0

OfficialFrozenWritten by BenchClub

Speed plus three checks we wrote: a Python function, one flight-search call, one fact in 8K tokens.

[email protected]

Run it

1.0 asks three questions three times each. Needle recall passed 61 of 63 tries and code 57 of 63, so almost every model looks the same; tool calling failed on one detail, the airport code. 1.1 replaces it.

What it asks

  • Speed

    3 times each · Not regraded

    Generation speed on an empty prompt and at 8K, and time to first token.

    Source
    Written by BenchClub
    How chosen
    Names and lengths we chose: speed@0, speed@8k, 256 tokens.
    Grading
    Timed by the engine.
  • Code

    1 · 3 times each · Not regraded

    Write one Python function (merge intervals).

    Source
    Written by BenchClub
    How chosen
    One question, written by BenchClub.
    Grading
    Unit tests. The site cannot regrade it yet.
  • Tool call

    1 · 3 times each · Regraded by the site

    Call search_flights with the right origin, destination and date.

    Source
    Written by BenchClub
    How chosen
    One question, written by BenchClub.
    Grading
    Argument match. Most failures were a city name where an airport code was expected.
  • 8K recall

    1 · 3 times each · Regraded by the site

    Find one code hidden in about 6.5K tokens.

    Source
    Written by BenchClub
    How chosen
    One question, written by BenchClub.
    Grading
    The code must appear in the answer.

Our reference runs

Our own runs from Oct 24 to Nov 5, three tries per check. These are the numbers that showed 1.0 cannot separate models.

ModelCodeTool call8K recall
MiniCPM5-2B15/152/1515/15
MiniCPM5-2B3/33/33/3
Qwen3.5-2B3/62/65/6
Qwen3.6-35B-A3B1/31/33/3
MiniCPM5-2B35/368/3635/36