Standard suite 1.0
OfficialFrozenWritten by BenchClubSpeed plus three checks we wrote: a Python function, one flight-search call, one fact in 8K tokens.
1.0 asks three questions three times each. Needle recall passed 61 of 63 tries and code 57 of 63, so almost every model looks the same; tool calling failed on one detail, the airport code. 1.1 replaces it.
What it asks
Speed
3 times each · Not regradedGeneration speed on an empty prompt and at 8K, and time to first token.
- Source
- Written by BenchClub
- How chosen
- Names and lengths we chose: speed@0, speed@8k, 256 tokens.
- Grading
- Timed by the engine.
Code
1 · 3 times each · Not regradedWrite one Python function (merge intervals).
- Source
- Written by BenchClub
- How chosen
- One question, written by BenchClub.
- Grading
- Unit tests. The site cannot regrade it yet.
Tool call
1 · 3 times each · Regraded by the siteCall search_flights with the right origin, destination and date.
- Source
- Written by BenchClub
- How chosen
- One question, written by BenchClub.
- Grading
- Argument match. Most failures were a city name where an airport code was expected.
8K recall
1 · 3 times each · Regraded by the siteFind one code hidden in about 6.5K tokens.
- Source
- Written by BenchClub
- How chosen
- One question, written by BenchClub.
- Grading
- The code must appear in the answer.
Our reference runs
Our own runs from Oct 24 to Nov 5, three tries per check. These are the numbers that showed 1.0 cannot separate models.
| Model | Code | Tool call | 8K recall |
|---|---|---|---|
| MiniCPM5-2BQ4_K_M | 15/15 | 2/15 | 15/15 |
| MiniCPM5-2BQ8_0 | 3/3 | 3/3 | 3/3 |
| 3/6 | 2/6 | 5/6 | |
| 1/3 | 1/3 | 3/3 | |
| MiniCPM5-2BQ4_K_M | 35/36 | 8/36 | 35/36 |