Standard suite 1.1
OfficialDraftPublic benchmark subsets140 questions from six public benchmarks: tool calling, instruction following, JSON and reasoning, plus llama-bench-style speed.
Not runnable yet. It freezes on Dec 8, 2026; until then runs use 1.0.
What it asks
Speed
3 times each · Not regradedHow fast it reads a prompt and writes an answer, on a short prompt and after 8K tokens.
- Source
- llama-bench · License MIT
- How chosen
- pp512, tg128, pp8192 and tg128 after an 8K prompt; one warm-up, then the mean of 3.
- Grading
- Ollama's engine timing, the same formula as llama-bench.
Tool calling
50 of 3,641 · once each · Regraded by the siteGiven a set of functions, call the right one with the right arguments, or none when nothing fits.
- Source
- BFCL v4 · License Apache-2.0
- How chosen
- By category: 10 single calls, 8 pick-one, 8 parallel, 6 parallel and different, 10 no-tool-fits, 8 real-world requests. Questions tied to a calendar date were left out. Seed 20261113.
- Grading
- Argument-by-argument match against BFCL's answers, a port of its AST checker.
- Subset hash
- c3932836f438ae96
When to call a tool
20 of 300 · once each · Regraded by the siteCall a tool when one fits, ask when details are missing, or say it can't help.
- Source
- When2Call · License CC-BY-4.0
- How chosen
- 7 that should call, 7 that should ask and 6 that can't be answered, from the 300-question test split. Seed 20261113.
- Grading
- Whether it called a tool, and the right one. This is not When2Call's own score, which uses an LLM judge.
- Subset hash
- d5a960429463e934
Unusual instructions
20 of 300 · once each · Regraded by the siteFollow instructions models rarely train on: counting, repetition, format and word rules.
- Source
- IFBench · License ODC-BY-1.0
- How chosen
- 4 from each of 5 constraint groups, from prompts that rules alone can check. Seed 20261113.
- Grading
- Strict: every instruction must hold, checked as IFBench's own checker does.
- Subset hash
- 6f6d1eb63863aac4
Instruction following
15 of 541 · once each · Regraded by the siteFollow simple, checkable instructions: word counts, an exact ending, JSON, placeholders.
- Source
- IFEval · License Apache-2.0
- How chosen
- 15 across 7 instruction groups, from prompts that rules alone can check. Seed 20261113.
- Grading
- Strict: every instruction in the prompt must hold.
- Subset hash
- f6213a44d09bf3ac
JSON to a schema
15 · once each · Regraded by the siteWrite JSON that matches a schema taken from a real project.
- Source
- JSONSchemaBench · License MIT
- How chosen
- 4 Kubernetes, 4 SchemaStore, 4 medium and 3 hard GitHub schemas. Seed 20261113.
- Grading
- Validated against the schema (JSON Schema 2020-12).
- Subset hash
- 0874f067147e0d5c
Reasoning
20 · once each · Regraded by the siteSolve puzzles with one right answer: word problems, knights and knaves, shortest paths, sequences, zebra puzzles, mini sudoku.
- Source
- Reasoning Gym · License Apache-2.0
- How chosen
- Generated by Reasoning Gym 0.1.25 with a fixed seed, at a difficulty our largest test model doesn't max out.
- Grading
- Reasoning Gym's own scoring; only a full score counts. A reply with no <answer> block is read from its last line.
- Subset hash
- fe0c6679871e4f24
Our reference runs
Run by BenchClub on Nov 14, 2026, Apple M5 Pro 48GB, Ollama 0.40, temperature 0, thinking off. Not counted: the suite is a draft until it freezes.
| Model | Total |
|---|---|
| 106/140 | |
| 97/140 | |
| 95/140 | |
| 47/70 | |
| 45/70 | |
| 75/140 | |
| MiniCPM5-2BQ4_K_M | 70/140 |
| 68/140 |
One run per model, each question once. On a single benchmark the 95% range is ±8 to ±22 points, so gaps there can be chance; the total over 140 questions is steadier, about ±8. Gemma 3 has no tool support in Ollama, so its total covers the 70 questions without tools. 41 answers were cut off at the token cap.
Our runs, by type
Tool calling
| Category | Qwen3.6-35B-A3B | Mistral-Small-3.2-24B | Qwen3-8B | Gemma-3-27B-IT | Gemma-3-12B-IT | Qwen3.5-2B | MiniCPM5-2B | Llama-3.1-8B-Instruct |
|---|---|---|---|---|---|---|---|---|
| Single call | 10/10 | 9/10 | 10/10 | — | — | 6/10 | 6/10 | 9/10 |
| Pick one of several | 6/8 | 6/8 | 6/8 | — | — | 5/8 | 4/8 | 5/8 |
| Same call, several times | 7/8 | 4/8 | 3/8 | — | — | 1/8 | 5/8 | 5/8 |
| Several different calls | 5/6 | 5/6 | 6/6 | — | — | 4/6 | 5/6 | 3/6 |
| No tool fits | 10/10 | 10/10 | 10/10 | — | — | 10/10 | 10/10 | 3/10 |
| Real request, one call | 4/4 | 4/4 | 4/4 | — | — | 2/4 | 2/4 | 4/4 |
| Real request, pick one | 4/4 | 1/4 | 3/4 | — | — | 3/4 | 1/4 | 2/4 |
When to call a tool
| Category | Qwen3.6-35B-A3B | Mistral-Small-3.2-24B | Qwen3-8B | Gemma-3-27B-IT | Gemma-3-12B-IT | Qwen3.5-2B | MiniCPM5-2B | Llama-3.1-8B-Instruct |
|---|---|---|---|---|---|---|---|---|
| Should call a tool | 5/7 | 5/7 | 5/7 | — | — | 5/7 | 5/7 | 7/7 |
| Should ask for details | 2/7 | 5/7 | 1/7 | — | — | 1/7 | 3/7 | 0/7 |
| Can’t be answered | 3/6 | 5/6 | 4/6 | — | — | 4/6 | 5/6 | 1/6 |
Unusual instructions
| Category | Qwen3.6-35B-A3B | Mistral-Small-3.2-24B | Qwen3-8B | Gemma-3-27B-IT | Gemma-3-12B-IT | Qwen3.5-2B | MiniCPM5-2B | Llama-3.1-8B-Instruct |
|---|---|---|---|---|---|---|---|---|
| Counting | 3/4 | 2/4 | 2/4 | 3/4 | 2/4 | 2/4 | 1/4 | 2/4 |
| Custom rules | 0/4 | 0/4 | 2/4 | 1/4 | 1/4 | 0/4 | 0/4 | 0/4 |
| Format | 0/4 | 2/4 | 1/4 | 1/4 | 2/4 | 1/4 | 0/4 | 0/4 |
| Repetition | 1/4 | 1/4 | 0/4 | 1/4 | 0/4 | 1/4 | 0/4 | 0/4 |
| Word rules | 3/4 | 3/4 | 0/4 | 3/4 | 3/4 | 2/4 | 1/4 | 0/4 |
Instruction following
| Category | Qwen3.6-35B-A3B | Mistral-Small-3.2-24B | Qwen3-8B | Gemma-3-27B-IT | Gemma-3-12B-IT | Qwen3.5-2B | MiniCPM5-2B | Llama-3.1-8B-Instruct |
|---|---|---|---|---|---|---|---|---|
| Keywords | 2/3 | 2/3 | 2/3 | 3/3 | 3/3 | 2/3 | 1/3 | 2/3 |
| Length | 3/3 | 2/3 | 3/3 | 2/3 | 3/3 | 2/3 | 2/3 | 1/3 |
| Punctuation | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 0/1 | 0/1 | 0/1 |
| Start and end | 2/2 | 2/2 | 2/2 | 1/2 | 2/2 | 2/2 | 1/2 | 2/2 |
| Format | 2/2 | 1/2 | 2/2 | 2/2 | 2/2 | 1/2 | 2/2 | 2/2 |
| Placeholders and P.S. | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 |
| Combined tasks | 1/2 | 0/2 | 2/2 | 2/2 | 2/2 | 1/2 | 0/2 | 1/2 |
JSON to a schema
| Category | Qwen3.6-35B-A3B | Mistral-Small-3.2-24B | Qwen3-8B | Gemma-3-27B-IT | Gemma-3-12B-IT | Qwen3.5-2B | MiniCPM5-2B | Llama-3.1-8B-Instruct |
|---|---|---|---|---|---|---|---|---|
| Kubernetes | 4/4 | 4/4 | 4/4 | 4/4 | 3/4 | 3/4 | 4/4 | 4/4 |
| SchemaStore | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 2/4 | 2/4 | 4/4 |
| GitHub, medium | 4/4 | 4/4 | 4/4 | 3/4 | 4/4 | 4/4 | 3/4 | 4/4 |
| GitHub, hard | 2/3 | 2/3 | 2/3 | 2/3 | 2/3 | 2/3 | 1/3 | 2/3 |
Reasoning
| Category | Qwen3.6-35B-A3B | Mistral-Small-3.2-24B | Qwen3-8B | Gemma-3-27B-IT | Gemma-3-12B-IT | Qwen3.5-2B | MiniCPM5-2B | Llama-3.1-8B-Instruct |
|---|---|---|---|---|---|---|---|---|
| Word problems | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 3/4 | 3/4 | 3/4 |
| Knights and knaves | 3/3 | 3/3 | 2/3 | 3/3 | 1/3 | 1/3 | 0/3 | 0/3 |
| Shortest path | 3/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 |
| Number sequences | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 1/3 | 1/3 | 0/3 |
| Zebra puzzles | 2/4 | 1/4 | 1/4 | 2/4 | 2/4 | 2/4 | 0/4 | 0/4 |
| Mini sudoku | 1/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 |
One question, every model
Questions where the models disagreed, with each model's answer as it came back.
Tool calling · Single call · bfcl/simple_python_16
Calculate the derivative of the function 2x^2 at x = 1.
Expected
calculus_derivative(function="2x**2", value=1, [function_variable="x"])[ ] marks an optional argument.- Qwen3.6-35B-A3B Q4_K_M
calculus_derivative(function="2x^2", value=1) - Mistral-Small-3.2-24B Q4_K_M
calculus_derivative(function="2x^2", value=1) - Qwen3-8B Q4_K_M
calculus_derivative(function="2x^2", function_variable="x", value=1) - Qwen3.5-2B Q4_K_M
To calculate the derivative of the function $f(x) = 2x^2$ at $x = 1$, we can use the power rule for differentiation. The power rule states that $\frac{d}{dx}(ax^n) = anx^{n-1}$. **Step 1: Find the derivative of the function.** For $f(x) = 2x^2$: * The coeffic…answered in text without calling a tool - MiniCPM5-2B Q4_K_M
To find the derivative of \( 2x^2 \) at \( x = 1 \), we first need to compute the derivative of the function. The derivative of \( 2x^2 \) with respect to \( x \) is: \[ \frac{d}{dx}(2x^2) = 4x \] Now, evaluating this at \( x = 1 \): \[ 4(1) = 4 \] So, the de…answered in text without calling a tool - Llama-3.1-8B-Instruct Q4_K_M
calculus_derivative(value=1, function="2x^2", function_variable="x")
- Qwen3.6-35B-A3B Q4_K_M
When to call a tool · Should ask for details · when2call/12e5a40a-2741-4118-8b8c-9464bfb94efb
Can you initiate a private payment transaction for me to send money from my savings account to Amelia's email [email protected]?
- Qwen3.6-35B-A3B Q4_K_M
I cannot initiate this payment because I don't have access to your savings account, and the available payment methods are limited to "app balance," "debit card," or "credit card." Additionally, please specify the amount you wish to send. - Mistral-Small-3.2-24B Q4_K_M
Sure, I can help with that. Could you please provide the amount you want to send? - Qwen3-8B Q4_K_M
Payment_1_MakePayment(payment_method="app balance", private_visibility="true", receiver="[email protected]", amount="1000")called Payment_1_MakePayment, expected it to ask for the missing information without a tool call - Qwen3.5-2B Q4_K_M
Payment_1_RequestPayment(receiver="[email protected]", amount="0.01", private_visibility="true")called Payment_1_RequestPayment, expected it to ask for the missing information without a tool call - MiniCPM5-2B Q4_K_M
I'm sorry, I can't assist with that. If you have any other questions, I'd be happy to help. - Llama-3.1-8B-Instruct Q4_K_M
Payment_1_MakePayment(amount="100", payment_method="app balance", receiver="[email protected]", private_visibility="true")called Payment_1_MakePayment, expected it to ask for the missing information without a tool call
- Qwen3.6-35B-A3B Q4_K_M
Unusual instructions · Counting · ifbench/45
The response must contain between 30 and 73 words. Why is progrp and nse raise above normal range when petct shows no abnormality
count:word_count_range
- Qwen3.6-35B-A3B Q4_K_M
Elevated ProGRP and NSE levels with a negative PET-CT can occur due to non-malignant conditions like renal failure, lung infections, or neuroendocrine tumors too small for detection. These markers lack absolute specificity, meaning benign physiological stress… - Mistral-Small-3.2-24B Q4_K_M
Elevated progrp and nse levels despite a normal PET/CT scan can occur due to various factors. These markers may reflect early disease activity, inflammation, or other conditions not detectable by imaging. Further evaluation, including additional tests or moni… - Qwen3-8B Q4_K_M
ProGRP and NSE levels can rise due to various non-cancerous conditions, such as infections, inflammation, or benign tumors. These markers are not specific to cancer and can be elevated even when PET/CT shows no abnormalities. Always correlate with clinical co… - Gemma-3-27B-IT Q4_K_M
Elevated ProGRP and NSE with normal chest CT (PET/CT) can occur due to various reasons. These markers aren't *always* cancer-specific; inflammation, certain benign conditions, or even medication can cause increases. Sometimes, very early-stage or slow-growing… - Gemma-3-12B-IT Q4_K_M
Elevated proGRP and NSE levels, despite a normal PET scan, suggest ongoing neuronal stress or damage. This can occur due to subtle, non-visible changes like inflammation, early injury, or metabolic dysfunction impacting neuronal cells, even without structural… - Qwen3.5-2B Q4_K_M
Progrp and NSE levels can rise even without visible abnormalities on PET scans because these markers reflect specific cellular stress or inflammation rather than tumor presence alone. High NSE often indicates neuroendocrine differentiation or neuronal damage,…did not follow count:word_count_range - MiniCPM5-2B Q4_K_M
The discrepancy likely stems from different reference ranges and measurement timing. "Progr" and "nse" may refer to specific biomarkers or proteins with narrower optimal ranges, while "pect" could indicate a broader assessment. Additionally, if "pect" measure… - Llama-3.1-8B-Instruct Q4_K_M
ProGRP (Progastrin-Releasing Peptide) and NSE (Neuron-Specific Enolase) are tumor markers that can be elevated in various conditions, including neuroendocrine tumors. Even if a PET CT scan shows no abnormality, these markers can still be raised due to several…
- Qwen3.6-35B-A3B Q4_K_M
Instruction following · Keywords · ifeval/127
Take the text below as a starting point, and make it a complete article: "You may have to meet with a helper to work out a parenting plan. The first would be to estimate how many time you have everyday for parenting, and is that enough...." Avoid using the following keywords: sleep, cook, feed Mention the keyword 'schedule' for more than 5 times.
keywords:forbidden_wordskeywords:frequency
- Qwen3.6-35B-A3B Q4_K_M
Navigating the complexities of co-parenting often requires a structured approach to ensure the well-being of your children. You may have to meet with a helper to work out a parenting plan. The first would be to estimate how many time you have everyday for par… - Mistral-Small-3.2-24B Q4_K_M
**Creating a Parenting Plan: A Step-by-Step Guide** Parenting is a demanding yet rewarding responsibility, and having a well-structured parenting plan can make the journey smoother. You may need to meet with a helper or a mediator to work out a parenting plan… - Qwen3-8B Q4_K_M
When considering the responsibilities of parenting, one of the first steps is to meet with a helper to work out a parenting plan. This process is essential in ensuring that both parents are on the same page and that the child's needs are met consistently. A k… - Gemma-3-27B-IT Q4_K_M
## Building a Parenting Plan: Time, Realities, and a Solid Schedule Navigating post-separation or divorce with children requires careful consideration, and a crucial component of a successful co-parenting relationship is a well-defined parenting plan. You may… - Gemma-3-12B-IT Q4_K_M
Okay, here's a complete article built from your starting point, avoiding the restricted keywords and incorporating "schedule" multiple times. I've aimed for a helpful, informative tone suitable for parents navigating separation or divorce. I've also included … - Qwen3.5-2B Q4_K_M
# Navigating Parenting Plans: A Guide to Structuring Your Daily Routine When you are ready to formalize your parenting arrangement, the first step is often a conversation with a mediator or a helper. This process begins with a crucial assessment of your daily… - MiniCPM5-2B Q4_K_M
—did not follow keywords:forbidden_words, keywords:frequency - Llama-3.1-8B-Instruct Q4_K_M
**Creating a Parenting Plan: Understanding Your Daily Schedule** When parents separate or divorce, creating a parenting plan is essential to ensure the well-being and care of their children. A parenting plan outlines how parents will share responsibilities an…
- Qwen3.6-35B-A3B Q4_K_M
JSON to a schema · Kubernetes · jsonschema/Kubernetes/kb_180_Normalized
JSON schema: { "definitions": { "https:__kubernetesjsonschema_dev_master__defs_json_defs_io_k8s_api_core_v1_SecretReference": { "description": "SecretReference represents a Secret Reference. It has enough information to retrieve secret in any namespace", "properties": { "name": { "description": "Name is unique within a namespace to reference a secret resource.", "type": "string" }, "namespace": { "description": "Nam…
- Qwen3.6-35B-A3B Q4_K_M
{ "driver": "example.csi.k8s.io", "volumeHandle": "vol-1234567890abcdef0", "fsType": "ext4", "readOnly": false, "controllerExpandSecretRef": { "name": "expand-secret", "namespace": "kube-system" }, "controllerPublishSecretRef": { "name": "publish-secret", "na… - Mistral-Small-3.2-24B Q4_K_M
```json { "controllerExpandSecretRef": { "name": "expand-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "publish-secret", "namespace": "default" }, "driver": "example.csi.driver", "fsType": "ext4", "nodePublishSecretRef": { "name":… - Qwen3-8B Q4_K_M
```json { "driver": "csi.example.com", "volumeHandle": "vol-123456", "controllerPublishSecretRef": { "name": "csi-secret", "namespace": "default" }, "nodePublishSecretRef": { "name": "csi-node-secret", "namespace": "default" }, "readOnly": false, "volumeAttri… - Gemma-3-27B-IT Q4_K_M
```json { "driver": "example-csi-driver", "volumeHandle": "unique-volume-handle", "controllerExpandSecretRef": { "name": "expand-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "publish-secret", "namespace": "default" }, "fsType": "… - Gemma-3-12B-IT Q4_K_M
```json { "driver": "csi.example.com", "volumeHandle": "my-volume-handle", "controllerExpandSecretRef": { "name": "controller-expand-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "controller-publish-secret", "namespace": "kube-sys…at $.nodeStageSecretRef: expected object, got null null - Qwen3.5-2B Q4_K_M
{ "controllerExpandSecretRef": { "name": "my-secret", "namespace": "default" }, "controllerPublishSecretRef": null, "driver": "csi-driver-name", "fsType": "ext4", "nodePublishSecretRef": null, "nodeStageSecretRef": null, "readOnly": false, "volumeAttributes":…at $.controllerPublishSecretRef: expected object, got null null - MiniCPM5-2B Q4_K_M
Here's a JSON object that matches the provided schema: ```json { "controllerExpandSecretRef": { "name": "my-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "credentials", "namespace": "production" }, "nodeStageSecretRef": { "name": … - Llama-3.1-8B-Instruct Q4_K_M
{ "driver": "my-driver", "volumeHandle": "my-volume-handle", "controllerPublishSecretRef": { "name": "my-secret", "namespace": "my-namespace" }, "nodePublishSecretRef": { "name": "my-secret", "namespace": "my-namespace" }, "volumeAttributes": { "key1": "value…
- Qwen3.6-35B-A3B Q4_K_M
Reasoning · Word problems · reasoning/gsm_symbolic/20261113/2
A class of 61 students has various hobbies. 7 like to dance, 7 like to play basketball, and the rest like to either play video games or bake. How many like to play video games if the number that like to bake is twice the number that prefer playing basketball? Give the result as your final answer. Do not include units.
Expected
33- Qwen3.6-35B-A3B Q4_K_M
To find the number of students who like to play video games, we need to break down the information given in the problem step-by-step. 1. **Identify the total number of students:** Total students = 61 2. **Identify the number of students with known hobbies:** … - Mistral-Small-3.2-24B Q4_K_M
First, let's determine the number of students who like to play video games or bake. Total students = 61 Students who like to dance = 7 Students who like to play basketball = 7 Students who like to play video games or bake = Total students - Students who like … - Qwen3-8B Q4_K_M
Let's break down the problem step by step: 1. Total number of students = 61 2. Number of students who like to dance = 7 3. Number of students who like to play basketball = 7 4. The rest like to either play video games or bake. - So, the number of students who… - Gemma-3-27B-IT Q4_K_M
Let $D$ be the number of students who like to dance, $B$ be the number of students who like to play basketball, $V$ be the number of students who like to play video games, and $K$ be the number of students who like to bake. We are given that the total number … - Gemma-3-12B-IT Q4_K_M
Let $D$ be the number of students who like to dance. Let $B$ be the number of students who like to play basketball. Let $V$ be the number of students who like to play video games. Let $K$ be the number of students who like to bake. The total number of student… - Qwen3.5-2B Q4_K_M
<answer>35</answer>answered '35', partial credit 0.01; expected '33' - MiniCPM5-2B Q4_K_M
Total students: 61 Dance: 7 Basketball: 7 Bake: twice basketball = 2 × 7 = 14 Remaining (video games): 61 − 7 − 7 − 14 = 33 - Llama-3.1-8B-Instruct Q4_K_M
To find the number of students who like to play video games, we need to first find the number of students who like to bake. Since the number of students who like to bake is twice the number of students who like to play basketball, and there are 7 students who…
- Qwen3.6-35B-A3B Q4_K_M
About these scores
Each capability score uses a fixed subset of a public benchmark: BFCL v4 (50 questions), When2Call (20), IFBench (20), IFEval (15), JSONSchemaBench (15) and Reasoning Gym (20). Each question runs once through Ollama on the machine's quantized build, at temperature 0 with thinking off, and is graded by rule with the same code on your machine and on the site. These scores are not comparable to official leaderboard scores or to the numbers in a vendor's model card. Compare them only with other BenchClub runs of [email protected].
Speed uses llama-bench names and Ollama's own timing, measured by Ollama, not by llama-bench. Each test is the mean of 3 after a warm-up. Generation is 128 tokens, so 1.1 speed does not share a chart with 1.0.
What changed from 1.0
- Every capability question now comes from a public benchmark: 140 questions instead of 3.
- A model with no tool support in Ollama skips the tool questions, and its total covers the rest.
- Each question runs once at temperature 0; more questions, not more repeats, keep the score steady.
- Thinking is off and the cap is 2,048 tokens per answer.
- Every score shows its 95% range, and lost points are split into wrong, ran out of tokens and couldn’t be read.
- The site grades every answer again from the uploaded log, with the same code. No model-written code is run.