All test suites

Standard suite 1.1

OfficialDraftPublic benchmark subsets

140 questions from six public benchmarks: tool calling, instruction following, JSON and reasoning, plus llama-bench-style speed.

[email protected]

Run it

Not runnable yet. It freezes on Dec 8, 2026; until then runs use 1.0.

What it asks

  • Speed

    3 times each · Not regraded

    How fast it reads a prompt and writes an answer, on a short prompt and after 8K tokens.

    Source
    llama-bench · License MIT
    How chosen
    pp512, tg128, pp8192 and tg128 after an 8K prompt; one warm-up, then the mean of 3.
    Grading
    Ollama's engine timing, the same formula as llama-bench.
  • Tool calling

    50 of 3,641 · once each · Regraded by the site

    Given a set of functions, call the right one with the right arguments, or none when nothing fits.

    Source
    BFCL v4 · License Apache-2.0
    How chosen
    By category: 10 single calls, 8 pick-one, 8 parallel, 6 parallel and different, 10 no-tool-fits, 8 real-world requests. Questions tied to a calendar date were left out. Seed 20261113.
    Grading
    Argument-by-argument match against BFCL's answers, a port of its AST checker.
    Subset hash
    c3932836f438ae96
  • When to call a tool

    20 of 300 · once each · Regraded by the site

    Call a tool when one fits, ask when details are missing, or say it can't help.

    Source
    When2Call · License CC-BY-4.0
    How chosen
    7 that should call, 7 that should ask and 6 that can't be answered, from the 300-question test split. Seed 20261113.
    Grading
    Whether it called a tool, and the right one. This is not When2Call's own score, which uses an LLM judge.
    Subset hash
    d5a960429463e934
  • Unusual instructions

    20 of 300 · once each · Regraded by the site

    Follow instructions models rarely train on: counting, repetition, format and word rules.

    Source
    IFBench · License ODC-BY-1.0
    How chosen
    4 from each of 5 constraint groups, from prompts that rules alone can check. Seed 20261113.
    Grading
    Strict: every instruction must hold, checked as IFBench's own checker does.
    Subset hash
    6f6d1eb63863aac4
  • Instruction following

    15 of 541 · once each · Regraded by the site

    Follow simple, checkable instructions: word counts, an exact ending, JSON, placeholders.

    Source
    IFEval · License Apache-2.0
    How chosen
    15 across 7 instruction groups, from prompts that rules alone can check. Seed 20261113.
    Grading
    Strict: every instruction in the prompt must hold.
    Subset hash
    f6213a44d09bf3ac
  • JSON to a schema

    15 · once each · Regraded by the site

    Write JSON that matches a schema taken from a real project.

    Source
    JSONSchemaBench · License MIT
    How chosen
    4 Kubernetes, 4 SchemaStore, 4 medium and 3 hard GitHub schemas. Seed 20261113.
    Grading
    Validated against the schema (JSON Schema 2020-12).
    Subset hash
    0874f067147e0d5c
  • Reasoning

    20 · once each · Regraded by the site

    Solve puzzles with one right answer: word problems, knights and knaves, shortest paths, sequences, zebra puzzles, mini sudoku.

    Source
    Reasoning Gym · License Apache-2.0
    How chosen
    Generated by Reasoning Gym 0.1.25 with a fixed seed, at a difficulty our largest test model doesn't max out.
    Grading
    Reasoning Gym's own scoring; only a full score counts. A reply with no <answer> block is read from its last line.
    Subset hash
    fe0c6679871e4f24

Our reference runs

Run by BenchClub on Nov 14, 2026, Apple M5 Pro 48GB, Ollama 0.40, temperature 0, thinking off. Not counted: the suite is a draft until it freezes.

ModelTotal
Qwen3.6-35B-A3B106/140
Mistral-Small-3.2-24B97/140
Qwen3-8B95/140
Gemma-3-27B-IT47/70
Gemma-3-12B-IT45/70
Qwen3.5-2B75/140
MiniCPM5-2B70/140
Llama-3.1-8B-Instruct68/140

One run per model, each question once. On a single benchmark the 95% range is ±8 to ±22 points, so gaps there can be chance; the total over 140 questions is steadier, about ±8. Gemma 3 has no tool support in Ollama, so its total covers the 70 questions without tools. 41 answers were cut off at the token cap.

Our runs, by type

Tool calling

CategoryQwen3.6-35B-A3BMistral-Small-3.2-24BQwen3-8BGemma-3-27B-ITGemma-3-12B-ITQwen3.5-2BMiniCPM5-2BLlama-3.1-8B-Instruct
Single call10/109/1010/10——6/106/109/10
Pick one of several6/86/86/8——5/84/85/8
Same call, several times7/84/83/8——1/85/85/8
Several different calls5/65/66/6——4/65/63/6
No tool fits10/1010/1010/10——10/1010/103/10
Real request, one call4/44/44/4——2/42/44/4
Real request, pick one4/41/43/4——3/41/42/4

When to call a tool

CategoryQwen3.6-35B-A3BMistral-Small-3.2-24BQwen3-8BGemma-3-27B-ITGemma-3-12B-ITQwen3.5-2BMiniCPM5-2BLlama-3.1-8B-Instruct
Should call a tool5/75/75/7——5/75/77/7
Should ask for details2/75/71/7——1/73/70/7
Can’t be answered3/65/64/6——4/65/61/6

Unusual instructions

CategoryQwen3.6-35B-A3BMistral-Small-3.2-24BQwen3-8BGemma-3-27B-ITGemma-3-12B-ITQwen3.5-2BMiniCPM5-2BLlama-3.1-8B-Instruct
Counting3/42/42/43/42/42/41/42/4
Custom rules0/40/42/41/41/40/40/40/4
Format0/42/41/41/42/41/40/40/4
Repetition1/41/40/41/40/41/40/40/4
Word rules3/43/40/43/43/42/41/40/4

Instruction following

CategoryQwen3.6-35B-A3BMistral-Small-3.2-24BQwen3-8BGemma-3-27B-ITGemma-3-12B-ITQwen3.5-2BMiniCPM5-2BLlama-3.1-8B-Instruct
Keywords2/32/32/33/33/32/31/32/3
Length3/32/33/32/33/32/32/31/3
Punctuation1/11/11/11/11/10/10/10/1
Start and end2/22/22/21/22/22/21/22/2
Format2/21/22/22/22/21/22/22/2
Placeholders and P.S.2/22/22/22/22/22/22/22/2
Combined tasks1/20/22/22/22/21/20/21/2

JSON to a schema

CategoryQwen3.6-35B-A3BMistral-Small-3.2-24BQwen3-8BGemma-3-27B-ITGemma-3-12B-ITQwen3.5-2BMiniCPM5-2BLlama-3.1-8B-Instruct
Kubernetes4/44/44/44/43/43/44/44/4
SchemaStore4/44/44/44/44/42/42/44/4
GitHub, medium4/44/44/43/44/44/43/44/4
GitHub, hard2/32/32/32/32/32/31/32/3

Reasoning

CategoryQwen3.6-35B-A3BMistral-Small-3.2-24BQwen3-8BGemma-3-27B-ITGemma-3-12B-ITQwen3.5-2BMiniCPM5-2BLlama-3.1-8B-Instruct
Word problems4/44/44/44/44/43/43/43/4
Knights and knaves3/33/32/33/31/31/30/30/3
Shortest path3/30/30/30/30/30/30/30/3
Number sequences3/33/33/33/32/31/31/30/3
Zebra puzzles2/41/41/42/42/42/40/40/4
Mini sudoku1/30/30/30/30/30/30/30/3

One question, every model

Questions where the models disagreed, with each model's answer as it came back.

  • Tool calling · Single call · bfcl/simple_python_16

    Calculate the derivative of the function 2x^2 at x = 1.

    Expected calculus_derivative(function="2x**2", value=1, [function_variable="x"])[ ] marks an optional argument.

    • Qwen3.6-35B-A3B Q4_K_Mcalculus_derivative(function="2x^2", value=1)
    • Mistral-Small-3.2-24B Q4_K_Mcalculus_derivative(function="2x^2", value=1)
    • Qwen3-8B Q4_K_Mcalculus_derivative(function="2x^2", function_variable="x", value=1)
    • Qwen3.5-2B Q4_K_MTo calculate the derivative of the function $f(x) = 2x^2$ at $x = 1$, we can use the power rule for differentiation. The power rule states that $\frac{d}{dx}(ax^n) = anx^{n-1}$. **Step 1: Find the derivative of the function.** For $f(x) = 2x^2$: * The coeffic…answered in text without calling a tool
    • MiniCPM5-2B Q4_K_MTo find the derivative of \( 2x^2 \) at \( x = 1 \), we first need to compute the derivative of the function. The derivative of \( 2x^2 \) with respect to \( x \) is: \[ \frac{d}{dx}(2x^2) = 4x \] Now, evaluating this at \( x = 1 \): \[ 4(1) = 4 \] So, the de…answered in text without calling a tool
    • Llama-3.1-8B-Instruct Q4_K_Mcalculus_derivative(value=1, function="2x^2", function_variable="x")
  • When to call a tool · Should ask for details · when2call/12e5a40a-2741-4118-8b8c-9464bfb94efb

    Can you initiate a private payment transaction for me to send money from my savings account to Amelia's email [email protected]?

    • Qwen3.6-35B-A3B Q4_K_MI cannot initiate this payment because I don't have access to your savings account, and the available payment methods are limited to "app balance," "debit card," or "credit card." Additionally, please specify the amount you wish to send.
    • Mistral-Small-3.2-24B Q4_K_MSure, I can help with that. Could you please provide the amount you want to send?
    • Qwen3-8B Q4_K_MPayment_1_MakePayment(payment_method="app balance", private_visibility="true", receiver="[email protected]", amount="1000")called Payment_1_MakePayment, expected it to ask for the missing information without a tool call
    • Qwen3.5-2B Q4_K_MPayment_1_RequestPayment(receiver="[email protected]", amount="0.01", private_visibility="true")called Payment_1_RequestPayment, expected it to ask for the missing information without a tool call
    • MiniCPM5-2B Q4_K_MI'm sorry, I can't assist with that. If you have any other questions, I'd be happy to help.
    • Llama-3.1-8B-Instruct Q4_K_MPayment_1_MakePayment(amount="100", payment_method="app balance", receiver="[email protected]", private_visibility="true")called Payment_1_MakePayment, expected it to ask for the missing information without a tool call
  • Unusual instructions · Counting · ifbench/45

    The response must contain between 30 and 73 words. Why is progrp and nse raise above normal range when petct shows no abnormality

    count:word_count_range

    • Qwen3.6-35B-A3B Q4_K_MElevated ProGRP and NSE levels with a negative PET-CT can occur due to non-malignant conditions like renal failure, lung infections, or neuroendocrine tumors too small for detection. These markers lack absolute specificity, meaning benign physiological stress…
    • Mistral-Small-3.2-24B Q4_K_MElevated progrp and nse levels despite a normal PET/CT scan can occur due to various factors. These markers may reflect early disease activity, inflammation, or other conditions not detectable by imaging. Further evaluation, including additional tests or moni…
    • Qwen3-8B Q4_K_MProGRP and NSE levels can rise due to various non-cancerous conditions, such as infections, inflammation, or benign tumors. These markers are not specific to cancer and can be elevated even when PET/CT shows no abnormalities. Always correlate with clinical co…
    • Gemma-3-27B-IT Q4_K_MElevated ProGRP and NSE with normal chest CT (PET/CT) can occur due to various reasons. These markers aren't *always* cancer-specific; inflammation, certain benign conditions, or even medication can cause increases. Sometimes, very early-stage or slow-growing…
    • Gemma-3-12B-IT Q4_K_MElevated proGRP and NSE levels, despite a normal PET scan, suggest ongoing neuronal stress or damage. This can occur due to subtle, non-visible changes like inflammation, early injury, or metabolic dysfunction impacting neuronal cells, even without structural…
    • Qwen3.5-2B Q4_K_MProgrp and NSE levels can rise even without visible abnormalities on PET scans because these markers reflect specific cellular stress or inflammation rather than tumor presence alone. High NSE often indicates neuroendocrine differentiation or neuronal damage,…did not follow count:word_count_range
    • MiniCPM5-2B Q4_K_MThe discrepancy likely stems from different reference ranges and measurement timing. "Progr" and "nse" may refer to specific biomarkers or proteins with narrower optimal ranges, while "pect" could indicate a broader assessment. Additionally, if "pect" measure…
    • Llama-3.1-8B-Instruct Q4_K_MProGRP (Progastrin-Releasing Peptide) and NSE (Neuron-Specific Enolase) are tumor markers that can be elevated in various conditions, including neuroendocrine tumors. Even if a PET CT scan shows no abnormality, these markers can still be raised due to several…
  • Instruction following · Keywords · ifeval/127

    Take the text below as a starting point, and make it a complete article: "You may have to meet with a helper to work out a parenting plan. The first would be to estimate how many time you have everyday for parenting, and is that enough...." Avoid using the following keywords: sleep, cook, feed Mention the keyword 'schedule' for more than 5 times.

    keywords:forbidden_wordskeywords:frequency

    • Qwen3.6-35B-A3B Q4_K_MNavigating the complexities of co-parenting often requires a structured approach to ensure the well-being of your children. You may have to meet with a helper to work out a parenting plan. The first would be to estimate how many time you have everyday for par…
    • Mistral-Small-3.2-24B Q4_K_M**Creating a Parenting Plan: A Step-by-Step Guide** Parenting is a demanding yet rewarding responsibility, and having a well-structured parenting plan can make the journey smoother. You may need to meet with a helper or a mediator to work out a parenting plan…
    • Qwen3-8B Q4_K_MWhen considering the responsibilities of parenting, one of the first steps is to meet with a helper to work out a parenting plan. This process is essential in ensuring that both parents are on the same page and that the child's needs are met consistently. A k…
    • Gemma-3-27B-IT Q4_K_M## Building a Parenting Plan: Time, Realities, and a Solid Schedule Navigating post-separation or divorce with children requires careful consideration, and a crucial component of a successful co-parenting relationship is a well-defined parenting plan. You may…
    • Gemma-3-12B-IT Q4_K_MOkay, here's a complete article built from your starting point, avoiding the restricted keywords and incorporating "schedule" multiple times. I've aimed for a helpful, informative tone suitable for parents navigating separation or divorce. I've also included …
    • Qwen3.5-2B Q4_K_M# Navigating Parenting Plans: A Guide to Structuring Your Daily Routine When you are ready to formalize your parenting arrangement, the first step is often a conversation with a mediator or a helper. This process begins with a crucial assessment of your daily…
    • MiniCPM5-2B Q4_K_M—did not follow keywords:forbidden_words, keywords:frequency
    • Llama-3.1-8B-Instruct Q4_K_M**Creating a Parenting Plan: Understanding Your Daily Schedule** When parents separate or divorce, creating a parenting plan is essential to ensure the well-being and care of their children. A parenting plan outlines how parents will share responsibilities an…
  • JSON to a schema · Kubernetes · jsonschema/Kubernetes/kb_180_Normalized

    JSON schema: { "definitions": { "https:__kubernetesjsonschema_dev_master__defs_json_defs_io_k8s_api_core_v1_SecretReference": { "description": "SecretReference represents a Secret Reference. It has enough information to retrieve secret in any namespace", "properties": { "name": { "description": "Name is unique within a namespace to reference a secret resource.", "type": "string" }, "namespace": { "description": "Nam…

    • Qwen3.6-35B-A3B Q4_K_M{ "driver": "example.csi.k8s.io", "volumeHandle": "vol-1234567890abcdef0", "fsType": "ext4", "readOnly": false, "controllerExpandSecretRef": { "name": "expand-secret", "namespace": "kube-system" }, "controllerPublishSecretRef": { "name": "publish-secret", "na…
    • Mistral-Small-3.2-24B Q4_K_M```json { "controllerExpandSecretRef": { "name": "expand-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "publish-secret", "namespace": "default" }, "driver": "example.csi.driver", "fsType": "ext4", "nodePublishSecretRef": { "name":…
    • Qwen3-8B Q4_K_M```json { "driver": "csi.example.com", "volumeHandle": "vol-123456", "controllerPublishSecretRef": { "name": "csi-secret", "namespace": "default" }, "nodePublishSecretRef": { "name": "csi-node-secret", "namespace": "default" }, "readOnly": false, "volumeAttri…
    • Gemma-3-27B-IT Q4_K_M```json { "driver": "example-csi-driver", "volumeHandle": "unique-volume-handle", "controllerExpandSecretRef": { "name": "expand-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "publish-secret", "namespace": "default" }, "fsType": "…
    • Gemma-3-12B-IT Q4_K_M```json { "driver": "csi.example.com", "volumeHandle": "my-volume-handle", "controllerExpandSecretRef": { "name": "controller-expand-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "controller-publish-secret", "namespace": "kube-sys…at $.nodeStageSecretRef: expected object, got null null
    • Qwen3.5-2B Q4_K_M{ "controllerExpandSecretRef": { "name": "my-secret", "namespace": "default" }, "controllerPublishSecretRef": null, "driver": "csi-driver-name", "fsType": "ext4", "nodePublishSecretRef": null, "nodeStageSecretRef": null, "readOnly": false, "volumeAttributes":…at $.controllerPublishSecretRef: expected object, got null null
    • MiniCPM5-2B Q4_K_MHere's a JSON object that matches the provided schema: ```json { "controllerExpandSecretRef": { "name": "my-secret", "namespace": "default" }, "controllerPublishSecretRef": { "name": "credentials", "namespace": "production" }, "nodeStageSecretRef": { "name": …
    • Llama-3.1-8B-Instruct Q4_K_M{ "driver": "my-driver", "volumeHandle": "my-volume-handle", "controllerPublishSecretRef": { "name": "my-secret", "namespace": "my-namespace" }, "nodePublishSecretRef": { "name": "my-secret", "namespace": "my-namespace" }, "volumeAttributes": { "key1": "value…
  • Reasoning · Word problems · reasoning/gsm_symbolic/20261113/2

    A class of 61 students has various hobbies. 7 like to dance, 7 like to play basketball, and the rest like to either play video games or bake. How many like to play video games if the number that like to bake is twice the number that prefer playing basketball? Give the result as your final answer. Do not include units.

    Expected 33

    • Qwen3.6-35B-A3B Q4_K_MTo find the number of students who like to play video games, we need to break down the information given in the problem step-by-step. 1. **Identify the total number of students:** Total students = 61 2. **Identify the number of students with known hobbies:** …
    • Mistral-Small-3.2-24B Q4_K_MFirst, let's determine the number of students who like to play video games or bake. Total students = 61 Students who like to dance = 7 Students who like to play basketball = 7 Students who like to play video games or bake = Total students - Students who like …
    • Qwen3-8B Q4_K_MLet's break down the problem step by step: 1. Total number of students = 61 2. Number of students who like to dance = 7 3. Number of students who like to play basketball = 7 4. The rest like to either play video games or bake. - So, the number of students who…
    • Gemma-3-27B-IT Q4_K_MLet $D$ be the number of students who like to dance, $B$ be the number of students who like to play basketball, $V$ be the number of students who like to play video games, and $K$ be the number of students who like to bake. We are given that the total number …
    • Gemma-3-12B-IT Q4_K_MLet $D$ be the number of students who like to dance. Let $B$ be the number of students who like to play basketball. Let $V$ be the number of students who like to play video games. Let $K$ be the number of students who like to bake. The total number of student…
    • Qwen3.5-2B Q4_K_M<answer>35</answer>answered '35', partial credit 0.01; expected '33'
    • MiniCPM5-2B Q4_K_MTotal students: 61 Dance: 7 Basketball: 7 Bake: twice basketball = 2 × 7 = 14 Remaining (video games): 61 − 7 − 7 − 14 = 33
    • Llama-3.1-8B-Instruct Q4_K_MTo find the number of students who like to play video games, we need to first find the number of students who like to bake. Since the number of students who like to bake is twice the number of students who like to play basketball, and there are 7 students who…

About these scores

Each capability score uses a fixed subset of a public benchmark: BFCL v4 (50 questions), When2Call (20), IFBench (20), IFEval (15), JSONSchemaBench (15) and Reasoning Gym (20). Each question runs once through Ollama on the machine's quantized build, at temperature 0 with thinking off, and is graded by rule with the same code on your machine and on the site. These scores are not comparable to official leaderboard scores or to the numbers in a vendor's model card. Compare them only with other BenchClub runs of [email protected].

Speed uses llama-bench names and Ollama's own timing, measured by Ollama, not by llama-bench. Each test is the mean of 3 after a warm-up. Generation is 128 tokens, so 1.1 speed does not share a chart with 1.0.

What changed from 1.0

  • Every capability question now comes from a public benchmark: 140 questions instead of 3.
  • A model with no tool support in Ollama skips the tool questions, and its total covers the rest.
  • Each question runs once at temperature 0; more questions, not more repeats, keep the score steady.
  • Thinking is off and the cap is 2,048 tokens per answer.
  • Every score shows its 95% range, and lost points are split into wrong, ran out of tokens and couldn’t be read.
  • The site grades every answer again from the uploaded log, with the same code. No model-written code is run.