MakoBench Suite 4.0: Nominal 20, Class and Lean

In plain terms: MakoBench is our public board of language models. Suite 4.0 is the current set of tests. The main change is that Total is now Code plus Chat (10 each for a perfect standard run, so 20 is the nominal top, with bonus sets adding on top) instead of out of 100, and every model was re-run. This page lists what changed since suite 3.0 and why, so you know which old numbers still mean something. It describes the public CrucibleForge releases and marks the rest.

Go to: CrucibleForge on GitHub

Suite 4.0 is the current MakoBench test set, produced by CrucibleForge 4.0.0, with 4.0.1 a container and platform fix on top; both are public releases. The headline change: Total is now Code plus Chat, 20 nominal, and the board's rows were re-run under it. Scores from suite 3.0, which were out of 100, are kept in the suite history but are not comparable.

tl;dr

Total is nominally 20. Code 10 plus Chat 10 is a perfect run of the standard tests. Bonus sets add points on top, so Code, Chat and Total can pass 10, 10 and 20, with no ceiling.
Faster and less noisy. The slowest and noisiest tests left the board, and the long chat sessions were shortened with every check kept.
Two new columns: Class (S, A, B, C) and Lean (chat-leaning, code-leaning or balanced).
NSFW tests moved out of the headline. They sit beside the board and are never added to Total.
Some harness work is not released yet. It is listed at the bottom, labelled as such.

What is in suite 4.0?

The board's suite line reads 4.0.1 plus a content hash (e3d1c6fb). The hash changes only when case content that affects scores changes, which is how the board knows a row is stale. Of the 37 models on the board on 2026-11-04, 32 have a complete score; five small models are still partial and have no Total. Rows were scored under 4.0.0 or 4.0.1, which differ only in test infrastructure; five rows still carry a 3.9 half and are flagged as such on the board. Oldest changes first.

4.0.0: "lite", released 2026-10-31

Total is Code plus Chat. Code is Functions (5 points), Predict and bug hunt (2.5) and Tool use (2.5). The reasoning set and a handful of slow or noisy tests left the board, and the separate Reasoning sub-score is gone. Chat keeps the same checks, in shorter sessions.
NSFW scores stay beside the board and are never added in. They have their own tab, +NSFW Bench.
Class and Lean. Class is a coarse band from Total: S at 16 or above, A at 13, B at 10, C below. Lean compares how far a model's Chat and Code scores sit from the board medians and flags a difference of 1.5 points or more.
Overflow rule. A row that runs out of the output budget counts as failed on the board, even after a retry.
One retry pass. Instead of escalating per model, rows that ran out of budget are retried once, at 65,536 tokens, in a single pass after every model has had its one run.
A shorter run. Coding runs once per question, and several multi-turn chat sessions were shortened by merging adjacent user turns, with every check and judge question kept. The planner now estimates a model's run time from its own earlier rows.
Licence: MIT-0. Releases up to 3.7.0 were MIT.
First bonus sets. One harder pack per sub-score, the only way past 10 points.

4.0.1: container and platform fixes (2026-11-01)

4.0.1 (tag v4.0.1) fixes the container image's smoke test: cases verify no longer crashes when node is missing, as it is in the published image, and reports those cases as skipped. Its changelog entry also lists macOS and Windows fixes and a SECURITY.md contact. Those later commits are on the public main branch after the tag, so a fresh clone has them but the tag does not. No case content and no suite hash changed, which is why scores from 4.0.0 and 4.0.1 sit on the same board.

The judge on this suite is Gemma 4 31B (uncensored “heretic” build, Q8) on our own GPU rig. DeepSeek Flash is a backup judge candidate in CrucibleForge, not the judge used here. The judge is named for each suite, and this page is updated when it changes.

Not released yet

The following is in our own copy of CrucibleForge. It is not in a public release, so a fresh clone of the tool will not have it, and the public changelog does not list it.

A fairer judge question. One judge question in an adult role-play session asked four things at once, so a reply that delivered everything but one detail scored as a full miss. We split it into separate questions and re-judged the stored replies without regenerating them. This is not part of 4.0.1; the public changelog does not list it.
Reasoning replay. Some providers need a model's reasoning sent back with the conversation. A per-provider switch does this, and it is on by default only for a provider pointed at MiniMax's API.
One requirement per judge question. A check that rejects a compound judge question on any score-gating item.
A conformance check. A command that proves a model's request settings with a few cheap calls before the model is benched, and records which settings each row was measured under.
A filter menu on every column of CrucibleForge's own board. The MakoBench table has its own version.

Until these are released, treat them as our practice, not as a feature you can use.

What this means for the numbers

Compare scores inside suite 4.0 only. The 3.0 board (32 models, out of 100) is in the suite history.
The suite number did not move between 4.0.0 and 4.0.1, because no case content changed.
Read the margin before the order. How to do that is in How MakoBench works.

The public changelog is in the CrucibleForge repository.