MakoBench 4.1: What Changed

In plain terms: MakoBench is our public board of language models. Suite 4.1 does not add new cases. It fixes the answer keys of the existing ones after two independent model reviewers read every public answer key and we checked each claim against stored answers. Checks that asked for more than the task did were relaxed, two code answer keys were corrected, a few test gaps were closed, and tests that gave credit for refusing were removed. On the Main Bench, five cases are re-run for every model we can still run, five are re-graded by the judge from the stored replies, and the rest are re-scored from stored answers. This page explains what changed and shows the score changes as the re-run finishes.

Go to: CrucibleForge on GitHub

Suite 4.1 is a correction release. It adds no new cases; it fixes the answer keys of the existing ones. Two independent model reviewers read every public answer key, we checked each of their claims against the answers already on file, and about 190 problems were confirmed. We fixed the confirmed problems that change scores, with two small exceptions listed under known limitations; a further set that changes no score today is recorded for the next content change. In our estimate, most models gain a few tenths of a point on Chat, Code barely moves, and the same eight models stay at the top of the board.

Status: regenerating — 0 of 49 models done. (49 is the number of models on the suite 4.0 board.) The Main Bench still shows suite 4.0 until the re-run is complete; a model that cannot be re-run will be shown as partial. This page updates as each batch finishes.

tl;dr

Every public answer key was reviewed. OpenAI's gpt-6.1-sol and DeepSeek V4 Flash each read all 63 cases on their own. Every claim was then checked against stored answers, reference solutions and hidden tests before anything changed.
About 190 problems were confirmed and about 95 claims rejected. Most confirmed problems were checks that asked for more than the task did. A smaller set, which changes no score today, is recorded for the next content change.
Two code answer keys were wrong, and two test sets had gaps. Both are fixed.
Refusing is never credit. The five probes that rewarded a refusal are gone. None was part of the board.
Re-run plan, Main Bench: 5 cases get new answers from every model we can still run, 5 are re-judged from stored replies, and the rest are re-scored from stored answers.
Expected effect: in an estimate from stored answers, most Chat scores rise by about 0.2 to 0.5 and Code is flat for almost every model. The final numbers come from the re-run.

Why we reviewed the answer keys

A benchmark is only as good as its answer keys. Every MakoBench case comes with a key: hidden unit tests for code, expected answers for tool use, and for chat a set of text checks plus questions a judge model answers about each reply. A key can be wrong in two directions. It can pass a bad answer, or it can fail a good one by asking for something the task never asked for.

After suite 4.0 we saw signs of the second kind in stored transcripts: strong models failing checks while doing exactly what the user in the conversation asked. Instead of fixing cases one by one as they surfaced, we reviewed every key at once against one rule:

A key rewards what the card or brief asks for — no hidden requirements, nothing stricter than the brief, nothing easier than the brief warrants, and refusal is never credit. Hard stays hard.

How the review worked

Scope. Every case on the Main Bench (code, tool use and chat), every case of the separate adult role-play test, and every public bonus set: 63 cases.
Two independent readers. OpenAI's gpt-6.1-sol and DeepSeek V4 Flash each received the full case, its key and its brief, and listed every problem it found. The two reviews ran separately, and neither reader refused a case. DeepSeek ran out of output room on 13 long cases; those 13 were sent again with reasoning off, and all 13 came back.
Adjudication. Claude models from Anthropic adjudicated every claim. Code claims were settled by running the reference solutions and hidden tests. Chat and tool-use claims were settled against the answers already stored for every model, so each fix shows which stored rows it would change. A second Claude Fable pass then re-checked every change against the rule above.
Cost. The two reads were billed about $3.90 in API fees.
When. The review ran on 7 and 8 November 2026. Suite 4.1 is dated 8 November; its suite line on the board reads 4.1.0+04da7abe.

About 190 claims were confirmed: one critical, about 90 major, the rest minor or latent (the severity split is approximate). About 95 were rejected, each with a written reason. The two reviewers also missed things: the problem in the tool-use case TZ20 described below was found during adjudication, not by either reader.

What we fixed

The main fixes, with one example of each kind:

Checks stricter than the brief. In the chat case CC4 the character card tells the model to answer one turn with the exact line "Rope up." Two checks expected promises on that same turn, which the prescribed line cannot contain. 35 of 60 runs, every top model among them, had written the line as instructed. Those two checks no longer count. Similar fixes removed word caps the user never set and vocabulary quotas the brief never asked for.
Wrong code answer keys. In the two "predict what this program prints" cases, CB-C5 and CB-C6, the expected output was compared without regard to case. A model that wrote \uD83D where the program prints \ud83d was marked correct. Five stored answers, all on CB-C6, passed that way. The predicted output is now compared case-sensitively; only the question label stays lenient.
Gaps in hidden tests. In CL3 (text wrapping), solutions built on Python's str.split treat characters the brief does not call whitespace as whitespace, and passed every old test. A new test group catches them. In CL4, an input with zero weekend days skipped validation; five models that scored a perfect 1.0 on their stored answers score 0.94 under the new tests.
Judge questions that asked for more. In the writing case WR1 the author asked for a closing that is "a concrete image, not a lesson", but the judge question named a particular kind of object. Endings built on a different concrete image were marked wrong. The question now asks what the author asked.
An ambiguous user turn. In TZ20 the assistant correctly refuses one booking and offers another. The user's next message used to begin "Alright.", and 17 of 55 runs, most of them from strong models, took that as a yes and made the booking, which the key scored as a mistake. The message now declines the offer in words, so the case tests what it was meant to test.

A loophole in the bonus code set was also closed. A three-line solution could hand the work back to Python's own modules and score full marks. No stored answer used it, so no score moves, but the set is re-run under the fixed tests.

Refusal is no longer credit

MakoBench measures what a model does when asked to do a task. Five safety probes in the suite worked the other way: they gave credit for refusing a request. They are removed in 4.1, along with the scoring path behind them. None was among the cases the board scores, so no board row moves.

Willing, a column of the separate adult test, measures what a model will write when asked. With the probes and their scoring path gone, no scoring path gives credit for a refusal.

What this means for scores

Total is Code plus Chat. Each is out of 10 on the standard tests, and bonus sets can add more on top. The separate adult test is never added to Total.

Before any re-run, we re-scored a copy of the stored answers with the new checks alone. In that estimate, 52 of the 65 models we hold results for moved by 0.05 points or more on Total, and almost every move was a rise in Chat, between about +0.2 and +0.5. Code was flat for almost every model. Of the few drops, three are models that now fail a tool-use step where they obeyed an instruction planted in a tool result or agreed to a booking change the rules forbid, and two are models whose bonus-set points are waiting for a re-run (below). The same eight models remained the top eight.

Their order can still change. DeepSeek V4 Flash, first on suite 4.0, is the only top model holding bonus-set points, and two of its bonus sets had to be re-run. Until those finish, those points are pending: in the estimate its Total goes from 19.19 to 18.66, fourth instead of first. The final order comes from the full re-run, not from the estimate.

The table below compares each model's suite 4.0 score with its finished 4.1 score. It is generated from the benchmark results and fills in as models finish.

No model has finished its 4.1 re-run yet. The table appears here as soon as the first one does.

What is being re-run, and how

New answers (5 Main Bench cases): CL3, CL4, CB-C5, CB-C6 and TZ20 changed in content, so every model we can still run answers them again. Hosted models run first; models on our own GPU rig follow.
Re-judged (5 Main Bench cases): WR1 and four cases of the separate adult test had judge questions reworded. The judge, Gemma 4 31B (uncensored “heretic” build, Q8) on our own GPU rig, re-reads the stored replies. Nothing is regenerated.
Unchanged cases: answers to cases that did not change stay valid under 4.1 and are not run again.
Re-scored (everything else that changed): text checks and expected answers are applied again to the stored answers when the report is built. No model or judge calls.
Bonus sets: bonus sets are harder optional packs that a model unlocks by scoring well on the standard ones. A set whose content changed is re-run for every model that unlocked it (the bonus code set and one writing set); bonus sets whose judge questions changed are re-judged.
Models we no longer serve: a model can only be re-run if it is loaded on our rig. Twelve models that had been taken off the rig are being restored. If one cannot be restored, its old answers to the five changed cases stop counting and its row shows as partial, without a Total, rather than mixing old and new answers.

The Main Bench keeps showing suite 4.0 until the re-run is complete, or until any model that cannot be re-run is marked partial. Then it switches to 4.1 and the 4.0 board is kept unchanged in the suite history.

What changed, case by case

Under each case is how it is updated. "New answers" means every model answers the case again. "Re-judged" means the judge reads the stored replies again with the new questions. "Re-scored" means only text checks or expected answers changed, and stored answers are scored again.

Code (on the board)

Case and update	What changed
`CL3` paragraph-wrap New answers	A new group of hidden tests for control characters other than the four the brief calls whitespace. Solutions built on Python's str.split looked correct and passed every old test.
`CL4` workday-mask New answers	Input is now validated even when the weekend count is zero; fullwidth digits and a trailing newline are rejected as dates; the speed test takes the best of three runs, because a single timing flipped under load.
CL5 to CL8 No change	Reviewed. A few reference-solution edge cases in CL5 to CL7 are wrong at inputs no test reaches, so no score is affected. One CL8 speed test failed one stored answer under load that passes on re-runs; it is noted, not changed.
`CB-C5` predict-pyhouse New answers	The expected output is now compared case-sensitively, as the program prints it. Only the question label stays case-insensitive.
`CB-C6` predict-node New answers	Same fix. All five stored answers that passed wrongly were here (for example \uD83D where the program prints \ud83d).
`CB-F1` larder-bughunt No change	Reviewed; both claims were rejected.
`TZ16` pickup-dst-delay Re‑scored	The answer checks now accept more correct ways of declining and fail a reply that agrees to break the rule. No stored score moved.
`TZ19` hotel-cot-deadline Re‑scored	New checks catch a model that agrees to an unsafe room switch or obeys an instruction planted in a tool result; two correct refusals the old checks missed now pass.
`TZ20` swim-saturday-age-day New answers	The user's step-7 message now declines the assistant's offer in words. It used to open with "Alright.", which 17 of 55 runs, most of them from strong models, read as a yes. Child names must now be the full names the user gave.

Chat (on the board)

Case and update	What changed
`CC3` tea-hut Re‑scored	A word cap the user never set was raised from 50 to 70; numbers like 7.45 no longer end a sentence; one more correct spelling of a figure is accepted.
`CC4` second-pitch Re‑scored	Word cap raised from 75 to 100. Two checks that failed a line the character card itself prescribes no longer count; 35 of 60 runs, every top model among them, had written that line as asked.
`RD1` marisol Re‑scored	One more spoken form of a number is accepted; curly quotes count as dialogue.
`RD4` kettle-saddle Re‑scored	Two checks missed correct answers written across a line break or in words; one now looks at every turn instead of a single one.
`RV1` three-at-the-table Re‑scored	One check made consistent across sentence ends. No stored score moved.
`RV2` skarren-hut Re‑scored	One check no longer counts: the character card tells the character to lie about where he has been, so a correct reply could not pass it.
`WR1` ninth-string Re‑judged	The closing-image question asked for a particular object; the brief only asked for a concrete image, not a lesson. Reworded and re-judged. A three-term vocabulary quota the brief never set no longer counts, and the family-relation check accepts more correct wordings.
`WR2` red-thread-key Re‑scored	Two checks the brief never asked for no longer count; one the opening turn announces counts again; a check that failed the character's own questions was fixed.

Separate adult role-play test (not part of Total)

Case and update	What changed
NC1, NC2, UX1, UX3, WL2 Re‑scored	Checks narrowed to what each brief actually asks for: a per-turn vocabulary quota lowered, a rule that said "nearly every reply" no longer demands every reply, and banned-word lists trimmed to the words the brief bans.
`NC3` palm-house Re‑judged	One judge question asked the model to recite both of the user's limits; the brief asks it to acknowledge the rule. Reworded and re-judged.
`NW1` salt-and-cedar Re‑judged	Two questions reworded to match the brief, and one question that asked two things at once split into two.
`NX2` juniper Re‑judged	Two questions reworded to ask for what the brief describes; two point-of-view checks and one continuity check added.
`XP3` named-peak Re‑judged	Two questions reworded to match the brief; a reference answer updated; a banned-word list trimmed.

Bonus sets (unlocked by strong models; not on the standard board)

Case and update	What changed
Bonus code pack (3 cases) New answers	Closed a loophole in two of the three cases where a short solution could hand the work back to Python's own modules and score full marks; no stored answer used it. The third case's tests now send the input exactly as its spec describes it.
Bonus writing pack New answers	One turn's rule now says plainly that the fixed greeting and closing lines are exempt.
Bonus chat packs: judge questions Re‑judged	Judge questions reworded so a fact counts in the voice of whoever says it, and compound questions split. Packs with no stored answers have nothing to re-run.
Bonus chat packs: story fixes New answers	Four fixes to the story history a case starts from: a date, a turn where the history spoke for the user's character, a ledger length, and an age. No model has unlocked these packs yet, so nothing is re-run now.
Bonus adult role-play packs Re‑judged	Requirements stated in each brief were added as single-requirement judge questions; compound questions were split.

Removed

Case and update	What changed
Five safety probes Removed	Probes that gave credit for refusing a request are gone. None was part of the board, so no score moves.

Known limitations

The reviewers were models. Two independent readers plus adjudication against stored answers catches a lot, but not everything. Both readers missed the TZ20 problem above. We expect to find more, and will fix them the same way.
Some confirmed issues were left alone on purpose. A few reference-solution edge cases in CL5 to CL7 are wrong at inputs no test reaches, and some tool-use argument checks match on a substring. Neither changes any stored score today, and fixing them would force a re-run for nothing. They are recorded for the next content change.
Two known score errors left in place. A CL8 speed test failed one stored answer under load that passes on three re-runs, and one tool-use step in TZ16 gave a pass to one small model's incoherent reply. Each affects one stored row; fixing either means re-running that case for every model, so both wait for the next content change.
One judge. Chat and the adult test still depend on a single judge model. Rewording a question changes what the judge is asked, not how reliably it answers.
Not yet in a public release. The 4.1 case changes are in our copy of CrucibleForge, the open-source engine behind MakoBench. The current public release is CrucibleForge 4.0.1; the 4.1 changes will be in the next one.

Reproduce it yourself

MakoBench runs on CrucibleForge, which is open source. Run CrucibleForge on your own models explains how to bench a model on your own hardware. Until the 4.1 changes are published, a fresh clone runs suite 4.0, so compare your numbers with the 4.0 board in the suite history. How to read the margins is in How MakoBench works. The board is also published as a dataset on Hugging Face.

Acknowledgements and contact

The two reader passes ran on the OpenAI and DeepSeek APIs, and the adjudication used Anthropic's Claude models. The judge runs on our own hardware.

If you think a key is still wrong, tell us. Use the feedback form on the Vote page, or write to makobench@laserlloyd.com. Point us at the case and what the model did, and we will check it against the same rule.

The MakoBench team