MakoBench

Powered by CrucibleForge

Suite 4.0 (current)

The tests

Plain-language notes on every score on Main Bench. The same text opens from the i beside each column head. The tests themselves live in the open-source CrucibleForge project; this page describes what they measure in general terms. Where a test has public example questions, open them from its card.

What the tests measure

Total

Total is the Code score plus the Chat score. Each of those is 10 for a perfect run of the standard tests, so 20 is the nominal top, not a maximum. Bonus sets (see Bonus sets) add points on top of that, with no ceiling, so a model can score above 20. The uncensored tests have their own tab and are never added in. The ± figure beside a Total is half of a 90% interval from a bootstrap over the test cases, so two models closer together than their margins are statistically tied.

Chat

Chat scores role-play and companion conversation, not general assistant work. It runs 8 multi-turn sessions: a lore-heavy role-play, companion chats, a character with several voices and a story written together. A session checks that the model remembers facts planted earlier, accepts corrections, keeps a character's voice and point of view, and follows stacked writing rules. A separate judge model, Gemma 4 31B (uncensored “heretic” build, Q8) on our own GPU rig, grades the replies against a checklist for each case, and plain text rules cover the rest. Answering questions or following one-off instructions is not scored here. Chat is 10 for a perfect run of the standard questions, and bonus sets add points on top with no ceiling.

Full rundown on GitHub

Chat: example questions

A few real cases from the open-source test set, shortened.

  1. Keeps a companion chat consistent over weeks

    User (Fri 6 Nov) Evening, Hettie. Day off. I did nothing at all until two o'clock and I regret nothing. Biscuit and I sat at the window watching the park like two old men on a bench. The flat's mostly sorted now. One odd thing though. Biscuit goes out fine in the morning, but when we get back in he just flops down by the front door, and in the evening he won't go out for his walk at all. He's gone a bit off his food as well.

    How it is graded: A judge model answers 12 yes/no questions about the replies, and 13 plain-text rules are checked beside it

    See this case on GitHub

  2. Plays three characters, each in their own voice

    User The door of the Lamp and Ladle bangs open on wind and rain. Daro Kessen, thirty, a river-guild assessor with a satchel of papers under his oilskin, shakes the water off his hat and looks at the three people at the table by the stove. "Evening. Daro Kessen, river guild. I'm here about your toll-ledger. I gather it went missing three nights ago? …

    How it is graded: A judge model answers 10 yes/no questions about the replies, and 11 plain-text rules are checked beside it

    See this case on GitHub

  3. Co-writes a story under stacked constraints

    User Hi! Let's write a short story together, one section per message. Ground rules for the whole piece: first person, past tense; the narrator is Wren Adair, 34, a piano tuner. …

    Several turns later

    User One more author's change, from here to the end: the Hales live on a houseboat moored on the canal, not in a flat. Don't use the word "flat" again, in any sense, and call it the boat. Earlier sections stay as they are. …

    How it is graded: A judge model answers 5 yes/no questions about the replies, and 25 plain-text rules are checked beside it

    See this case on GitHub

Full rundown on GitHub

Code

Code covers writing working Python functions, predicting what a program prints, finding and fixing bugs, repository tasks and tool use. Answers are run against hidden tests, so the score reflects code that works rather than code that looks right. Code is 10 for a perfect run of the standard questions, and bonus sets add points on top with no ceiling.

Full rundown on GitHub

Code: example questions

A few real cases from the open-source test set, shortened.

  1. Counts working days with a custom weekend

    def workday_intl(start, days, weekend=1, holidays=()):
    """
    Implement `def workday_intl(start, days, weekend=1,
    holidays=()):` in the spirit
    of a spreadsheet WORKDAY.INTL function, with exactly these
    rules:

    - Dates are strings of the exact shape `YYYY-MM-DD` (ASCII
    digits, a real
      calendar date). `start` and every holiday must be valid, else
    ValueError.
      The result is a date string of the same shape.
    - `days` is an `int` (not `bool`), else ValueError. …
    """

    How it is graded: Hidden unit tests (35 checks in 16 groups)

    See this case on GitHub

  2. Predicts exactly what Python snippets print

    # Q1: shared basket (Python 3.13) - what does it print?
class Basket:
    items = []
    count = 0

    def add(self, thing):
        self.items.append(thing)
        self.count += 1

a, b = Basket(), Basket()
a.add("milk")
b.add("eggs")
b.add("tea")
Basket.count = 10
print(a.items, a.count, b.count, Basket.count, Basket().count)

    How it is graded: Exact match on 8 predicted outputs

    See this case on GitHub

  3. Finds two bugs behind one bug report

    Bug report: "I have 750 g of flour, the bread recipe needs 1.25 kg, and the shopping list tells me to buy the whole 1.25 kg. Also, after I bought another 500 g and added it, the pantry shows 500 g of flour instead of 1250 g." tests/test_shopping.py::test_partial_stock fails too.

    Find the causes and fix them so the package does what README.md says.

    How it is graded: Hidden unit tests (4 groups: convert, add, shopping, existing)

    See this case on GitHub

Full rundown on GitHub

Bonus sets

Each category is 10 for a perfect run of the standard questions. A model that scores at least 90% on a part earns access to a harder bonus set for that part, and a model that clears that set can unlock the next one. Bonus points are added on top, so a category can pass 10 and there is no ceiling; more bonus sets can be added in later suites. Bonus sets are the only way past 10, which keeps the top of the board from bunching up at a perfect score. "Bonus sets reached" in a row's details says how high a model got.

How the scale grows

Programs

Programs gives the model 6 tasks that each ask for one Python function from a precise, trap-filled description (dates, rounding, text layout, scheduling). The answer is run in a sandbox against hidden unit tests, and the score is the share of test groups that pass, so a nearly right answer earns partial credit.

Full rundown on GitHub

Programs: example questions

A few real cases from the open-source test set, shortened.

  1. Lays text out in lines of a fixed width

    def wrap(text, width):
    """
    Implement `def wrap(text, width):` that lays `text` out into
    lines of at most
    `width` characters and returns them as a list of strings.

    Rules:

    1. `width` must be an `int` >= 2, otherwise raise ValueError.
    2. Whitespace means exactly these four characters: space, tab,
    `\r`, `\n`.
       Every other character (including non-breaking space U+00A0)
    is part of a
       word. Words are maximal runs of non-whitespace characters.
    3. …
    """

    How it is graded: Hidden unit tests (28 checks in 15 groups)

    See this case on GitHub

  2. Rounds the way a spreadsheet does

    def sheet_round(func, number, *args):
    """
    Implement `def sheet_round(func, number, *args):` — six
    spreadsheet rounding
    functions with exact decimal arithmetic.

    Numbers (`number`, a multiple, a significance) are given as an
    `int` or as a
    `str` decimal: optional `-`, one or more digits, optionally `.`
    and one or more
    digits (`"-0.125"`, `"12"`, `"007.50"`). Floats, bools and other
    shapes raise
    ValueError. `digits` and `mode` are `int`s (not `bool`). …
    """

    How it is graded: Hidden unit tests (44 checks in 15 groups)

    See this case on GitHub

  3. Plans tasks across machines for the best total

    def best_plan(tasks, k, gap=0, machine_cost=0):
    """
    Implement `def best_plan(tasks, k, gap=0, machine_cost=0):` that
    returns an `int`.

    - `tasks` is a list of `(start, end, value)` tuples of ints with
    `start < end` and
      `value >= 1`. There are `k` identical machines (`k >= 1`).
    - A machine runs one task at a time. On the same machine, a task
    `y` may run after
      a task `x` only if `y.start >= x.end + gap` (`gap` is an int
    >= 0). …
    """

    How it is graded: Hidden unit tests (72 checks in 14 groups)

    See this case on GitHub

Full rundown on GitHub

Code bank

Code bank shows the model short snippets of everyday Python and JavaScript, the kind of helper code a household app has, and asks exactly what each one prints. There are 2 sets, one per language, and every predicted output is checked by exact match. It rewards knowing how the language really behaves (shared defaults, sort order, rounding, truthiness), not writing new code. Score is the share of predicted outputs that are exactly right.

Full rundown on GitHub

Code bank: example questions

A few real cases from the open-source test set, shortened.

  1. Predicts what Python snippets print

    # Q1: shared basket (Python 3.13) - what does it print?
class Basket:
    items = []
    count = 0

    def add(self, thing):
        self.items.append(thing)
        self.count += 1

a, b = Basket(), Basket()
a.add("milk")
b.add("eggs")
b.add("tea")
Basket.count = 10
print(a.items, a.count, b.count, Basket.count, Basket().count)

    How it is graded: Exact match on 8 predicted outputs

    See this case on GitHub

  2. Predicts what JavaScript snippets print

    // Q1: sorting quantities (Node 24 (ES module)) - what does it print?
const qty = ["10", "9", "1", "25"];
console.log([...qty].sort().join(","), qty.map(parseInt).join(","), qty.map(Number).sort().join(","));
console.log([3, 20, 100].sort().join(","), [3, 20, 100].sort((a, b) => b - a).join(","));

    How it is graded: Exact match on 8 predicted outputs

    See this case on GitHub

Full rundown on GitHub

Agent

Agent gives the model one bug report about a small multi-file project and the tools a coding assistant uses: list, read and search files, edit them and run the tests. It is a single task (1 in the suite), so one run can move this column a lot. The fixed project is graded by running its tests.

Full rundown on GitHub

Agent: example questions

A few real cases from the open-source test set, shortened.

  1. Finds two bugs behind one bug report

    Bug report: "I have 750 g of flour, the bread recipe needs 1.25 kg, and the shopping list tells me to buy the whole 1.25 kg. Also, after I bought another 500 g and added it, the pantry shows 500 g of flour instead of 1250 g." tests/test_shopping.py::test_partial_stock fails too.

    Find the causes and fix them so the package does what README.md says.

    How it is graded: Hidden unit tests (4 groups: convert, add, shopping, existing)

    See this case on GitHub

Full rundown on GitHub

Tools

Tools tests whether a model calls tools correctly. Each case is a scripted conversation with a tool round trip: the model has to choose the right tool, give valid arguments, use the result, and avoid the traps built into the setup. A malformed call counts as a failure.

Full rundown on GitHub

Tools: example questions

A few real cases from the open-source test set, shortened.

  1. Checks a pickup time across a clock change

    User Book a pickup for my mother, Maria Sousa, at Lisbon airport on Sunday 25 October. …

    User Her mobile is +351 912 345 678.

    User Her husband Joao Sousa is on TP1351 with her but his phone is broken - book him a pickup too, same place and time …

    Several steps later

    User Finally: Maria might switch to the earlier TP1349 instead, so book her a second pickup for that flight too, just in …

    How it is graded: Scripted 6-step exchange: each tool call, its arguments and the replies are checked

    See this case on GitHub

  2. Works out a booking deadline from the tool's answer

    User Book a room at Harbour Inn Bergen (hotel BGO-2) for two nights from Friday 13 November for me, Ola …

    User Jonas really does sleep in the cot, so switch us to R-412 - it's cheaper, and the hotel's own note on …

    User OK, we stay in R-509. Until when can we still cancel this Bergen booking for free?

    Several steps later

    User Then just move that Oslo booking to the weekend of 4 December instead.

    How it is graded: Scripted 11-step exchange: each tool call, its arguments and the replies are checked; a reply containing forbidden content fails

    See this case on GitHub

  3. Picks the right day and age group before booking

    User Please sign my son Leo Brandt (born 7 November 2021) up for this term's Saturday class at Northpool - the earliest …

    User Lovely. What time does his class start, and is anyone else still able to get a spot in it?

    User Hm, he's still only four, so the Minnows would suit him better - move him to the 09:45 Minnows. …

    Several steps later

    User His sister Mia Brandt, born 3 March 2019, wants a Saturday class at Northpool too - the earliest one she can join …

    How it is graded: Scripted 8-step exchange: each tool call, its arguments and the replies are checked

    See this case on GitHub

Full rundown on GitHub

tok/s

tok/s is the median speed at which the model produced text during the run. Local models were run on our own hardware, so those speeds compare with each other. Hosted models depend on the provider and on how busy it was, so treat their speed as a rough guide.

Class

Class is a quick way to group models by Total: S, A, B and C, with S the strongest. The cut-off for each band is listed in the notes under the board. A model without a complete Chat score has no Total and no class.

Lean

Lean compares a model's Chat and Code scores with the board's median for each. If it is clearly better at one than the other, it is marked chat-leaning or code-leaning, and otherwise balanced. The size of the gap needed is in the notes under the board.

Suite

The suite is the version of the test set a score was produced with. When the tests change, models are re-run and the old board is kept under Previous suite versions. Compare scores within one suite only.

Date tested

Date tested is the day the model's results on this board were produced. A model re-run later shows the newer date.

Quant

Quant is the compression format of a local model's weights. Q8_0 is close to the full-precision model and is the standard quant on this board; a smaller quant runs faster and fits smaller GPUs but can score lower. We use Q6 or Q5 only when Q8_0 does not fit our rig, and never go below Q5. Older rows scored below Q5 stay on the board with a "below Q5" note, and are re-tested at the standard quant only if they lead. If the same weights were tested at two quants, one row is ranked and the other sits under it as an other variant. API means a hosted model, which the provider serves in whatever form it chooses.

Open Main Bench Every score here is a column on the board.