How MakoBench Works: Open LLM Benchmark Explained

Go to: CrucibleForge on GitHub

MakoBench is the public name of our model board. The engine that produces every number on it is CrucibleForge, an open-source benchmark tool (MIT-0, no conditions) that you can run on your own machine against any OpenAI-compatible endpoint. We run the board. You can run the tool.

What is MakoBench?

A board of language models, all scored the same way, side by side. At the time of writing it lists 49 models, one row per model: 24 that we run on our own GPU rig and 25 hosted ones reached through their APIs. They were tested between 2026-11-01 and 2026-11-07. 41 have a complete score and a rank; 5 are still partial and have no Total, and 8 are provisional (see "How do I read a score?"). Other quants and serving tiers of the same weights sit under their main row and are not ranked. For what we test and what we leave out, read What MakoBench tests, and what it doesn't.

We built it because most public leaderboards answer "which model is smartest?", and we wanted something narrower: which model is good at the two jobs we actually use models for, writing working code and holding a conversation, and how far apart the choices really are. You can sort, filter and export every column, so you can ask your own version of that question.

CrucibleForge is the engine

CrucibleForge is the tool underneath. It generates the answers, grades them, works out the margins and renders the board. It is free, open source and on GitHub: github.com/LaserLloyd/CrucibleForge. The licence is MIT No Attribution (MIT-0), so you can use, change and ship it with no conditions.

CrucibleForge is the method and the code. MakoBench is our board: the models we chose, the hardware we ran them on, and the notes we add. If you doubt a score, you can run the same tool on the same model and check.

What does it run on?

One GPU rig for local models, and plain API calls for hosted ones. CrucibleForge talks to anything that speaks the OpenAI chat format: llama.cpp, LM Studio, Ollama, vLLM, or a hosted API. A run takes roughly 25 to 30 minutes per local model on a fast GPU, judging included. The machine that runs CrucibleForge does not need a GPU; point it at a hosted API and the whole board runs over the network.

Speed (tok/s) only compares between models run on the same hardware. The local rows share our rig, so their speeds compare. A hosted model's speed depends on its provider and how busy it was.

What does it test?

Two categories make up the headline, 10 points each on the standard tests.

Code, 10 on the standard tests

Code is worth 10 points: Functions 5, Predict and bug hunt 2.5, Tool use 2.5. Each is graded by running the answer, not by reading it. The board's table breaks the same work into four columns, shown as a percentage of what each part scored: Programs, Code bank, Agent and Tools. The columns help you see where a model is strong. Total uses the weights above.

What the four columns cover:

• Programs. Write a Python function from a description. Hidden unit tests run in a sandbox, and partial credit is the share of test groups that pass.
• Code bank. Predict exactly what short Python and JavaScript snippets print. Every answer is checked by exact match, so it rewards knowing how the language really behaves.
• Agent. A bug hunt done the way a coding assistant works: the model gets tools to list, read and search a small project, edit it and run the tests. It is one task, so this column is the noisiest.
• Tools. Scripted tool-calling conversations with traps built in, such as a wrong tool or a bad argument.

Chat, 10 on the standard tests

Chat measures role-play and companion conversation: multi-turn sessions of long lore-heavy role-play, an everyday companion chat, a character with several voices, and a story written together under stacked constraints. A session checks whether the model remembers a fact planted earlier, accepts a correction, keeps a character's point of view and obeys the constraints it was given. It does not score plain question answering or one-off instruction following, so read Chat as how good a model is company in a story, not as how good an assistant it is.

How the scale grows

Every model takes the same standard test, and a perfect run scores 10 in each category. A model that scores at least 90% on a part unlocks a harder bonus pack for it, which adds points on top. That is the only way past 10, so Code, Chat and Total can pass 10, 10 and 20, and there is no ceiling. When the top models start clearing a pack, we add a harder one. That keeps one shared test for every model, from a small local model to a frontier API, and nobody tops out.

NSFW tests: separate on purpose

Some models are built to go further with adult fiction, and some people want exactly that, so the board measures it. Those scores live on their own tab, +NSFW Bench, and are never added to Total. Every character in those tests is an adult. A higher number only says the model goes further. Whether that helps depends on what you want it for.

How are answers judged?

Code never needs an opinion; the tests pass or they do not. Chat does, so CrucibleForge uses a judge model. On this board the judge is Gemma 4 31B (uncensored “heretic” build, Q8) on our own GPU rig. DeepSeek Flash is a backup judge candidate in CrucibleForge, not the judge used here. The judge is named for each suite, and this page is updated when it changes. Each case has a checklist of yes/no questions, for example "did the character stay in first person?", and a few plain text checks run beside them. The judge answers the checklist; it does not hand out a free-form grade.

Two rules keep the judge honest. A judge question must ask exactly one thing, because a question that chains several requirements turns a near-miss into a full miss. We check this in our own copy of CrucibleForge; that check is not in a public release yet. And before a judge is used it has to pass a calibration check, so a judge that quietly refuses or softens its task gets caught.

How do I read a score?

• Margin. The ± beside Total is half of a 90% interval from a bootstrap over the test cases. Models closer together than that are statistically tied. The ± covers which cases were drawn in one run, not run-to-run noise: when we ran the same weights at two quants or serving tiers, the totals differed by up to about two points. At the time of writing the best local model scores 16.1 ± 0.9 and the next one 15.2 ± 1.0: treat them as level, not as first and second.
• Class. A coarse band from Total: S at 16 or above, A at 13, B at 10, C below. The board had 12 in S, 16 in A, 9 in B and 4 in C.
• Lean. Chat-leaning or code-leaning when a model's Chat and Code scores sit 1.5 points or more apart relative to the board medians, otherwise balanced. It tells you which job a model is better at.
• Coverage. A model that has not finished all 33 cases shows its numbers but has no Total, class or rank.
• Provisional. If CrucibleForge's own compatibility check fails for a model, it is likely that our harness mis-measures it. That row keeps its score but is marked with a warning sign (click it for the explanation), has no rank and no class, and is re-run.
• Run cost. For hosted models, what one run of the standard Code and Chat tests cost at the provider's published price: the tokens sent in and the tokens written back, thinking included. It leaves out bonus sets, retries, the judge (it runs on our own hardware) and any cache or off-peak discount, so it is close to what you would pay at list price. A ~ means the provider did not bill per token (a flat plan), so we priced the token counts at list price; a * means some requests came back without usage data, so the figure may be low. A dash means the model ran on our own hardware and there is no bill.
• Blocked. A hosted provider can refuse to return a reply and say its content filter blocked it. The row then carries a small "blocked" mark and a note in its details. A blocked reply scores zero, the same as a failed answer, so a model that is blocked a lot looks worse than it might in your own use.
• Built on. What the model is built on. A hosted model shows its product line, such as Claude Opus. A local model shows the base weights it was trained from, such as Gemma 4 31B, so a fine-tune sits next to the model it came from. Type a base name in the filter, like gemma 4, to list everything built on it.
• Older version. When a vendor's newer release of the same line is on the board and has a rank, the older row carries an "older version" mark. "Latest versions only" is ticked by default and hides those rows; untick it to show them, and the box remembers your choice, and a link to a hidden row still opens it. Fine-tunes and merges are never hidden. They are not a vendor's release, so no newer version replaces them.
• MAX. The model reached the top of every set in one part of the tests. The tests cannot tell models apart above that point, so a harder set is due for that part. The row's details name the part.
• Quant. Local models are scored at Q8_0 where one exists. If Q8_0 does not fit our rig, the row uses Q6 or Q5 and says "Q8_0 does not fit our rig". A few older rows were scored below Q5; they stay on the board with their quant shown and a "below Q5" note. We re-test one at the standard quant only if it leads: first on its board (Main or +NSFW), first on a headline column, or first in its size class. A vision-language model is marked "text suite only": we test its text answers, not its images.

A new version is linked by one fact written when the model is added: which model it replaces. There is no release-date table to keep. Both models must be official releases. If a new hosted model arrives with a Built on tag that another official row already uses and nothing links the two, the next board refresh prints a warning with the exact command to fix it, so a newer version cannot slip by without a link. A newer row that is provisional or unranked never hides an older ranked one.

Why do scores only compare within a suite?

The suite is the version of the test set. The current one is 4.0 (4.0.1 plus a content hash), and each row records the suite it was scored with. When the tests change in a way that moves scores, models are re-run and the old board is kept, and the suite history lists every version. The previous suite, 3.0, scored 32 models out of 100, a different scale, so a 3.0 number and a 4.0 number are not comparable. What changed in 4.0 is in Suite 4.0: what changed.

What does the board not tell you?

• It is not a ranking of the best model. It measures two jobs. A model that is weak at both can still be the right pick for its licence, price or speed.
• Each case is run once per model. That is why the margin matters more than the order, and why a gap of a point or two is a tie.
• The judge is a model too. Chat scores inherit its taste. A checklist narrows the judge's taste but does not remove it.
• Hosted models can change under the same name. A provider can re-route or retire a model id, which is why every row shows the date it was tested.
• It does not measure long documents, vision, speech or languages other than English.

Request a model, and vote

The Vote page lists every model people have asked for. Search for a model first; if it is already there, give it a +1. You get one vote per model. If it is not there, request it with its Hugging Face id, a Hugging Face link, or a pasted model card, and it joins the list.

Requests sit in five tabs, each with a count: Waiting, Needs funding, Tested, Couldn't test and Declined. Sort by test order, votes, newest, oldest or name, filter by size (under 10B, 10 to 40B, 40 to 120B, 120B and up, or unknown), and use "Show 25 more" to page through long lists. The page address remembers your choices, so you can share a view.

Test order. Most votes go first. A model with one vote is tested oldest first while it is under two days old; older single-vote requests wait until someone else votes for them. A gift tied to a model also moves it up within its group, and never changes its score. We decide what actually runs, so a vote moves a model up the list but does not guarantee a run.

What we understood. If you ask for a quantized copy (a GGUF, AWQ or MLX repo, for example), we test the original model it was made from, and the vote counts for that model. The page tells you: "You asked for X. We'll test it as Y." If you really did mean that exact build, press "No, I meant this exact build"; it becomes its own request and we review it by hand. If we cannot match what you typed to a model on Hugging Face, we look it up and add it to the list ourselves.

Declined requests stay visible with a short reason. We never decline a model for being too big; see the next section.

What our rig can test, and what needs funding

MakoBench runs on one home machine with four graphics cards: two RTX 5090s and two RTX 3090s, 112 GB of video memory in all. We test every model at Q8_0, the 8-bit build that is very close to the original. When a model's Q8_0 does not fit, we step down to Q6 and then Q5, and the row says so. We never go below Q5, because below that you would be measuring the shrinking, not the model. In practice that means models up to about 135 billion parameters, tested with 32,000 tokens of context.

Bigger models do not fit, so we cannot test them here. We mark them Can't test locally and they sit in the Needs funding tab. If a hosted service runs the model at 8-bit or better, we can test it there, which costs real money, so the request gets a funding goal. The goal is about four times the largest cost we expect for the run, because a remote test also needs setup, sometimes a second attempt, checking, and scoring on our own machine afterwards, and payment fees come out of it too. Judging still happens on our rig, the same way as for every other row. If no hosted service offers the model, the row says "Too big to test for now" and has no goal. Votes still count, and we will test it if that changes.

If a model reaches its goal, we test it and it goes on the board like any other. If we cannot test it after all, or it turns out to fit our rig, the money goes to running the bench instead. Money never changes a score.

How we stay independent

Nobody pays us to test a model or to change a result. The suite, the judge and the settings are the same for every row, and CrucibleForge is open, so anyone can re-run a model and check. We sell no paid services, and if that changes this page will say so.

Run CrucibleForge yourself

You need Python 3.12 or newer with uv, or Docker, plus an endpoint to test. The short version, which the install guide walks through with real output:

git clone https://github.com/LaserLloyd/CrucibleForge.git
cd CrucibleForge
uv sync --python 3.13
uv run crucibleforge config --init      # writes models.yaml; add your provider and model
uv run crucibleforge status             # is everything reachable?
uv run crucibleforge all --models my-model --judge my-provider:my-judge --yes

Then open results/report.html. Coding answers run in a bwrap sandbox on Linux; on Windows and macOS use the Docker image, where the container is the sandbox. The README covers the web GUI, detached runs and refreshing a whole board.

Why MakoBench?

The shortfin mako is commonly described as the fastest shark in the ocean. Wikipedia's article on the shortfin mako gives burst speeds of up to about 74 km/h, though estimates vary and such figures are hard to measure. That is the idea behind the name: we built the suite to run quickly, and CrucibleForge, the open-source engine, does the work underneath.

Related: where the tool started and Suite 4.0: what changed.