# What MakoBench Tests, and What It Doesn't

URL: https://www.laserlloyd.com/makobench/docs/what-makobench-tests-and-what-it-does-not/
Published: 2026-11-06
Updated: 2026-11-06
Description: MakoBench scope: which models we test, which we leave out (image, video, 3D, embeddings, translation tools) and the rules a row must meet to be ranked.
Reuse: free for personal use — full policy at https://www.laserlloyd.com/llms.txt

MakoBench has one job: to tell you how good a general text model is at two things we use models for. We say plainly what that covers, because a score only means something for a model the tests were built to measure.

tl;dr

In: anything whose main output is text and that can take an instruction or a conversation and answer it.
Out: image, video, audio and 3D generators, embedding and classifier models, single-task tools, and base checkpoints.
Two jobs: writing working code, and role-play or companion conversation. Adult writing is a separate test.
Ranked rows are fair rows: the whole suite ran, our compatibility check passed, and the row is the main one for that model.

What we test

We test a model whose input is text, whose main output is text, and that answers a chat-style instruction or conversation. That covers:

chat and instruct models of any size, running locally or behind a hosted API;
reasoning models, run the way their makers say to run them, with the same output budget as every other row;
coding models that also chat. They take the code tests and the chat tests, and a weak chat score is a result, not a reason to drop them;
uncensored, abliterated and role-play fine-tunes of the above;
vision-language chat models, on the text tests only. Their row says "text suite only", because we did not measure what they do with an image.

The two jobs are Code and Chat, 10 points each on the standard tests, and Total is the two added together. Code is writing working Python functions, predicting what snippets print, a tool-assisted bug hunt and correct tool calls. Chat is role-play and companion conversation: remembering facts, keeping a character and following writing rules. It is not a test of plain question answering or one-off instruction following, so a model can be a strong assistant and a middling Chat score. A separate adult-fiction test has its own tab, the last one, and is never added to Total.

What we don't test

Models that don't output text. Image, video, speech and music generators, and 3D, CAD and scene generators. The tests cannot score a picture or a mesh, so any number would be noise.
Models that output a vector or a label. Embedding models, rerankers, classifiers, and safety, guard or reward models.
Single-task text tools. Translation models, captioners, speech-to-text and OCR tools, and models tuned only to summarise or extract. They are good at one narrow thing that Code and Chat do not measure.
Base checkpoints. A model with no chat template cannot follow the prompts the suite sends.
Models CrucibleForge cannot run. The engine checks that it can talk to a model properly before it counts a score. If that check fails, the model is declined.

When a request falls in one of these groups we decline it with a short note on the Vote page, and the model never reaches the board. We never decline a model for being too big; that is covered below. Models we only pick automatically also have to be permissively licensed and not gated; we can still add one ourselves.

What has to be true before a row is ranked

The whole suite ran. A model with partial coverage shows its numbers but has no Total and no rank.
The compatibility check passed. If it failed, CrucibleForge's setup is probably measuring the model wrongly. The row stays visible, marked with a warning sign and "provisional", has no rank, and we re-run it.
One row per model. The standard quant for local models is Q8_0. If Q8_0 does not fit our rig we use Q6, then Q5, and the row says so. We never go below Q5. A second quant, or a faster serving tier of the same hosted model, sits under the main row as an other variant and is not ranked.
Older rows below Q5 stay visible. Their quant is shown, with a "below Q5" note in the row details. We re-test one at the standard quant only when it leads: first on its board (Main Bench or the adult-fiction bench), first on a headline column, or first in its size class.
Hosted rows carry the date they were tested. A provider can change or retire a model under the same name, so a hosted row is a snapshot.

What our rig can test, and what needs funding

MakoBench runs on one home machine with four graphics cards: two RTX 5090s and two RTX 3090s, 112 GB of video memory in all. We test every model at Q8_0, and step down to Q6 and then Q5 only when Q8_0 does not fit. In practice that means models up to about 135 billion parameters, tested with 32,000 tokens of context.

Bigger models are marked Can't test locally, never declined. If a hosted service runs the model at 8-bit or better, people can fund the test. The funding goal is about four times the largest cost we expect for the run, because a remote test also needs setup, a possible second attempt, checking, and scoring on our own machine, and payment fees come out of it. If no hosted service offers the model, the row says "Too big to test for now" and has no goal; votes still count, and we will test it if that changes. If a funded test cannot run, or the model turns out to fit our rig, the money goes to running the bench. Money never changes a score. A gift tied to a model moves it up within its group and never changes its score.

Why this much care

A benchmark that ranks everything ends up ranking things the tests cannot measure, and a score that cannot mean anything is worse than no score. Keeping the board to models the tests fit, and ranking only fair measurements, is what lets two numbers on it be compared. For how the scores are built and read, see How MakoBench works, and if you want a model tested, use the Vote page.