Suite 4.0 (47 models tested)

The tests (suite 4.0)

The notes below describe each column as it is used today. Suite 4.0 measured these columns on its own, older tests and scale.

What the tests measure

Total

Total is the Code score plus the Chat score. Each of those is 10 for a perfect run of the standard tests, so 20 is the nominal top. Bonus packs (see Bonus sets) add points on top of that, so a model can score above 20; in this suite Code can reach 13 and Chat 15 with every pack. A row's bars run to that most-possible figure, with a tick where the standard set ends. The uncensored tests have their own tab and are never added in. The ± figure beside a Total is half of a 90% interval from a bootstrap over the test cases, so two models closer together than their margins are statistically tied.

Chat

Chat scores role-play and companion conversation, not general assistant work. It runs 8 multi-turn sessions: a lore-heavy role-play, companion chats, a character with several voices and a story written together. A session checks that the model remembers facts planted earlier, accepts corrections, keeps a character's voice and point of view, and follows stacked writing rules. A separate judge model, Gemma 4 31B (uncensored “heretic” build, Q8) on our own GPU rig, grades the replies against a checklist for each case, and plain text rules cover the rest. Answering questions or following one-off instructions is not scored here. Chat is 10 for a perfect run of the standard questions; bonus packs add points on top, up to 15 in this suite.

Code

Code covers writing working Python functions, predicting what a program prints, finding and fixing bugs, repository tasks and tool use. Answers are run against hidden tests, so the score reflects code that works rather than code that looks right. Code is 10 for a perfect run of the standard questions; bonus packs add points on top, up to 13 in this suite.

Programs

Programs gives the model 6 tasks that each ask for one Python function from a precise, trap-filled description (dates, rounding, text layout, scheduling). The answer is run in a sandbox against hidden unit tests, and the score is the share of test groups that pass, so a nearly right answer earns partial credit.

Code bank

Code bank shows the model short snippets of everyday Python and JavaScript, the kind of helper code a household app has, and asks exactly what each one prints. There are 2 sets, one per language, and every predicted output is checked by exact match. It rewards knowing how the language really behaves (shared defaults, sort order, rounding, truthiness), not writing new code. Score is the share of predicted outputs that are exactly right.

Agent

Agent gives the model one bug report about a small multi-file project and the tools a coding assistant uses: list, read and search files, edit them and run the tests. It is a single task (1 in the suite), so one run can move this column a lot. The fixed project is graded by running its tests.

Tools

Tools tests whether a model calls tools correctly. Each case is a scripted conversation with a tool round trip: the model has to choose the right tool, give valid arguments, use the result, and avoid the traps built into the setup. A malformed call counts as a failure.

tok/s

tok/s is the median speed at which the model produced text during the run. Local models were run on our own hardware, so those speeds compare with each other. Hosted models depend on the provider and on how busy it was, so treat their speed as a rough guide.

Run cost

Run cost adds up what each request of one full run cost at the provider's published price: the tokens sent in and the tokens written back, including thinking. It covers the standard Code and Chat tests. It leaves out bonus sets, retries, the judge (it runs on our own hardware) and any cache or off-peak discount, so it is close to what you would pay at list price. A ~ means the provider did not bill per token (a flat plan), so we priced the token counts at list price. A * means some requests came back without usage data, so the figure may be low. A provider's content filter can block a reply. The row then says how many were blocked, and each blocked reply scores zero.

Class

Class is a quick way to group models by Total: S, A, B and C, with S the strongest. The cut-off for each band is listed in the notes under the board. A model without a complete Chat score has no Total and no class.

Lean

Lean compares a model's Chat and Code scores with the board's median for each. If it is clearly better at one than the other, it is marked chat-leaning or code-leaning, and otherwise balanced. The size of the gap needed is in the notes under the board.

Suite

The suite is the version of the test set a score was produced with. When the tests change, models are re-run and the old board is kept under Previous suite versions. Compare scores within one suite only.

Date tested

Date tested is the day the model's results on this board were produced. A model re-run later shows the newer date.

Built on

A fine-tune keeps its base's name here and is never hidden by Latest versions only. That box is on by default and hides a vendor's older release when a newer release of the same line is on the board and has a rank. Typing a base name in the filter, such as gemma 4, lists everything built on it.

Quant

Quant is the compression format of a local model's weights. Q8_0 is close to the full-precision model and is the standard quant on this board; a smaller quant runs faster and fits smaller GPUs but can score lower. We use Q6 or Q5 only when Q8_0 does not fit our rig, and never go below Q5. Older rows scored below Q5 stay on the board with a "below Q5" note, and are re-tested at the standard quant only if they lead. If the same weights were tested at two quants, one row is ranked and the other sits under it as an other variant. API means a hosted model, which the provider serves in whatever form it chooses.

Open Main Bench Every score here is a column on the board.