The notes below describe each column as it is used today. Suite 3.0 measured these columns on its own, older tests and scale.

What the tests measure

Total

Total is the Code score plus the Chat score. Each of those is 10 for a perfect run of the standard tests, so 20 is the nominal top, not a maximum. Bonus sets (see Bonus sets) add points on top of that, with no ceiling, so a model can score above 20. The uncensored tests have their own tab and are never added in. The ± figure beside a Total is half of a 90% interval from a bootstrap over the test cases, so two models closer together than their margins are statistically tied.

Chat

Chat scores role-play and companion conversation, not general assistant work. It runs 8 multi-turn sessions: a lore-heavy role-play, companion chats, a character with several voices and a story written together. A session checks that the model remembers facts planted earlier, accepts corrections, keeps a character's voice and point of view, and follows stacked writing rules. A separate judge model, Gemma 4 31B (uncensored “heretic” build, Q8) on our own GPU rig, grades the replies against a checklist for each case, and plain text rules cover the rest. Answering questions or following one-off instructions is not scored here. Chat is 10 for a perfect run of the standard questions, and bonus sets add points on top with no ceiling.

Code

Code covers writing working Python functions, predicting what a program prints, finding and fixing bugs, repository tasks and tool use. Answers are run against hidden tests, so the score reflects code that works rather than code that looks right. Code is 10 for a perfect run of the standard questions, and bonus sets add points on top with no ceiling.

Suite

The suite is the version of the test set a score was produced with. When the tests change, models are re-run and the old board is kept under Previous suite versions. Compare scores within one suite only.

Date tested

Date tested is the day the model's results on this board were produced. A model re-run later shows the newer date.

Open Main Bench Every score here is a column on the board.