# 4 Harness Defects That Skewed My LLM Benchmark Scores

URL: https://www.laserlloyd.com/makobench/news/llm-benchmarking-lessons-learned/
Published: 2026-11-02
Updated: 2026-11-02
Description: Four harness defects skewed an LLM benchmark score: dropped reasoning, empty truncation rows, stale model ids, compound judge questions. Free kit inside.
Reuse: free for personal use — full policy at https://www.laserlloyd.com/llms.txt

In the overnight run that finished at midnight on 2026-11-02, a new hosted model scored below its older sibling: 13.3 of 20 for MiniMax-M3.1-Flash against 14.2 for MiniMax-M2.5. The part of the number we could not trust was our harness's.

tl;dr

The score was mostly real; the interesting part was not. An audit put the two harness defects at roughly +0.4 to +1.7 points, true score around 13.7 to 14.9; the gap to the leaders was genuine model quality.
Three harness habits make a model look dumb. Dropping reasoning is one: the same probe recalled 0 of 10 without it, 8 of 10 with it. "Ran out of tokens" rows were empty (13 of 13, 10 of 10 across all stored transcripts), never partial. A model list is a list, not a promise: three retired ids were still listed, and every call 404'd.
One judged question asked four things at once. A judge element is a single yes/no, so a reply that delivered the scene but missed one beat scored as a full miss, and the heat factor went with it.
A provably right fix is still not a measurement. The error bar runs 0.8 to 1.8 points and the reasoning fix moved Overall by less than that; one coding case passed on 2026-11-01 and failed on 2026-11-02 under the same harness.

What you end up with

A free zip, llm-conform 1.0, answering one question per model you benchmark: does my harness speak this model's protocol? Six checks through your own request path — about 21 API requests per replay mode — with the evidence saved next to each.

$ python3 llm_conform.py --config my-config.toml

MODEL my-model  (https://api.example.com/v1)
  c1   PASS  served; endpoint reports model 'my-model'
  c2   PASS  reasoning arrives via 'reasoning_content'; answer text is clean
  c3   FAIL  your replay mode loses this model's reasoning: 'content-only' recalled 0/3
              but 'reasoning_content' recalled 3/3 - use reasoning_content
  c4   WARN  tool round trip ok (lookup_record -> result -> answer); an unanswered tool call
              is tolerated by the endpoint, but it must still be answered or removed
  c5   PASS  the sampler was accepted
  c6   PASS  a medium task fits in max_tokens=4096 (finish_reason=stop)

  VERDICT FAIL  (1 fail, 1 warn, 4 pass, 0 info)

Three runs of one harness, differing only in how it sends assistant turns back. One mode loses the reasoning.

Exit code is 0 when every check passes, 1 on a warning, 2 on a failure, so it drops into CI. No packages, no vendor: the examples use https://api.example.com/v1, my-model and $MY_API_KEY.

What four defects did our LLM benchmark hide?

The board after the fixes. The ± after a score is half the 90% bootstrap interval over the cases. The board calls two rows a tie when the paired 90% interval of their difference contains 0, not when the individual bars overlap.

The MiniMax rows sit in a band from 11.7 to 14.4 with error bars of 0.8 to 1.8 points, each overlapping its neighbours.

ModelBeforeAfter the fixesRoom
MiniMax-M3.1-Flash13.313.5 ± 1.614.3
MiniMax-M313.414.4 ± 1.615.3
MiniMax-M2.514.211.7 ± 1.8=
MiniMax-M2.5-highspeed12.814.0 ± 0.8=
MiniMax-M2.7-highspeed13.813.5 ± 1.1=
MiniMax-M2.712.612.6 ± 1.8=
deepseek-flash19.219.2 ± 0.8—
deepseek-pro17.517.5 ± 0.8—

"Before" is the board of 2026-11-02 in the morning; "after" is the board rebuilt at 20:18 and refreshed at 20:55 once two rows were re-run. Room is the score with the out-of-budget rows re-run larger. The largest move is MiniMax-M2.5 dropping 2.5 points: its Func sub-score fell 5.0 to 3.1, single-turn code the fix cannot touch.

1. The harness threw the reasoning away

Our multi-turn runner fed back only the visible text of the assistant turn. MiniMax's docs require the opposite: the complete response, reasoning_content included, in the conversation history. The probe asked for a random six-digit number in the reasoning only and reply "ok"; the next turn asked which number.

ModelReasoning droppedReasoning replayed
MiniMax-M31 of 107 of 10
MiniMax-M3.1-Flash0 of 108 of 10
MiniMax-M2.52 of 106 of 10

With the reasoning dropped the model produced reasoning on 1 turn out of 12 on the case we measured, RD4-kettle-saddle; replayed, on 6 to 12 out of 12 — the harness suppressed the thinking, it did not lose it. The fix is a per-provider switch, on by default for a provider pointed at MiniMax's API and off for every other provider, plus one request field so the thinking arrives in the same channel. The judged conversation stays visible-text-only: reasoning is context, not answer.

A harness that replays only the visible text answers with a different number every time: there was never anything to recall.

A second leak sat in the tool loop: an assistant turn appended without its tool call and result came back as a second call, failing a check that expects one. Across the stored transcripts 45 tool calls were dropped, and 3 of 3 of the "N tool calls, expected 1" failures re-issued one.

2. "Ran out of tokens" meant empty, not partial

Our board ran at 16,384 output tokens. The vendor's recommended value for the two big models is 131,072 — eight times more. Every truncated row was empty, not partially answered: across all stored transcripts, 13 of 13 for MiniMax-M3 and 10 of 10 for MiniMax-M3.1-Flash, 100% of them, and none for the two smaller models, whose reasoning is far shorter. The board run itself is a much thinner slice: MiniMax-M3 had 2 truncated-empty rows, both from the case WR1-ninth-string, and MiniMax-M3.1-Flash had 0. That one case is the only one that still feeds the score: at most +0.574 points for M3 (and nothing for M3.1-Flash), and possibly zero, because a non-empty answer could still fail its checks.

Re-running the failing cases at the recommended budget turned zeroes into answers: one case that returned finish=length with 16,383 of its 16,384 tokens spent on reasoning and zero visible characters came back at 131,072 with a complete 8,897-character Python function, another with 4,466 characters. Both of those cases have since left the board, so they cost it zero points. So the board keeps a Room column — without it, "equal budget" means "equal budget plus a silent zero for the model that thinks longest".

3. The model list was stale

Three ids in the vendor's registry were marked retired, yet the vendor's own model endpoint still listed them and every call returned HTTP 404 with "no longer available" and the code 2069. A model list is a list, not a promise: probe the model you are about to benchmark — one cheap call — and treat "the endpoint lists it" as a hint, never as evidence. Check c1 does this, and also catches aliasing.

Still listed by the vendor, every call 404s: disable them, and probe the model you are about to use.

The same rule applies one step earlier. The vendor's own FAQ says the M2.5 and M2.7 families retire on 2026-11-18, and that on the M Plan / Token Plan, requests to them have been auto-routed to M3.1 Flash since 2026-10-19. We probed that on 2026-11-02: the API still reports the requested id, and with reasoning_split: false the M3.1-Flash id returns HTTP 400 while the M2.5 and M2.7 ids return 200 with no reasoning at all. From outside you cannot tell what serves them. So if the FAQ applies to this account, four of the six MiniMax rows are M3.1-Flash under other names, and their spread (11.7 to 14.0) is one model's run-to-run noise, not a ranking.

4. One judged question asked four things at once

A judge element is a single yes/no, and a no caps the dimension it guards. One element in an adult role-play test session asked the judge to confirm four things at once — whether the act was on the page, how it was staged, whether the character climaxed, and whether the other character's name was said. A reply that delivered everything but the name scored no and took the heat factor with it; the name was also checked separately, so the same detail was charged twice.

Measured on the stored rows: of the 17 rows ever judged against that compound element, 10 failed it. A row's heat factor is the share of its explicitness checks passed plus one vote for the judge's overall explicitness score, so one no takes down a quarter of its heat. And single sessions are noisy: on the first adult case both MiniMax rows score 0.73, so it cannot separate the siblings; on the second, MiniMax-M3 leads 0.89 to 0.44; on another adult case the order flips (0.0 for M3 against 0.91 for M3.1-Flash). One session per case is a sample, not a ranking.

The fix splits the element into one question about the page and one about staging, grades the name once, and re-judges the stored transcripts instead of regenerating them. It also added a lint: any gating element whose question joins more than two requirements is now an error, with the pre-fix text kept as a regression fixture. The whole case set passes the lint: 463 judge elements over 393 cases. One row was judged under both questions and failed both — it really had missed that beat — with heat factor 0.75 and case score 0.62. The fix removed an unfair charge; it did not rescue a model that did not do the thing.

One question joins four requirements, the other two are fine. Exit code 2, so it can gate a run.

5. A correct fix still does not move the board

The reasoning fix is provably right: the same probe went from 0 of 3 recalled to 3 of 3, every time, in three separate runs with different puzzles. And the board moved less than its own error bar: the mini chain probe went from 0 to 3, the Deep RP part of the score rose by 0.2 to 0.4 points on the two M3 models, and Overall moved by less than its ±. MiniMax ignores the seed parameter, so every MiniMax row is one sample: two codebank cases, five identical runs each, gave element-rate spreads of 0.250 and 0.500.

One coding case passed on 2026-11-01 and failed on 2026-11-02 under the same harness and prompt — two honest runs, opposite verdicts. Checking it turned up a second glitch: a stream that ended after the reasoning, with no finish reason, was accepted as a finished empty answer and scored zero. It is now treated as a dropped stream and retried. So the order of operations is the part worth stealing: verify the protocol, then repeat the session before you believe a one-point move.

The root cause, and the check that catches it

None of the four defects was a bug in a model or in the scoring code, and they do not all have the same cause — the stale model list is vendor-side, the compound judge element is case design, the token budget a config choice. Four root causes:

The request protocol per provider and model was assumed, never verified against the model's behaviour or the vendor's documentation.
Nothing proved, per model, that the conversation the harness sends is one the model can use.
Rows did not record the protocol that produced them, so a harness fix could not tell which rows were stale.
Case lint did not catch compound gating questions.

The fix is a conformance step before every benchmark, not a special case in the runner: six checks per model — about 21 API requests per replay mode — through the harness's own request path, never a hand-rolled request, because then the check proves nothing about the harness. The record it writes carries a protocol fingerprint, so a harness change can be detected.

The kit in the download is that step, extracted so it works on any OpenAI-compatible endpoint: the same six checks, the same evidence files, the rubric linter, and an offline test suite of 44 tests against a fake local endpoint. It is a download, not a subscription. What it will not do is tell you which model is better — that is your job, and the honest version needs repeats and a published error bar.

Where this leaves us

The M2.5 and M2.7 families retire on 2026-11-18 — four of the six MiniMax rows — and plan traffic for them moves to another model, so those board rows need a decision before that date. The crucibleforge conform command exists and caught the reasoning defect live; making the benchmark refuse a model that fails it is the next step, not done yet. We still lack the repeats needed to rank anything inside a one-point band — and that band is the honest answer.

Related: the benchmark tool itself (276 cases, 166 hard), what our agents actually cost, MiniMax M3 as a coding agent, and StudioForge.