My AI Agent Built a Benchmarking Tool for My AI Agents

My AI coding agent Hermes wrote a benchmarking tool. It tests the speed, intelligence, and agent-fitness of my local models — the same models other AI agents run on — and it spits out JSON and a terminal table. The tool is 1,660 lines of Python called bench-llm. The hook: an AI agent built a benchmarking tool for AI agents.

What you end up with

One command against one model gives you this in your terminal:

$ bench-llm gemma-4-31b-it

============================================================
  bench-llm — Benchmarking: gemma-4-31b-it
  GGUF Status:   READY
  GGUF Path:     /home/puppy/.lmstudio/models/google/gemma-4-31b-it-Q4_K_M.gguf
  GGUF Size:     16.5 GB
  Arch:          gemma4
  Quant:         Q4_K_M
============================================================

  🔥 Warming up model... OK (0.23s TTFT)

  ⚡ SPEED BENCHMARKS
  ----------------------------------------
    short_50 (131 chars, 3 runs)...
      TPS:  mean=84.2 median=83.8
      TTFT: mean=0.231s median=0.228s
    medium_200 (323 chars, 3 runs)...
      TPS:  mean=85.7 median=85.2
      TTFT: mean=0.312s median=0.308s
    long_800 (769 chars, 3 runs)...
      TPS:  mean=82.1 median=81.5
      TTFT: mean=0.541s median=0.537s

  🔍 Validating speed plausibility...
    Confidence: HIGH

  🧠 ABILITY TESTS
  ----------------------------------------
    Running: Code Generation (median_of_list)... PASS
    Running: Logic Puzzle (Pet Ownership)... PASS
    Running: JSON Compliance... PASS
    Running: Instruction Following... PASS
    Running: Summarization (Key Facts)... FAIL

    Ability Score: 4/5

  🤖 AGENT FITNESS TESTS
  ----------------------------------------
    Running: Task Acknowledgment... PASS
    Running: Format Adherence... PASS
    Running: Conciseness (Output/Input Ratio)... PASS

    Agent Fitness Score: 3/3

  📄 Results saved: ~/benchmarks/gemma-4-31b-it-2026-08-11.json

================================================================================
  BENCHMARK RESULTS: gemma-4-31b-it
================================================================================

  📊 SUMMARY TABLE
  ──────────────────────────────────────────────────────────────────────
  Model ID                         gemma-4-31b-it
  GGUF Size                        16.5 GB
  Architecture                     gemma4
  Quantization                     Q4_K_M
  T/s (short, mean)                84.2
  TTFT (short, mean)               0.231s
  Ability Score                    4/5 (80.0%)
  Agent Fitness Score              3/3 (100.0%)
  Confidence                       🟢 HIGH
  ──────────────────────────────────────────────────────────────────────

It also writes a full JSON payload to ~/benchmarks/ with every raw measurement, every test response, and a structured summary — so you can feed the results back into your agent and ask things like "which of my 36 models should handle bulk JSON work?"

What it tests

Three dimensions, each chosen because they matter for an always-on agent stack.

⚡ Speed

Three prompt lengths — short (~50 chars, "explain what a variable is"), medium (~200 chars, "explain recursion"), and long (~800 chars, "compare three programming paradigms") — with three warmup runs followed by the real measurements.

What gets measured:

• TTFT (Time To First Token) — how long before the model starts typing. Critical for chat UX; anything over 500ms feels sluggish.
• Throughput (tokens/sec) — how fast it generates once it starts. Matters for long code blocks and reports.
• Prefill T/s — how fast it processes the input prompt before generating. Hidden but real cost.

It also runs a plausibility check against known hardware ceilings. If your 30 GB model claims 200 T/s on a 7900 XTX, bench-llm flags it — that number's probably contaminated by speculative decoding or a pre-warmed cache.

🧠 Ability

Five tests that are trivial for a competent model and a dead giveaway for a struggling one:

• Code Generation — Write median_of_list(numbers) with edge cases (empty list, even length, odd length, don't mutate input). The function is executed inside bench-llm against six test cases — it's not pattern-matching against the response text, it's actually running the code.
• Logic Puzzle — A four-pet ownership puzzle (Alice/Fish, Bob/Bird, Carol/Cat, Dan/Dog). Regex extracts the answer assignments from the response and checks all four.
• JSON Compliance — "Return a JSON object with these exact keys." Tests whether the model can produce valid, schema-conformant JSON on demand — the bare minimum for tool-calling.
• Instruction Following — Write numbers 1–10, mark evens with *, sum them. Tests precise format adherence: the answer is trivial, the format is the whole point.
• Summarization — A dense paragraph about helium-based quantum computing. Checks whether six key technical facts survived the compression. Most models drop at least one.

🤖 Agent Fitness

This is where bench-llm does something I haven't seen in other benchmarking tools. It sends the model a real agent task — search syslog for ERROR lines, group by service, produce a JSON report — and evaluates the response as an agent supervisor would:

• Task Acknowledgment — Does the model understand what it was asked to do? Can it outline an approach and identify blockers? (A model that dives straight into hallucinating syslog entries fails here.)
• Format Adherence — Does the output match the requested JSON schema? Checks for the services array, total_errors integer, and scan_period string.
• Conciseness — What's the output-to-input ratio? Models that respond to a 1,300-character prompt with 20,000 characters of rambling get flagged as VERBOSE. An agent that can't be concise burns your context window on every turn.

The three agent-fitness tests share the same prompt — they're evaluating different aspects of the same response. This keeps the benchmark fast while still measuring the qualities that matter when a model is running unsupervised.

How it was built

Hermes — the coding agent on my box — wrote the entire thing. I gave it a rough spec: "I need a tool that benchmarks local models in LM Studio, tests speed and smarts, outputs JSON and a table." It went away, came back with a working script, and we iterated.

The process was:

1. First pass: Speed tests only — connect to LM Studio's OpenAI-compatible API, stream tokens, measure TTFT and T/s.
2. Second pass: Ability tests — coding (with actual execution), logic puzzles, JSON schema compliance, instruction following, summarization.
3. Third pass: Agent fitness — the syslog task, response evaluation as if the model were an agent in my stack.
4. Polish passes: GGUF file resolution (find the actual model file on disk, report its size and quantization), speed plausibility validation, the summary table, --quick mode, --list for model discovery, the pre-flight --check command.

All four iterations happened over one afternoon. The final script is 1,660 lines. Every line was written by Hermes — I reviewed, tested on real models, flagged issues ("this throughput number can't be right for a 123B model"), and it fixed them. The plausibility validator was born from exactly that kind of feedback loop.

Design decisions that stuck

• Streaming, not polling. bench-llm uses OpenAI-compatible streaming to get per-token timing. A polling approach would blur the TTFT measurement and lose the token-level granularity.
• Actual code execution for the coding test. It doesn't ask "does this look like a median function?" — it exec()s the code in a sandboxed namespace and runs six test cases. No amount of confident-looking output survives that.
• Regex-based answer extraction for logic/JSON tests. Benchmarks that require the model to output a precise format fail harshly on models that add "Sure! Here's your answer:" before the actual output. bench-llm's checkers extract the payload from whatever wrapper the model puts around it — testing the model's content, not its verbosity.
• VRAM cleanup between benchmarks. It runs pkill llama-server and waits for LM Studio to show zero loaded models before each run. Without this, a previous model's KV cache can contaminate the next benchmark's TTFT measurement.
• GGUF file resolution. It maps LM Studio's model IDs back to the actual GGUF files on disk using fuzzy matching on publisher name and architecture. This gives you model size, quantization, and whether the file even exists — answering "is this model actually ready to benchmark?" before wasting a run.

Setup

Three steps:

# 1. Install the one dependency
pip install openai

# 2. Make sure LM Studio is running with at least one model loaded
# (it should be listening on localhost:1234 — that's the default)

# 3. Run it
./bench-llm --list                         # see what's available
./bench-llm gemma-4-31b-it --quick         # smoke test
./bench-llm gemma-4-31b-it                 # full benchmark

That's it. No config files, no API keys to configure (LM Studio accepts any string), no database to set up. Results land in ~/benchmarks/ automatically.

Optional but useful: copy bench-llm into your PATH so you can run it from anywhere.

cp bench-llm ~/bin/
chmod +x ~/bin/bench-llm

Real results from my box

I ran bench-llm against a handful of the 36 models I have loaded in LM Studio. Here's what came back, unedited:

* The DeepSeek R1 8B model's 0.42s TTFT reflects its reasoning overhead — it thinks before it types, hence the asterisk. That's not a bug, it's the architecture.

The takeaway: for pure speed on routine tasks, the 27B Qwen is the winner — 91.5 T/s with 0.19s TTFT and perfect scores across the board. The 120B Nemotron is the heavy-lifter — half the speed, but it aces everything. The DeepSeek R1 8B is the dark horse: fast tokens but its reasoning preamble drags TTFT up and its smaller size shows in the ability tests. The Dark Scarlett model is good-but-not-great — fine for chat, but it missed the format adherence test in agent fitness.

These numbers are from my hardware (Threadripper, dual RTX 3090s, 48 GB VRAM). Yours will differ. That's the point of having the tool.

What makes this different

There are plenty of LLM benchmarks. Most of them are weird academic puzzles (MMLU) or leaderboard-optimized trivia contests. bench-llm does three things that matter for someone actually running local models:

1. It tests agent qualities, not chatbot qualities. Format adherence, task acknowledgment, and conciseness are what make or break a model in an unsupervised agent loop. A model that's great at conversation but can't follow a JSON schema is useless in a tool-calling workflow.
2. It validates its own measurements. The plausibility checker catches numbers that can't be right — speculative decoding contamination, pre-warmed caches, measurement bugs. If a benchmark tool can't tell you when its own output is garbage, you can't trust any of it.
3. It's one script. No Docker, no database, no GPU driver dependencies, no 50 GB dataset download. One Python file, one pip install, and it works against whatever you have loaded in LM Studio right now.

Gotchas

• LM Studio must be running first. bench-llm doesn't start LM Studio for you — it connects to localhost:1234. If nothing's listening, it fails with a clear error.
• The VRAM cleanup is aggressive. It runs pkill llama-server between benchmarks to clear the KV cache. If you have other llama-server processes you want to keep alive, don't run bench-llm — or modify the cleanup step.
• Summarization is the hardest test. Almost every model I've tested drops at least one key fact. If a model scores 4/5 with summarization as the only failure, that's a strong result — don't read it as a deficiency.
• Reasoning models get slower TTFT. DeepSeek R1 and other thinking models spend time in a reasoning phase before producing output. bench-llm measures TTFT from the first token of any kind — including reasoning tokens. That makes reasoning models look slower than they feel in practice, because the thinking tokens stream during what would otherwise be dead air. The JSON output separates TTFT into ttft_reasoning_s and ttft_content_s so you can see the split.
• The coding test exec()s model output. It's sandboxed in a fresh namespace and the test functions are pure, so the blast radius is zero — but if the thought of running LLM-generated code makes you uncomfortable, skip the ability tests with --speed-only.
• 36 models take a while. A full benchmark on one model takes 2–5 minutes depending on speed. Running all 36 would take hours. Use --quick for comparisons and save the full suite for your top contenders.

Download

bench-llm is a single Python script plus a README. No build step, no install wizard — unzip and run.

The zip includes:

• bench-llm — the 1,660-line benchmarking script
• README.md — condensed setup and usage instructions

Both files are written by Hermes, my coding agent. The script is MIT-licensed — use it, modify it, ship it in your own toolchain.

Downloads

• Download bench-llm (zip) 18 KB

Free for personal use. If it saves you an afternoon, the coffee button's nearby.