My AI Agent Built a Benchmarking Tool for My AI Agents
- Category
- AI & Local LLM
- Posted
- August 11, 2026
- Updated
- November 9, 2026
- By
- Jacob Lloyd — written with AI assistance, post-project
- Read time
- 20 min read
In plain terms: Our AI coding agent wrote bench-llm, a single Python file that tests how fast a local model is and whether it can do simple jobs. It still works and is free to download. It got too easy within two weeks, so we replaced it with CrucibleForge, a bigger free tool that asks 276 questions and runs the code the model writes to check it.
On 2026-08-11 we asked our coding agent for a tool that benchmarks the local models in our agent stack. It came back with bench-llm: 1,660 lines of Python that measure speed in LM Studio, run the code a model writes, and check whether the model can handle a small agent task. Within two weeks every model we cared about was scoring 5/5 on it, so we rebuilt it as CrucibleForge: 276 cases as of October 2026, 166 of them hard, open source on GitHub. This page covers both. Use bench-llm to get a first number out of a model in five minutes. Use CrucibleForge when you need to rank models that all pass the easy tests.
tl;dr
- bench-llm: one Python file, free download. It tests Speed (time to first token, tokens/sec), Ability (5 tests: code, logic, JSON, instructions, summarization) and Agent Fitness (3 checks on one syslog task). Needs Python 3.8+,
pip install openai requestsand LM Studio. - Why it was retired: 5 ability tests and 3 agent checks stop telling models apart once good models pass them all.
- CrucibleForge 3.6: 276 cases, 166 hard. Code runs in a sandbox and is graded on the result. It works with any OpenAI-compatible endpoint, has a web GUI, and now runs in Docker and on Windows. MIT licence, github.com/LaserLloyd/CrucibleForge.
- What the board shows: Kimi K3.1 Pro has the highest score on the 2026-11-08 board at 19.3 of 20, though the board’s own paired test calls all 26 models a tie; MiniMax M3 has a judged score for the first time (14.6, and 15.5 once its re-runs count in) at about $0.57 a run, and the whole board cost $68.21.
Changelog: 2026-08-26, bench-llm retired and results moved to the successor suites. 2026-09-16, CrucibleForge section rewritten, leaderboard rebuilt from the saved run files, cover redrawn. 2026-10-19, CrucibleForge section updated for 3.6.0 (what's new): current install, the one-command benchmark, Docker. 2026-11-02, the model board rebuilt from the 2026-11-02 run (suite 4.0.0). 2026-11-09, the hard-tier board rebuilt from the 2026-11-08 run (suite 4.1.0): 26 judged models, run cost back on the board, MiniMax M3 scored for the first time.
What you end up with
One command against one model prints this (sample run, trimmed):
$ bench-llm gemma-4-31b-it
bench-llm — Benchmarking: gemma-4-31b-it
GGUF Size: 16.5 GB Quant: Q4_K_M
⚡ SPEED BENCHMARKS
short_50 TPS: mean=84.2 TTFT: mean=0.231s
medium_200 TPS: mean=85.7 TTFT: mean=0.312s
long_800 TPS: mean=82.1 TTFT: mean=0.541s
🔍 Validating speed plausibility... Confidence: HIGH
🧠 ABILITY TESTS
Code Generation (median_of_list)... PASS
Logic Puzzle (Pet Ownership)....... PASS
JSON Compliance.................... PASS
Instruction Following.............. PASS
Summarization (Key Facts).......... FAIL
Ability Score: 4/5
🤖 AGENT FITNESS TESTS
Task Acknowledgment / Format Adherence / Conciseness: PASS
Agent Fitness Score: 3/3
📄 Results saved: ~/benchmarks/gemma-4-31b-it-2026-08-11.json
The JSON file holds every raw timing, every model response and a summary. You can hand it back to your agent and ask "which of our models should do bulk JSON work?"
What bench-llm tests
Speed
It sends three prompt lengths (about 50, 200 and 800 characters), does warm-up runs, then measures three things. TTFT (time to first token) is how long you wait before text appears; over 500 ms feels sluggish in chat. Throughput (tokens/sec) is how fast it writes once started. Prefill T/s is how fast it reads your prompt.
It also runs a plausibility check against known hardware limits. If a 30 GB model claims 200 T/s on a 7900 XTX, bench-llm flags the number, because speculative decoding or a warm cache has probably skewed it.
Ability
- Code Generation: write
median_of_list(numbers)with the edge cases handled. bench-llm executes the function against six test cases instead of reading the text. - Logic Puzzle: a four-person pet-ownership puzzle. The checker pulls all four assignments out of the reply.
- JSON Compliance: return an object with exact keys. This is the minimum a model needs for tool calling.
- Instruction Following: list 1–10, mark the evens, sum them. The maths is trivial; the format is the test.
- Summarization: compress a dense paragraph and keep six named facts. Most models drop at least one.
Agent Fitness
The model gets one realistic agent task: search syslog for ERROR lines, group them by service, return a JSON report. Three checks score that one reply:
- Task Acknowledgment: does it outline an approach and name what it cannot do? A model that invents syslog lines fails.
- Format Adherence: is there a
servicesarray, atotal_errorsinteger and ascan_periodstring? - Conciseness: the output-to-input ratio. A 20,000-character answer to a 1,300-character prompt is flagged VERBOSE, because an agent that rambles fills its context window on every turn.
How it was built
Our spec to the coding agent was one sentence: "we need a tool that benchmarks local models in LM Studio, tests speed and smarts, outputs JSON and a table." It built the tool in four passes over one afternoon: speed tests over LM Studio's OpenAI-compatible streaming API, then the ability tests, then the agent task, then polish (finding the GGUF file on disk, the plausibility check, --quick, --list, --check). The agent wrote every line. Our part was running it on real models and pushing back. The plausibility check exists because we told the agent one throughput number was impossible for a 123B model.
Design choices worth copying into your own tools:
- Stream, don't poll. Per-token timing is the only way to get an honest TTFT.
- Run the code. Confident-looking code that doesn't run is a fail, not a pass.
- Extract the answer before checking it. The checkers strip "Sure! Here's your answer:" wrappers. That way the test scores the content, not the preamble.
- Start every run from an empty GPU. The previous model's cache otherwise skews TTFT. How bench-llm does this is the first gotcha below.
Setup
# 1. Install both dependencies (requests reads LM Studio's model metadata;
# without it every model shows as "not found")
pip install openai requests
# 2. Start LM Studio with at least one model; it listens on localhost:1234
# 3. Run it
./bench-llm --list # what's available
./bench-llm gemma-4-31b-it --quick # smoke test
./bench-llm gemma-4-31b-it # full benchmark
There are no config files and no database. LM Studio accepts any string as an API key. Results land in ~/benchmarks/. Copy the script into a folder on your PATH if you want to run it from anywhere.
Gotchas
- It kills llama-server. To start each run clean it calls
pkill -f llama-serverand waits for LM Studio to report zero loaded models. Any other llama-server on the machine dies too. Don't run it on a box that serves anything else, or remove that step. - The coding test runs model output with
exec(). It uses a fresh namespace, which is not a sandbox. Run it as a user with nothing to lose, or skip the ability tests with--speed-only. - Reasoning models look slow. TTFT counts the first token of any kind, thinking included. The JSON splits it into
ttft_reasoning_sandttft_content_s. - A 4/5 with only Summarization failing is a strong result. Nearly every model misses at least one fact.
- A full run takes 2–5 minutes per model. Use
--quickto compare many, and run the full suite on the finalists. - LM Studio must already be running. bench-llm connects to localhost:1234 and does not start anything.
Results: base models on our rig and mini PC
These numbers come from the intermediate suite that followed bench-llm (same speed method, more tests, report of 2026-08-18). "Rig" is a headless GPU box with 2× RTX 5090 and 2× RTX 3090. "APU" is the mini PC's integrated GPU (when it still ran a local model server). Compare speeds only within one device. Code, Tools, Instruct and Reason are pass rates.
| Model | Device | Gen tok/s | TTFT | Code | Tools | Instruct | Reason |
|---|---|---|---|---|---|---|---|
| Gemma 4 26B-A4B (QAT, Q4) | rig | 229.0 | 110 ms | 88% | 100% | 50% | 95% |
| Nemotron 3 Nano 4B (Q8) | APU | 18.5 | 175 ms | 92% | 70% | 94% | 95% |
| Gemma 4 12B | APU | 10.1 | 633 ms | — | — | — | — |
The Gemma 4 26B mixture-of-experts model (4B active) is the everyday workhorse on the rig: 229 T/s, 110 ms to first token and perfect tool calling. Its weak column is instruction following, which is what an unsupervised agent loop depends on most. The 4B Nemotron on the mini PC follows instructions better (94% vs 50%). It is about twelve times slower (18.5 vs 229 T/s), and on a different device. Pick by the column your workload stresses, not by the fastest number.
What replaced it: CrucibleForge
bench-llm died of success. Five ability tests make a good smoke alarm and a poor ranking: once every model scores 5/5 and 3/3, the tool has nothing left to say. CrucibleForge is the rewrite. It is now about 12,000 lines of Python, and its rule is that a hard question should still be trivial to mark.
- 166 hard cases. Competition maths with integer answers, logic grids with unique solutions, and coding problems whose tests enforce the right complexity (an O(n²) inversion count times out). Also tool-use traps with decoy tools and prompt injection, 12k–24k-token long-context retrieval, and multi-constraint instructions.
- Code is graded by running it in a
bwrapsandbox with no network, a read-only system and a 15-second limit. A pass needs a sentinel string on stdout, so a program cannot fake success by exiting zero. Withoutbwrap(always the case on macOS and Windows) it refuses to run model code unless you pass--allow-unsandboxed. - Prose answers get a gold answer. The judge model sees the correct answer and only decides whether the reply means the same thing. Any competent instruct model can do that, so the judge stops being the weak point.
- Any OpenAI-compatible endpoint. LM Studio, Ollama, llama.cpp, vLLM, DeepSeek, OpenRouter and others. Hosted models run cases concurrently and report token cost. API keys come from environment variables only.
- It recovers answers that thinking models lose. A reasoning model that spends its whole budget thinking returns an empty answer. The run notices, asks again with thinking off, and counts how often that happened.
recoverre-runs just those rows from an old run. - It shares GPUs politely. On our StudioForge rig a run takes a lease on the cards it needs and gives back whatever it displaced when it ends. bench-llm's
pkillsolved the same problem by killing everyone's models. - A web GUI on
127.0.0.1:8777lets you pick models and categories, watch the log and open any failed row: prompt, reply, reasoning and the judge's note. It needs a token before it will listen on anything but loopback.
Try it
The simplest route is Docker, because the container is also the sandbox for the code models write. On Linux, with Docker installed:
git clone https://github.com/LaserLloyd/CrucibleForge.git && cd CrucibleForge
cp deploy/compose/.env.example deploy/compose/.env # LLM_BASE_URL defaults to port 1234 on the Docker host
docker compose -f deploy/compose/compose.yml up -d
docker compose -f deploy/compose/compose.yml logs app | grep "GUI:" # open the printed http://127.0.0.1:8777/?token=... URL
Or natively, which on Linux gets you bwrap isolation for the coding cases:
sudo apt-get install -y bubblewrap
git clone https://github.com/LaserLloyd/CrucibleForge.git && cd CrucibleForge
uv sync --python 3.13
uv run crucibleforge config --init # writes models.yaml; set base_url to your server's /v1
uv run crucibleforge status # is the provider up? how many cases?
uv run crucibleforge all --models my-model --fresh --yes
uv run crucibleforge gui # http://127.0.0.1:8777
Pass --python 3.13: on a bare machine a plain uv sync can pick a newer Python than the project supports. On macOS and Windows there is no bwrap, so native runs need --allow-unsandboxed before the subcommand (model-written code then runs as you), or use Docker Desktop. If your model server wants an API key, put the variable's name in api_key_env on that provider in models.yaml; the key itself stays in the environment.
all runs the one benchmark, profiles/bench.yaml: 49 cases chosen as the smallest set that still separates models, then judging, then the board. Each model gets two scores from 0 to 100. Coding comes from deterministic graders (programs, tool use, instruction following, reasoning and maths). Chat is judged writing, role-play and steerability, including adult-content categories this page does not cover. The rest of the 276 cases stay in the suite for targeted runs with --cases or --categories. The other verbs are run, judge, recover, report, pairwise, models, bench-all, import-openclaw and cases.

models.yaml and no results: which providers answer, which judge scores the benchmark, and the case set (276 cases, 166 hard). Nothing local was running, hence the DOWN badges.The command we trust most is the one that checks the questions themselves. It runs every reference solution in the sandbox, re-derives every maths answer by brute force and rebuilds every long-context haystack. This is its output on a clean copy of 3.6.0, on 2026-10-19:
$ uv run crucibleforge cases verify
276 cases checked, 0 with problems (36 reference solutions executed, 37 answers re-derived)
The test suite runs the same check, so a broken question can't quietly fail every model and pass for a hard tier doing its job.
What's new in 3.6
CrucibleForge 3.6.0 (18 October 2026) changes how the project installs and runs, not what it measures. No case, grader or metric changed, and results stamped 3.5.1 count as the same case set, so nothing on an existing board goes stale.
- Docker. A read-only, non-root image of about 190 MB with the GUI on
127.0.0.1:8777and yourmodels.yamland results on a volume that survives upgrades. Inside the container the coding cases run withoutbwrap, because the container is the sandbox; it can still reach the network, so treat it as a place where untrusted code runs. - Windows. The test suite passes on Windows and CI has a Windows leg. One limit: the lock that stops two benchmarks running at once is not enforced on Windows, so start one at a time.
- A sandbox self-test. A
bwrapthat is installed but cannot actually isolate (inside a container, or with user namespaces blocked) is caught before the run starts. The run is then refused (or, with--allow-unsandboxed, runs unsandboxed and says so up front) instead of failing every coding case. - API keys and clearer errors.
statussays when a server answers 401 or 403 and names the variable to set, which is what a StudioForge 2.0 gateway with keys needs. - Fixed on 2026-10-19: the GUI's "Judge pending rows" button scored rows with a fallback judge whose verdicts do not count on the board; it now uses the benchmark's judge, as the command line always did, and the dashboard names that judge.
Upgrading. Natively, git pull && uv sync --python 3.13. If your models.yaml points StudioForge at port 8080, change it to 1234: 8080 is StudioForge's panel, and the API is on 1234. An existing models.yaml is never rewritten for you. In Docker, git pull then docker compose -f deploy/compose/compose.yml up -d --build; the volume is kept, and down -v is the one command that deletes it.
What a report looks like
report writes a Markdown and a JSON file. Each model gets a summary row and a per-category row with raw counts. Here is DeepSeek V4 Pro's per-category row:
| Model | Coding | Math | Tool use | Instruction | Reasoning | Long-context |
| deepseek-pro | 89% (54/61) | 95% (21/22) | 94% (31/33) | 100% (31/31) | 100% (37/37) | 97% (29/30) |
Every failure is listed with its reason. All seven of its coding failures look like the first line below. The second line is one of its two tool-use misses:
- deepseek-pro [coding] CZ01-prime-census: TRUNCATED at 32768 tokens (32768 reasoning tokens); no code in response
- deepseek-pro [tooluse] TZ06-three-parallel-one-turn: 1 calls, expected 3
That is the useful part. The model didn't write bad code; it thought until the 32,768-token budget ran out and never wrote any. A bigger budget or a lower reasoning effort would change that score. A pass rate on its own would not tell you that.
Hosted models also get a Run cost in the report, the price of one run of the standard tests (see the MakoBench documentation).
Hard-tier results
This is the board as it stood on 2026-11-09, rebuilt from the run of 2026-11-08 on suite 4.1.0 and judged by gemma-4-31B. The scoring is the one described above: every category is out of 10 on the standard test, Overall is out of 20 (Coding plus Chat), and a category only goes past 10 by unlocking a ladder pack — three harder questions, worth 1 point, given only after 90% on everything below. Coding is the deterministic half (functions, prediction and bug hunting, tool use); Chat is the judged half (deep role-play, conversation, character, creative writing). The ± after a score is half the 90% interval of a bootstrap over the cases, so two models whose intervals overlap are a tie, not a ranking.
Class is the coarse band a score falls in (S from 16, A from 13, B from 10). Rows finished is the share of rows that answered inside the standard output budget — a row that ran out counts as failed — and a few rows still have a retry pending, so some scores can still move. Run cost is what one standard run cost at the provider’s price, generation only; the judge runs on our own cards. The report also prints a median generation speed, but that is only comparable between models run on the same host and these are different providers’ machines, so we leave it out.
Fresh judged — and one tie for the whole board
- One run, one judge, one day. All 26 rows below were run and judged on 2026-11-08, suite 4.1.0, with gemma-4-31B as the judge.
- The board’s own paired test cannot separate any two of them. Every adjacent pair’s 90% interval contains zero, and the ties chain all the way down, so the report gives all 26 rows the same rank label. Read 19.3 down to 11.3 as a point estimate, not a ranking.
- MiniMax M3 finally has a judged score. The previous board gave it a Coding number and no Overall. Here it stands at 14.6 ± 1.7, and the report’s Room column puts it at 15.5 once its two budget-exhausted rows are re-run on a bigger budget — one more retry is still pending. At about $0.57 a run it costs a seventh of the top row.
Hard tier — full 26-model board
| Model | Overall | Coding | Chat | Class | Rows finished | Run cost |
|---|---|---|---|---|---|---|
| Kimi K3.1 Pro | 19.3 ± 0.5 | 10.0 ± 0.1 | 9.3 ± 0.5 | S | 100% | $4.07 |
| Grok 4.7 | 19.2 ± 0.3 | 10.0 ± 0.1 | 9.2 ± 0.3 | S | 100% | $3.91 |
| Claude Opus 5.5 | 19.2 ± 0.5 | 10.0 ± 0.1 | 9.2 ± 0.5 | S | 100% | $4.50 |
| DeepSeek V4 Flash | 18.8 ± 0.8 | 11.2 ± 0.4 | 7.6 ± 0.6 | S | 92% (4 ran out) | $0.14 |
| Claude Sonnet 5.5 | 18.6 ± 0.4 | 10.0 ± 0.1 | 8.6 ± 0.4 | S | 100% | $2.36 |
| GPT-6.1 Terra | 17.9 ± 1.0 | 9.4 ± 0.6 | 8.5 ± 0.7 | S | 100% | $0.63 |
| DeepSeek V4 Pro | 17.8 ± 0.7 | 10.1 ± 0.5 | 7.7 ± 0.4 | S | 89% (5 ran out) | $0.57 |
| GPT-6 Luna | 17.7 ± 0.6 | 9.7 ± 0.2 | 8.0 ± 0.5 | S | 100% | $0.08 |
| Claude Opus 5 | 16.7 ± 1.4 | 8.3 ± 1.2 | 8.4 ± 0.4 | S | 100% | $5.07 |
| Claude Opus 4.8 | 16.3 ± 0.6 | 9.2 ± 0.5 | 7.1 ± 0.4 | S | 100% | $4.37 |
| Claude Fable 5.1 | 16.1 ± 0.9 | 7.7 ± 0.5 | 8.4 ± 0.7 | S | 100% | $10.23 |
| Claude Opus 4.7 | 16.1 ± 0.8 | 9.3 ± 0.6 | 6.8 ± 0.5 | S | 100% | $4.75 |
| Claude Fable 5 | 16.0 ± 1.9 | 7.5 ± 1.9 | 8.5 ± 0.6 | S | 100% | $9.70 |
| Claude Haiku 5.5 | 15.3 ± 1.1 | 9.2 ± 0.7 | 6.1 ± 0.9 | A | 100% | $0.11 |
| Claude Opus 4.6 | 15.3 ± 1.3 | 8.0 ± 1.3 | 7.3 ± 0.3 | A | 100% | $3.38 |
| Claude Sonnet 5 | 15.3 ± 0.9 | 8.8 ± 0.8 | 6.4 ± 0.4 | A | 100% | $2.81 |
| Claude Sonnet 4.6 | 14.6 ± 1.6 | 7.6 ± 1.4 | 7.0 ± 0.7 | A | 97% (1 ran out) | $2.98 |
| MiniMax M2.5 Highspeed | 14.6 ± 0.8 | 8.3 ± 0.7 | 6.3 ± 0.4 | A | 100% | ~$0.63 |
| MiniMax M3 | 14.6 ± 1.7 | 8.5 ± 0.8 | 6.0 ± 1.4 | A | 94% (2 ran out) | ~$0.57 |
| Claude Opus 4.5 | 13.8 ± 1.4 | 7.1 ± 1.2 | 6.7 ± 0.7 | A | 100% | $3.27* |
| MiniMax M3.1 Flash | 13.6 ± 1.7 | 8.0 ± 0.9 | 5.6 ± 1.4 | A | 94% (2 ran out) | ~$0.28 |
| MiniMax M2.7 | 13.4 ± 1.7 | 6.7 ± 1.6 | 6.7 ± 0.5 | A | 100% | ~$0.30 |
| MiniMax M2.7 Highspeed | 13.2 ± 1.2 | 7.6 ± 0.9 | 5.6 ± 0.8 | A | 100% | ~$0.62 |
| Claude Sonnet 4.5 | 13.2 ± 1.1 | 6.4 ± 1.1 | 6.8 ± 0.4 | A | 100% | $1.94* |
| MiniMax M2.5 | 12.2 ± 1.8 | 6.4 ± 1.6 | 5.8 ± 0.8 | B | 100% | ~$0.31 |
| Claude Haiku 4.5 | 11.3 ± 1.2 | 5.7 ± 1.0 | 5.6 ± 0.6 | B | 100% | $0.63* |
Cost summary: one standard run of all 26 rows cost $68.21 in provider tokens, generation only — the judge runs on our own cards, and the adult-content category this page does not cover is priced separately. Twelve of the 26 rows came in under a dollar, $4.87 between them; the other 14 account for $63.34, and the two most expensive, Claude Fable 5.1 and Claude Fable 5, are $19.93 of that. Six of the figures are estimated from token counts rather than billed, and three carry an asterisk because some requests came back without usage data, so the real total is a little higher.
What is not on the board. Thirty-two rows are fine-tunes we run on our own cards, and the report marks every one of them partial — their runs did not cover the full standard case set — so none carries an Overall and none is ranked. Four of them failed outright as well: three were refused a GPU lease and one was stopped mid-run. Eight hosted models are pending too: three have twenty rows the judge has not scored, and five were rejected by their provider before they could run. Every row above still carries a few verdicts judged under the previous grading, pending a re-judge, so the numbers can move a little.
Hosted rows carry their product names. Twenty-four rows mix results from the 4.0.1 suite with this 4.1.0 run; DeepSeek V4 Flash and DeepSeek V4 Pro also mix in 3.9.0. A "~" on a cost means the figure was estimated from token counts at list price, and a "*" that some requests came back without usage data, so it may be low. Four pending rows carry registry labels with a retired model name this site does not print.
What we take from it:
- Nobody is ahead. Every adjacent pair on the board has a paired 90% interval that contains zero, and those ties chain from the top row to the bottom one, so the report gives all 26 rows the same rank label. Kimi K3.1 Pro’s 19.3 and Claude Haiku 4.5’s 11.3 are eight points apart on paper and indistinguishable on this run — treat the order as a point estimate.
- Cost is the axis that separates them. One standard run of the 26 rows cost $68.21. DeepSeek V4 Flash buys 18.8 points for $0.14; Claude Fable 5.1 buys 16.1 for $10.23, about 85 times less score per dollar. Twelve rows came in under a dollar, and four of those score 16 or better.
- Coding carries the top of the board, Chat does not. Only DeepSeek V4 Flash (11.2) and DeepSeek V4 Pro (10.1) are above 10 on Coding, and both unlocked a ladder pack; no model reaches 10 on Chat, where the best score is Kimi K3.1 Pro’s 9.3. If you are wiring a model into an agent, the Coding column is the one that matters.
- Output budget is the quiet filter. Five rows ran out of budget — DeepSeek V4 Pro (5 rows), DeepSeek V4 Flash (4), MiniMax M3 and MiniMax M3.1 Flash (2 each) and Claude Sonnet 4.6 (1) — and on the board a row that ran out counts as failed. Where a re-run exists, the Room column carries the score with it counted in: MiniMax M3’s 15.5 and MiniMax M3.1 Flash’s 14.5 are both 0.9 above their board score, while DeepSeek V4 Flash’s and Claude Sonnet 4.6’s are still pending.
- Provider filters cost real points. Four rows had replies blocked by the provider’s content filter — Claude Fable 5 (6), Claude Opus 5 (2), Claude Fable 5.1 (2) and Claude Opus 5.5 (1) — and a blocked reply scores zero. Five rows also had one verdict the judge could not parse.
Download
The download is bench-llm, the retired original, not CrucibleForge (that lives on GitHub). The zip holds two files: bench-llm, the 1,660-line script, and README.md with condensed setup notes. There's nothing to build or install beyond the two packages: unzip and run. The agent wrote both files, and they are MIT-licensed. The dependencies are openai and requests. Read the gotchas before you run it on a machine that serves anything else.
Related: StudioForge (the GPU server these runs lease cards from), What my AI agents actually cost (why local speed matters), DeepSeek Harness (a coding agent these models back), and the local agent stack they run in.
Downloads
Free for personal use. If it saves you an afternoon, the coffee button's nearby.