Run CrucibleForge on Your Own Models: Install Guide

Go to: CrucibleForge on GitHub

CrucibleForge is the open-source tool that produces every score on the MakoBench board. It is free (MIT-0, no conditions) and you can run it on your own models, on your own machine. This page walks through the install and a first run. Every command below was run on a fresh clone of the public repository, and the output is pasted from that run. No GPU or no time? Ask us to run your model on our rig.

What do you need before you start?

• Python 3.12 or newer with uv, or Docker. uv will fetch Python for you.
• A model to test behind an OpenAI-compatible /chat/completions endpoint.
• A judge, which is any capable instruct model, local or hosted. Code is graded by running it, but Chat needs a judge.
• On Linux, bubblewrap (bwrap). Model-written code runs inside it. On Windows and macOS there is no bubblewrap; you pass --allow-unsandboxed or use the Docker image, where the container is the sandbox.

How do you install it?

This is the native route on Linux. Docker is the simpler route if you would rather not install Python tools. The README has copy-paste blocks for Windows, Linux and macOS, with Docker Desktop or Compose as method 1 on each. We did not run the Docker route for this page.

git clone https://github.com/LaserLloyd/CrucibleForge.git
cd CrucibleForge
uv sync --python 3.13
uv run crucibleforge --version
crucibleforge 4.0.1

That is version 4.0.1, the current public release at the time of writing. Now create your config file:

uv run crucibleforge config --init
wrote models.yaml - edit providers/models, then `crucibleforge status`

The starter file lists providers for LM Studio, Ollama, llama.cpp or any local server, DeepSeek, OpenRouter, OpenAI and Open WebUI, plus four example models. Nothing runs until you enable one.

How do you connect your model?

models.yaml needs a provider, a model and a judge.

1. Provider. Where the model is served. Any server that speaks /chat/completions works with type: openai. Local examples from the README: http://localhost:11434/v1 for Ollama, http://localhost:1234/v1 for LM Studio, http://localhost:8000/v1 for vLLM. A hosted API takes api_key_env: NAME_OF_ENV_VAR. The key itself never goes in the file.
2. Model. An entry with a name, the provider, the model_id the server knows and a context_length.
3. Judge. Passed per run as --judge provider:model_id.

To see what a server offers, ask it. This is a hosted provider with the key in the environment:

uv run crucibleforge models discover deepseek
  * deepseek-flash
  * deepseek-v4-pro

2 models on deepseek (* = in registry)

The star means the model is already in the registry. The starter file ships deepseek-flash enabled, so no edit was needed for our run. For your own model, add an entry like this:

models:
  - name: my-model
    provider: ollama
    model_id: qwen3:8b
    context_length: 32768

How do you check the setup?

export DEEPSEEK_API_KEY=...       # only for a hosted provider or judge
uv run crucibleforge status

Without the key set, the status output for the hosted provider reads like this (trimmed to the relevant lines):

suite revision: 4.0.1+e3d1c6fb
providers:
  ollama         openai      http://localhost:11434/v1       DOWN conc=8
  deepseek       openai      https://api.deepseek.com/v1     UP   conc=8 key: $DEEPSEEK_API_KEY (NOT SET)
    reachable, but it wants an API key: set $DEEPSEEK_API_KEY (HTTP 401)
registry (models):
  [enabled ] deepseek-flash   deepseek   deepseek-flash    $0.14/0.28 per 1M
cases: 393 total (235 hard)
smoke subset: 37 cases
benchmark (bench): 33 cases

DOWN means nothing answered at that address, which is expected for local servers we were not running. UP with a 401 means the host is reachable and wants a key. The case count matters: the whole set has 393 cases, and the board uses 33 of them.

What does a run cost before you start it?

Add --plan to any run and nothing is called. It prints what would run and an estimate:

uv run crucibleforge all --models deepseek-flash --judge deepseek:deepseek-v4-flash --plan
  T1 - every model: 29 rows (29 cases) + 4 perf probes
  one round - hosted API (nothing loaded): 29 rows, 8 requests at once
  estimate: generation 18.5 min; judge 16.0 min ... -> ~34 min
(plan only - no model, provider or judge was called; times are estimates)

That is the full board run on a hosted model: about 34 minutes by the tool's own estimate. It is an estimate, not a measurement. For a local model the README says roughly 25 to 30 minutes on a fast GPU box.

What does a smoke run look like?

The --smoke flag runs a small tagged subset end to end: generation, judging and the report. It is the quickest way to prove your setup works. We used DeepSeek Flash as both the model and the judge, which is the pairing the README's quickstart uses, and ran it outside DeepSeek's peak hours.

uv run crucibleforge all --smoke --models deepseek-flash --judge deepseek:deepseek-v4-flash --yes
... profile bench: 4 cases, thinking cap 24576, judge deepseek:deepseek-v4-flash
... link check passed: deepseek reachable, data channel OK
... run: models=['deepseek-flash'] cases=4 smoke=True fresh=False
... [deepseek-flash] RD1-marisol r1 done (15 rows)
... judging done: 1 rows, 0 unparsable judge verdicts, 0 empty generations, 0 errored
run summary:
  deepseek-flash: 15 rows, load 0.0s (deepseek), $0.0229
timing: generation 1.2 min, judge 14 s, report 8 s, total 1.6 min
wrote results/report.md, results/failures.md, results/report.html

The run took 1.6 minutes and cost $0.0229. The 15 rows are one role-play case plus the speed probes. Two warnings are worth knowing about:

• The judge's API rejected a structured-output request, so CrucibleForge fell back to free-text JSON for that session. Scores still came out.
• The judge failed the calibration check for adult-content text. CrucibleForge then judged only the other rows and left the adult-content rows pending for a judge that can handle them. The README says most hosted judges behave this way. Those scores sit beside the board and are never added to Total.

How do you read the result?

Open results/report.html in a browser, or read results/report.md. Our smoke run printed one row:

| Model          | Overall | Class | Coding | Chat        | tok/s | Suite | Notes                                          |
| deepseek-flash | -       | -     | -      | 2.5 +/- 0.1 | 159.7 | 4.0.1 | partial (4/33 cases); judged by deepseek-v4-flash |

The Chat figure of 2.5 is not a score for the model. It is the one role-play case that ran, out of the part of Chat that case belongs to. The row says partial (4/33 cases), and a partial model has no Overall, no Class and no rank. That is the rule on the main board too. Run the full set to get a real Total (Code plus Chat, 20 nominal). For what each column means, read How MakoBench works and What MakoBench tests.

Speed (tok/s) only compares between models run on the same hardware. A hosted model's speed depends on its provider and how busy it was.

How do you run the full benchmark?

uv run crucibleforge all --models my-model --judge my-provider:my-judge --yes

Add --fresh to archive the model's earlier transcript first. Results land in results/ next to models.yaml: the scorecard, the failures, the HTML report and a transcript of every prompt and answer. The web GUI is one command (uv run crucibleforge gui) and listens on loopback only by default.

A few things to know first

• Coding cases run model-written code. Use bubblewrap on Linux or the Docker image elsewhere. --allow-unsandboxed runs that code as you, with your files readable.
• Hosted runs cost money. Run --plan and the smoke test first.
• The suite includes adult-fiction cases that sit beside the board and are never added to Total. The README has a content note about them.
• The judge changes Chat scores. On the MakoBench board the judge is Gemma 4 31B (uncensored “heretic” build, Q8) on our own GPU rig. Your Chat numbers only line up with the board if you use the same judge; Code is graded by running it, so it compares either way.
• Scores compare within one suite version. This page is about suite 4.0; see what changed in Suite 4.0.

Did everything check out?

Nearly. We also ran uv run crucibleforge cases verify, which re-derives the gold answers offline. It checked 393 cases and reported one with a problem: a code-bank case whose reference solution timed out on a signal test. That run shared our machine with other work, so we cannot say if it is a real fault or a slow box. We have not reported it upstream yet. If you hit the same case, you are not alone.

Want us to run it for you?

Related: How MakoBench works, Suite 4.0: what changed, and the board.