StudioForge: A GPU-Only LLM Server That Replaced LM Studio on My Rig, Run Remotely by My Agents

StudioForge is the local model server I wrote to replace LM Studio on my GPU rig: a llama.cpp supervisor with an OpenAI-compatible API on port 1234 — LM Studio's port, on purpose — a browser control panel, a recovery sidecar that answers when the main process will not, and a management plane published over MCP so an agent on another machine can run it without a shell.

It exists because of an incident log that stopped being funny. A 200 that proved nothing: LM Studio answers unrouted paths with a success status and an error body, its own log reading Returning 200 anyway. /v1/models listing everything downloaded rather than loaded, so "what is actually resident?" had no answer. Reasoning models overflowing an 8,192-token window, because the context I set in the client's config was not the context the load used — in LM Studio the window is fixed at load time, and nothing told me. And the expensive one: a model that did not fit got cut down instead of refused — the documented behaviour is that it "will automatically reduce the GPU offload size … and the rest in the system RAM", which trades GPU speed for system-RAM speed and reports success either way.

If you read the dsh write-up you have met this server already: the provider block labelled "StudioForge (GPU rig)" is this. It is what the eight-agent stack reaches across the network for heavy work, what DisPatch talks to, and what OpenClaw Email asks for a chat model and an embedding model. My OpenClaw write-up named the problem — "minutes of model load time, VRAM juggling between services" — and what follows is the answer, every rule carrying the measurement that forced it.

What it does, in one picture

What it is not:

• Not a model. It runs the GGUFs you have, and fetches more into the same folder and layout LM Studio uses.
• Not an inference engine. llama.cpp does the maths; this decides which process runs with which flags on which cards.
• Not a chat app. There is a Chat tab, and it exists to prove the real request path works.
• Not a cluster. One box, its own cards. llama.cpp's RPC backend exists and is not wired up.
• Not a CPU inference server. There is no --n-gpu-layers value other than 999 anywhere in the codebase.
• Not a second API surface. No Ollama /api/generate, no KoboldCpp API — the OpenAI surface, an LM Studio-flavoured /api/v0 mirror and the /api management REST, and that is the lot.

What runs where

Piece	Where	What it is for
Gateway	the GPU host, one process, 1234	/v1, /mcp (19 tools), /api. Holds the registry, the planner and the supervisor.
Control panel	same process, second uvicorn, 8080	Dashboard, Setup, Models, Download, Chat, Server, Logs.
Watchdog	a separate process, 1235	Its own MCP server, 10 recovery tools. It outlives what it supervises.
llama-server children	18100–18200, loopback only	One per loaded model; a crash takes down one model, never the gateway.
sfctl companion	the agent's machine	A pure HTTP client — Python 3.11+, no CUDA, no server dependency. Also the stdio MCP bridge.
GGUF library	models.dir, where it already is	Indexed in place; nothing copied. LM Studio keeps using the same folder.

What the table says:

• Only the GPU host installs anything. A client needs a base URL; an agent needs one small Python package, and only for the management tools.
• The children are invisible from outside. They bind 127.0.0.1 and nothing else, so the gateway is the sole public surface — which is what makes one API key a real boundary.

How a request flows

Everything a client can get wrong is checked before the first byte, because a bad request should get a real 4xx (a 404 for a model id that does not exist) with a JSON body rather than an error frame buried inside a 200 SSE stream — clients handle the former and routinely mishandle the latter. What GET /health says on my rig:

{"status": "ok", "version": "1.26-08-23", "uptime_s": 12690.6,
 "loaded_models": ["ggml-org/SmolVLM-256M-Instruct-GGUF/SmolVLM-256M-Instruct-Q8_0"],
 "busy": {"active_requests": 0, "busy_models": [], "loading": [], "testing": null},
 "draining": false, "instance": "primary",
 "boot": {"phase": "ready", "ready": true, "elapsed_s": 0.2, "error": null},
 "engine": {"ok": true, "tag": "b10425", "variant": "cuda", "smoke_tested": true},
 "gpu_count": 4, "models_indexed": 34, "can_serve": true}

• can_serve is the answer to the 200-that-proves-nothing problem. It is false while the first library scan runs, while status stays ok because the process is alive and that is what a liveness poller asks. GET /health?deep=true runs a real 8-token completion (an embeddings call for embedding models) against every loaded model — and with nothing loaded it answers no_models_loaded rather than passing, because a probe that cannot fail is worse than no probe.
• The local-model alias resolves. local-model, default, auto and current all map to models.default_model. LM Studio clients fall back to that literal string, so 404-ing it breaks them for nothing.
• A cold model does not look like a hang. The open stream carries : loading <model id> (5s) every five seconds, then : prefilling <model id> (Ns) until the first real token — SSE comment lines every parser ignores. A silent socket for that long trips a read timeout, and a retrying client piles more prefill onto a saturated batch.
• One load at a time, machine-wide. Two cold models once planned simultaneously onto the same cards; one died with CUDA error: out of memory, its retry evicted the other new model, and that client's request hit a dead child.
• A request-level ttl moves the idle timer and nothing else. ttl: 0 is the wire form of pinned everywhere here, so it is ignored rather than honoured — a client sending {"ttl": 60} used to unpin what its owner had pinned.

The truth about VRAM: the planner

This is the part nobody else does, so here it is in detail rather than in adjectives. core/planner.py is 3,188 lines answering one question before a model starts: given the VRAM genuinely free on each card right now, what is the best window, cache quality and slot count this model can have — and if the answer is none, what should I tell you?

The ladder, and what it refuses to trade

The rungs drawn there are illustrative, not the shipped defaults: the mechanism is exact, the numbers an example. What config.example.yaml ships is target_ctx: 1048576 as the aim and default_ctx: 8192 as the floor, the aim clamped to the model's trained window first — though on a first run tune_for_hardware writes default_ctx: 16384 when the smallest card has 24 GiB or more, and 8192 from 12 GiB up; my rig runs a floor of 128000. Going past a trained window needs RoPE scaling and degrades quality, so a tier above it is never offered. An explicit ctx_size is a one-rung ladder.

Two passes, the second only when the first failed everywhere. The reason is a load at 12:03 one afternoon: 79,832 MB free, another 19,423 MB reclaimable from one idle model. At that budget 262144 on a q4_0 cache fitted at 96,004 MB, and 65536 at full f16 fitted at 95,236 MB. What loaded was 8192/f16 at 89,860 MB — the model paid the full price of the eviction and got the smallest window on the ladder.

KV is not one number per model

Almost every VRAM calculator online computes KV as layers × heads × head-dim × 2 × bytes × context. Right for a Llama, badly wrong for two families people actually run: Gemma 3 and 4 interleave five sliding-window layers per full one at half the head dimension, and Qwen3.5, 3.6 and 3.8 declare full_attention_interval = 4, so a KV cache exists on every fourth layer and the rest are Gated-DeltaNet recurrent layers with a fixed per-sequence state.

The cost of getting that wrong: a Gemma-4 31B asked for 262,144 tokens was estimated at 480 GiB of KV and capped at 65,536, while the calibration log had recorded predicted_mb=95615 actual_mb=40037 for weeks. After the geometry fix, predicted and real both land at 38 GiB across two 5090s at n_ctx=262144 — a 4× unlock that took the whole Gemma-4 fleet with it. Charging KV to every layer of a Qwen3.5 was the same bug in a different hat: a straight 4× overcharge.

Two details worth stealing. The sliding-window cell count must mirror llama.cpp exactly; a flat 1.25× multiplier was wrong in the dangerous direction, 3.6× under at four slots. And attention_kind is derived from the layer geometry rather than general.architecture — where it cannot be derived it reports unknown, which means "distrust every KV number here", never "assume the cheap case".

Cache quality is chosen inside each rung rather than traded for a wider window: f16/f16 → q8_0/q8_0 → q8_0 K + q4_0 V, with symmetric q4_0 gone from every automatic path. Not taste — with a q4_0 K cache Qwen2.5-7B reproduces only 11.7% of the tokens its f16 self produces, while a matched q8_0/q8_0 pair sits at KL divergence 0.0018.

Four cards, two generations

The rig is two RTX 5090s and two RTX 3090s — nominally 32 GB and 24 GB cards, which the panel counts as 31.84 GiB and 24.0 GiB, 111.7 GiB in total — on driver 610.88, CUDA driver 13.3. Single card first, always — a split model on PCIe with no NVLink is meaningfully slower — and the planner overrides that only when the single-card placement is starved at one slot, the split at least doubles it, every added card is at least as capable, and you left the slot count on auto. That last condition is not politeness: a split runs at its slowest member's pace.

Placement (1.5B Q4_K_M, two 3090s, 8k ctx)	Generation	Prompt processing
One 3090	352.5 tok/s	2803.6 tok/s
Two, -sm layer	344.4 tok/s	2722.5 tok/s
Two, -sm tensor	294.3 tok/s	1182.0 tok/s
Two, -sm row	fails: error loading model: device CUDA2 does not support split buffers

What the table says:

• One card beat two on both axes. Layer split costs about 2% of generation; tensor split costs 17% of generation and 58% of prompt processing — so tensor mode is opt-in, and only a measurement may choose it.
• -sm row is dead on CUDA. The parser accepts it and the load then fails, so it is gated before the child spawns rather than after.

Two placement details only appear when you measure per card. The output layer is charged to the last device, because quantizers keep embedding and output tensors at Q6_K or Q8_0 even inside a Q4 file: a 27B planned as --device CUDA1,CUDA0 --tensor-split 0.5079,0.4921 landed 15.52 GiB on CUDA0 — the last device, the one the split gave less to — against CUDA1's 14.48. And llama.cpp opens a CUDA context on every visible device — ~0.22 GiB on a 3090, 0.43 GiB on a 5090 — which is why the placement column has a 512 MiB floor.

How many conversations a placement is worth

llama.cpp's --ctx-size is the total KV budget shared across slots, not the per-slot window — a widely misread flag, and upstream's README does not spell it out. A load with --ctx-size 4096 and no --parallel reports total_slots: 4: 1,024 tokens per conversation. StudioForge launches with ctx_per_slot × parallel. Then: how many slots are worth having, which is where I stopped trusting my own arithmetic.

Concurrent	Per stream	Aggregate	p50	p95	Achieved batch
1	302.8 tok/s	302.8 tok/s	0.41 s	0.41 s	1.00
2	225.3 tok/s	425.3 tok/s	0.46 s	0.49 s	1.84
4	134.5 tok/s	436.0 tok/s	0.83 s	1.00 s	3.46
8	83.3 tok/s	576.9 tok/s	1.57 s	1.77 s	6.03

Qwen2.5-1.5B-Instruct-Q4_K_M, one RTX 3090, 8,192 tokens per slot, f16 KV, eight slots launched, 512-token prompts, 192 generated tokens each.

What the table says:

• The estimator said 8. The measurement said 2. At four slots each stream drops to 44% of solo speed, under a 65% floor; the rule takes the largest of 1/2/4/8 clearing that floor while still gaining 15% of aggregate over the level below.
• The aggregate never stops climbing — eight slots move 1.9× the tokens one does — while a single conversation collapses to 27%. A rule maximising aggregate would pick 8, and every user would experience a model three times slower than the card can run it.
• The batching is real, not queueing. Achieved batch rising 1.00 → 1.84 → 3.46 → 6.03 proves shared decode steps; reproduced across three runs within 2%.

A second run on two 3090s at 32,768 per slot went 301.7 → 230.5 per stream and answered 2 again. Incidentally the catalog had predicted 308.0 tok/s for that placement and the run measured 301.7 — 2% out, better than I expected. Rows now carry max_parallel (how many fit) beside recommended_parallel (how many are worth running).

Two flags I measured before trusting

Speculative decoding is a single-stream win. Qwen3.8-27B Q5_K_S with an MTP head, one 3090, four distinct 256-token prompts, prompt cache off: no speculation 37.75 tok/s; draft-mtp at depth 3 gave 50.70 tok/s, +34.3%, at 0.528 acceptance. Depth 4 goes down to 47.48, because acceptance falls to 0.446 and every extra rejected token was verified for nothing. ngram-mod managed +0.4% and emitted no drafts at all.

Which brings the trap: the same prompt three times measured +751% on that 27B. Repeat one prompt and you are measuring the prompt cache and calling it drafting. Above four slots auto now returns none and says why — "speculation is a single-stream win and hurts a saturated batch" — after a run loaded a 27B at --parallel 8 and auto still chose draft-mtp, seeing the MTP head and not the slot count.

Micro-batch buys prefill for VRAM. Same 1.5B, a 5,166-token prompt: -ub 512 (the engine default) gave 15,232 tok/s at 1492 MiB; -ub 1024, 17,307 tok/s (+13.6%) at 1562 MiB; -ub 2048, 18,061 tok/s (+18.6%) at 1702 MiB. It was off for a long time because the compute buffer grows with -ub, the planner did not model it, and an unmodelled buffer turns a fit into an out-of-memory. The planner charges it now, rounded up so it errs toward refusing, and raises the micro-batch automatically only above four slots.

The refusal, with the numbers

Every launch passes --fit off and --n-gpu-layers 999, and the second is a constant, not a setting. That matters more than it used to: the pinned build b10425 ships -fit, --fit [on|off] — "whether to adjust unset arguments to fit in device memory" — defaulting to on, alongside --n-gpu-layers auto; both landed upstream in PR #16653 in December 2025. That pair is precisely a silent partial-offload path: a reasonable default for a general-purpose server, and the exact behaviour this project exists to refuse. When nothing fits, the answer is HTTP 507 with the arithmetic in the body, trimmed here:

HTTP 507  {"error": {"code": "insufficient_vram", "type": "server_error", "message":
 "Cannot load 'lmstudio-community/gemma-4-31B-it-QAT-GGUF/gemma-4-31B-it-QAT-Q4_0'
  entirely in VRAM: needs 29.09 GiB, 20.90 GiB usable. largest single GPU offers
  20.90 GiB usable (headroom 10% reserved). Suggestions: set KV cache type to q8_0
  (roughly halves KV cache VRAM for a small quality cost); VRAM is held by other
  processes: 5.87 GiB held by python.exe (pid 45072) on CUDA1; 0.83 GiB held by
  dwm.exe (pid 2468) on CUDA0; …; clear the per-model device override so the
  planner can use other GPUs",
 "studioforge": {
   "required_bytes": 31235974510, "available_bytes": 22438368871,
   "per_gpu_free": {"3": 22438368871},
   "max_ctx_that_fits": null, "max_parallel_that_fits": null,
   "suggestions": ["set KV cache type to q8_0 (roughly halves KV cache VRAM
                   for a small quality cost)",
                   "VRAM is held by other processes: …",
                   "clear the per-model device override so the planner can use
                   other GPUs"],
   "notes": ["wanted up to 262144 tokens of context but not even the 128000 floor
             fits in the VRAM available right now",
             "device placement forced by per-model device_override"],
   "estimate_mb": {"weights_bytes": 16818.2, "kv_bytes": 8575.0,
                   "compute_bytes": 2438.6, "mmproj_bytes": 1145.1,
                   "mmproj_compute_bytes": 512.0, "cuda_context_bytes": 300.0,
                   …, "total": 29788.9},
   "vram_holders": [ … one entry per process, per card … ],
   "busy_models": [], "retry_after_s": null }}}

That is a real refusal, captured while writing this: the 31B asked to load onto one RTX 3090. The prose names the shortfall and the holders; error.studioforge carries the same failure as data — the bytes required and available, free VRAM per card, the estimate broken into weights, KV, compute buffer, projection model and CUDA context, every process holding VRAM, and notes saying which rung of the ladder it was standing on when it gave up. When a smaller window would fit, max_ctx_that_fits names it, computed on the per-layer geometry so the offer is one the next load accepts; here not even the floor fit, so it is null rather than a number that would fail. A refusal that is not about a busy model carries no retry_after_s, because "try again later" is bad advice when nothing will change.

Pins, TTLs, leases and the rebalancer

A pin is a desired state, not an exemption. It used to mean TTL zero, exclusion from every eviction ladder and one warm-up at startup — and not the fourth thing: a pinned model was never re-loaded, so a child that crash-looped past its per-model max_restarts sat at state="failed" holding nothing. A reconciler now rides the 15-second sweep, backing off from 60 s to a 900 s ceiling. The one thing that beats a pin is a person: an explicit unload marks the id suppressed and it stays down.

A lease is a card belonging to one model. A leased card is absent from every other model's GPU view — not ranked last, not an option — and the owner is forced onto exactly those cards, sized by the estimator, in the split mode its own benchmark measured fastest there. An already-leased card is a 409 conflict, never a takeover. A lease with no model holds cards for something outside the server: reserve_gpus(devices=[3], reason="ComfyUI render") is why my image generation and my language models stopped fighting.

The rebalancer fixes yesterday's good decision. At 13:42 a 27B was planned onto cards [1, 3] — a cross-tier split sharing GPU1 — because a 31B held [1, 0, 2] and an image-generation app held 7.5 GiB of GPU2. At 13:53 the 31B shrank to [0, 1]; from then [2, 3] sat free and strictly better, and the 27B stayed put. So an idle model is now moved when a no-evict plan at its exact current settings lands it off every shared card. Estimated tokens per second never justify a move.

It looks once a minute, and only when the world changed: only on a quiet box, only for a model idle five minutes, one move per model per 30 minutes — because a relocation is a reload and a reload drops the prompt cache, and on this rig's long-conversation workload that cache was 93% of a 98k-token prompt. Eviction has three hard rules besides — never a pinned model, never one mid-request, never a loading instance — and a leased card is not in the planner's view to begin with. A just-in-time load can never set force.

When things break

VRAM dies with the process that took it. On Windows the children live in an anonymous job object created with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, so the kernel kills every member when the last handle closes — however the parent ends. Anonymous, because a named job would be shared with anything that guessed the name. Linux gets a PR_SET_PDEATHSIG shim, which covers a kill -9 of the gateway; it is a best effort rather than a kernel guarantee, so the startup sweep and reclaim_orphan_engines catch what slips through.

Which is why there is a startup sweep. On August 18, 2026, ~10 GiB on GPU0 and ~15.6 GiB on GPU1 were unavailable with "everything stopped". The holders were three llama-server.exe children of a python -m pytest tests -q run a coding agent had started and that had since exited. Every holder is now classified — ours, child-of-live-process, orphan, other-instance, foreign — and only orphan is ever killed, safe by construction because nothing else launches binaries out of our engines tree.

Naming who holds what took two goes. NVML reports zero used memory per process under Windows, so sizes come from the counter Task Manager's "Dedicated GPU memory" column reads — but that is a per-process total across adapters, so the device column was wrong: a process reported on CUDA0,1,2,3 actually held 15.52 GiB on CUDA0 and nothing on the 3090s. The fix joins the adapter LUID to a PCI bus address, and one trap earns the paragraph: the bus number in NVML's busId is hexadecimal. "00000000:42:00.0" is bus 66, not 42 — a confident wrong answer, indistinguishable from a right one.

An unload is verified, not announced. The supervisor re-checks the pid with a creation-time guard against pid reuse, escalates a survivor to a forced tree-kill, and records VRAM before and after; a survivor raises a 500 telling you to kill it manually. An unload that reports success while the process stays resident is the most expensive lie this system can tell, because every later load is then planned against VRAM that is not free.

Exit codes are vocabulary. 2 is a config error naming the key. 3 is a port conflict, and the tray never respawns onto the conflicting port — it waits for the holder to answer /health as a StudioForge server and attaches instead. 75 is "restart requested": the server drains, sets the code and shuts down gracefully — 1.0 s from request to exit, measured — and the tray respawns without spending a crash attempt. That distinction exists because a GUI restart once produced two servers racing for 1234, and three counted crashes later the tray sat on Crashed — see the logs folder beside a healthy server it could no longer stop.

When the gateway is wedged rather than dead, you talk to the watchdog: a separate always-on process on 1235 built from argparse and stdlib logging, so it starts even when config.yaml is the broken thing. Its ten tools are health, get_config, set_config, restart_server, kill_model, nuke_all_models, reclaim_orphan_engines, tail_logs, gpu_status and rollback_update. It re-implements the orphan rule locally rather than importing the module that owns it — the recovery process must not import the stack it repairs.

The control panel

The panel on 8080 is a second uvicorn server inside the same process, sharing the gateway's object graph by reference — there are no absolute URLs in it at all, which is what makes it work identically over plain HTTP on a mesh VPN and behind an HTTPS front end.

Four lines further down the same buffer are the ones I read when a load surprises me — the command line it built, the process that answered, and the planner marking its own homework (timestamps and logger names trimmed):

model_spawn  argv='C:\Users\<you>\…\engines\b10425\llama-server.exe
  --model E:\LLM\Models\…\gemma-4-31B-it-QAT-Q4_0.gguf --host 127.0.0.1 --port 18100
  --n-gpu-layers 999 --ctx-size 262144 --parallel 1 --device CUDA1,CUDA0
  --tensor-split 0.5368,0.4632 --split-mode layer --cache-type-k q8_0 --cache-type-v q8_0
  --flash-attn on --fit off --cache-reuse 256 --cache-ram 32603 --reasoning-format deepseek
  --mmproj …'  port=18100 source=jit:/v1/chat/completions
model_ready  pid=34732 port=18100 source=jit:/v1/chat/completions
load observation  actual_mb=37046 predicted_mb=33031 ratio=1.122 ctx=262144
  devices=[1, 0] per_device_mb={'0': 18908, '1': 18138}
[warning] a device holds more than its planned share  devices=[1, 0]
  overruns={'CUDA0': {'planned_mb': 15886, 'actual_mb': 18908}}
  detail='llama.cpp places the output layer on the last device of the list; the planner
  now charges it there, so a persistent overrun means the charge is too small for this model'

That is the planner catching itself out by 3 GB on one card, 4 GB across the pair, and saying which card and why — the output layer landed on the last device, exactly where it charges for it, and the charge was still short.

Using it as a harness backend

Six clients on my network talk to this thing and only one of them knows it is StudioForge. That is the point. Five of them are drawn below; OpenClaw Email is the sixth.

Any OpenAI client

Two environment variables. server.api_key is null out of the box, so any non-empty string works — most OpenAI clients refuse to start with an empty one.

export OPENAI_BASE_URL=http://my-gpu-rig:1234/v1
export OPENAI_API_KEY=not-required        # any non-empty string while server.api_key is unset

curl http://my-gpu-rig:1234/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "<id from /v1/models>", "messages": [{"role": "user", "content": "hello"}]}'

from openai import OpenAI
client = OpenAI(base_url="http://my-gpu-rig:1234/v1", api_key="none")   # any non-empty string, until you set one
print(client.models.list())   # every downloaded model; naming an unloaded one loads it on demand

GET /v1/models lists everything downloaded, LM Studio-style, adding state and — when resident — ctx_per_slot, max_parallel and parallel_limited_by, because a context length alone is ambiguous once a model runs more than one slot. Ids round-trip: the full publisher/repo/file id, a bare filename, or publisher/name, case-insensitively. DisPatch needed nothing but a new base URL; OpenClaw Email is the more demanding client, wanting a chat model and an embedding model and calling /v1/models at startup to ask what is really being served.

OpenClaw, on another machine

The agent box installs one small wheel: sfctl, which targets Python 3.11 rather than the server's 3.12 because the machine running the agent frequently lags the rig, and which deliberately does not depend on the server package — no CUDA, no planner, no registry.

sfctl servers add rig http://my-gpu-rig:1234 --api-key <PIN> --use
openclaw mcp add studioforge --command sfctl --arg mcp

Or by hand — the detail that costs people an afternoon. OpenClaw's key is mcp.servers, nested under mcp: the flat mcpServers map is right for Claude Code, Cline and LibreChat, and is not a key OpenClaw's schema knows. Inference is a separate path, under models.providers — note baseUrl, with a lower-case rl:

// ~/.openclaw/openclaw.json
{ "mcp": { "servers": {
      "studioforge": { "command": "sfctl", "args": ["mcp"] } } },
  "models": { "providers": {
      "studioforge": {
        "baseUrl": "http://my-gpu-rig:1234/v1",
        "apiKey": "not-required",
        "api": "openai-completions",
        "models": [ { "id": "<id from /v1/models>", "name": "Rig 27B", "contextWindow": 131072 } ] } } } }

What the agent gets is one merged tool list of 29: the gateway's 19 plus the watchdog's 10, three renamed recovery_* — get_config and set_config because they collide with gateway tools, health for symmetry. restart_server keeps its bare name, because it is the name the error message for a dead management tool tells the agent to call. When the main server is down the bridge still advertises all 19 management tools with a note appended — an agent that cannot see load_model does not know the capability exists. The loop it runs:

1. list_models(limit=N) — the catalog, newest download first. Read the recommended row.
2. load_model(**row["load_args"]) — pass it through unchanged; an agent that has chosen a row is done choosing.
3. load_recommended(model_id, ctx_size=N) when what you know is the context you need — the one load path that refuses rather than shrinks.
4. Inference over HTTP, not MCP. Naming an unloaded model loads it, with planner defaults rather than the row you were reading.
5. model_options(model_id) when the recommended row is not enough: every context tier, with speeds.
6. search_models → repo_details → download_model to get something new.
7. pin_model for the model that must always answer; reserve_gpus / release_gpus for cards of its own.
8. server_status and connection_info — what is resident, who holds VRAM, every address it answers on.

And the paragraph I am most pleased with, served to every client on connect:

INFERENCE IS NOT HERE. This server exposes no chat/completion/generation tool
by design. To actually run a prompt, use the OpenAI-compatible HTTP API on the
gateway port (POST /v1/chat/completions, /v1/embeddings; GET /v1/models).
Naming an unloaded model in a request just-in-time loads it, so you usually do
not need load_model at all -- reach for it only to pre-warm a model or to load
one with non-default context/quantization settings.

dsh, Claude Code, and five one-liners

dsh (DeepSeek Harness) switches models by editing one YAML file, hot-reloaded for the next request. The provider block is camelCase — baseURL, apiKeyEnv:

# ~/.dsh/settings.yaml
llm-pi-ai:
  providers:
    gpu-rig:
      displayName: StudioForge (GPU rig)
      apiKeyEnv: STUDIOFORGE_PLACEHOLDER_KEY   # a reference, not a value
      api: openai-completions
      baseURL: http://my-gpu-rig:1234/v1
      defaultContextWindow: 131072
      compat:
        supportsDeveloperRole: false   # many local servers reject role: "developer"
        maxTokensField: max_tokens
      models:
        - id: <id from /v1/models>

apiKeyEnv is a reference, never a literal — and a keyless local server still needs some credential referenced, because the OpenAI-compatible client insists on a bearer token. Claude Code, by contrast, cannot route its own inference here at all: its gateway protocol reference lists Anthropic Messages, Bedrock and Vertex, and none is /v1/chat/completions. So StudioForge is its tool, not its brain — claude mcp add studioforge -- sfctl mcp (the -- is mandatory) hands it all 29.

bench-llm produced the rig figures people quote back at me — Gemma 4 26B-A4B (QAT, Q4) at 229.0 tok/s, 110 ms to first token, measured through LM Studio at the time — and it is a plain OpenAI client, so a base URL is all it needs; but it runs pkill -f llama-server between benchmarks, which kills every StudioForge backend on the box. The rest are one line each: Open WebUI, OPENAI_API_BASE_URL; LibreChat, a custom endpoint with baseURL and models.fetch: true; aider, OPENAI_API_BASE then --model openai/<id>; Continue, provider: openai plus apiBase. The base-URL key is spelled differently in every one of them, which is the most reliable source of wasted evenings in this ecosystem.

What an agent picks from

The catalog makes a model choice a lookup rather than a guess: sorted newest download first, one row per context tier, each carrying fits against live free VRAM, devices, KV types, max_parallel, recommended_parallel, a confidence, an if_gpus_idle column and load_args. A row is one real plan call, so it cannot promise what a load would refuse — and if_gpus_idle is the difference between an agent giving up and an agent calling unload_model.

repo_details is the one worth naming: it reads a GGUF header remotely, over HTTP Range requests — 2–15 MB, mostly the tokenizer's length-prefixed string arrays, cached on disk — instead of downloading 20 GB to find out whether it fits, and a CDN that answers a ranged request with a 200 and the whole body is detected and refused. What comes back is a context_fit matrix from the same planner a real load uses:

Quant	1× RTX 5090	2× RTX 5090	All four cards
BF16 (51.8 GiB)	— weights alone do not fit	32k at q8_0	256k
Q8_0 (27.9 GiB)	—	256k	256k
Q5_K_M (19.3 GiB)	128k at q8_0	256k	256k
IQ2_M (10.5 GiB)	256k	256k	256k

From the repo's OpenClaw guide, computed on this rig, for unsloth/Qwen3.8-27B-GGUF. max_ctx is the largest window at a full-quality f16 cache; a q8_0 figure appears only where it reaches further.

What the table says: the quantization decides the window, not the card count — BF16 to Q5_K_M turns "does not fit at all" into 128k on one card — and it is the planner's own answer, so where a tier is only reachable by quantizing the cache the matrix says "at q8_0" rather than counting it as a win. For real numbers the playbook is three steps: a placement benchmark (each GPU mode under its own lease, throughput from llama-server's own timings), then benchmark_parallel on the winner, then reserve_gpus to lock it in. Never benchmark a model somebody is mid-conversation with.

Drop-in for LM Studio

Compatibility was the design constraint, and the repo lists what was borrowed so nobody has to guess: port 1234; /v1/models listing downloaded rather than loaded; just-in-time load; idle TTL; the per-request ttl; the publisher/repo/ layout used in place, so there is no import step and both programs share one library; the /api/v0/models mirror; and the lmstudio://open_from_hf deep link. Migrating a client is a host change, not a host-and-port change.

Checked against LM Studio's own changelog, docs and bug tracker on 2026-08-23, version 0.4.21 (released August 12, 2026). Row 2 is in its public bug tracker; rows 3–4 are what my own client had to work around on the 0.3.x API. I have not re-tested any of them against 0.4.21, so read them as "documented at some point", not "broken today".

What the table says: it is a better desktop app than this will ever be — polished GUI, MLX on Apple Silicon, an Anthropic-compatible endpoint, a mobile companion, a team shipping fortnightly — and the split is philosophical rather than featural. LM Studio's default posture is best-effort: make it run somehow. Mine is refuse-and-explain. Where an agent decides what loads, best-effort is the wrong default, because nothing downstream can tell fast from slow without measuring.

How it compares

Checked from primary sources on 2026-08-23. A question mark means unknown, not "no". GPUStack's workers are Linux-only; vLLM has no native Windows support (WSL or forks only).

What the table says:

• JIT loading and idle TTL are not novel and I am not claiming them. LM Studio, Ollama, llama-swap and LocalAI all do both, and llama-swap's per-group swap / exclusive / persistent flags are a genuinely elegant policy engine.
• Three things there are rare: refusing to run rather than spilling to CPU (only TabbyAPI comes close, and only because ExLlama has no CPU path); a planner that sizes context and slots to the cards it actually found; and management published as MCP tools, which nothing else in the category does that I can find.
• The nearest thing upstream is llama.cpp's own router, a fair fight: multi-model with process isolation, free, in the binary you have. What it lacks is memory-based rather than count-based eviction, pins, leases, and a refusal carrying numbers; it does have an idle sleep (--sleep-idle-seconds), though a /metrics poll wakes it.

Several good ideas are borrowed, and the repo says which. Ollama's Modelfile became virtual models, so two personas over one base share a single llama-server, and its keep_alive became the per-request ttl. TextGen's three-tier settings surface was adopted directly, raw "extra flags" included, with flags validated against the pinned engine's own --help at save time. KoboldCpp's single-artifact philosophy is why engines live in versioned directories.

Install

Windows, the reference platform, in four steps: install Git, Python 3.12+, uv and a current NVIDIA driver; clone the repo or unzip the download; double-click launchers\Update StudioForge.bat, which despite its name is the first-run step — it builds the virtualenv, installs the newest llama.cpp release that has a build for your driver, smoke-tests it and pins to it (b10425 is the build this article was measured on, not the one you will get); then launchers\Start StudioForge.bat, or launchers\StudioForge Tray.bat if you want it in the notification area, and the panel opens at http://127.0.0.1:8080 on its Setup tab.

Linux, four lines — plus cmake and a CUDA toolkit whose nvcc matches your driver, because upstream publishes no Linux CUDA archive at any tag and the engine is built from source once per version:

git clone https://github.com/LaserLloyd/StudioForge.git && cd StudioForge
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e ".[dev]"
.venv/bin/studioforge serve --open     # first run builds the engine

For a headless box, deploy/ holds two systemd user units — user units on purpose, because the process must run as the login user that owns the model library, the venv and the GPU device nodes. The watchdog is deliberately not BindsTo= the gateway and uses Restart=always: it exists to be up when the gateway is not. Then sudo loginctl enable-linger "$USER", the same pattern as the headless ComfyUI write-up. First run opens on Setup, where Detect LM Studio library probes the downloadsFolder in ~/.lmstudio/settings.json first.

Service	Default port	Config key
Gateway — /v1, /api, /mcp	1234	server.port
Web control panel	8080	gui.port
Recovery watchdog	1235	watchdog.port
llama-server children (loopback only)	18100–18200	gateway.child_port_start / _end

What the table says: port 1234 means "the local model server" on both of my machines and they are not the same thing — the agent box runs its own on 127.0.0.1:1234, loopback, while the rig serves my-gpu-rig:1234 across the mesh VPN — and only three ports are ever reachable, with config validation refusing a collision between any service port and the child range at load time.

The data directory rule is SF_DATA_DIR first, then the folder of a --config file, then <repo>/data in a checkout — the full order, and why data_dir is never written back into config.yaml, is in docs/SETUP.md. One instance owns one data directory, enforced by an exclusive OS lock; a second instance is read-only.

Security, honestly

One rule underneath all of it: reads, inference and residency stay open; changing the box does not. With server.api_key unset, a mutating request to a box-changing route is accepted only from a caller on this machine, or with the MCP PIN sent as X-MCP-Pin or as the bearer token — anything else gets 403 remote_admin_requires_credential. The gated set is config, restarts, engines, updates, VRAM reclaim, downloads, leases, deletes and the two per-model writes that outlive the instance. The problem it fixed was mine: anyone on the LAN could PATCH /api/config, set server.api_key themselves and lock me out — while the MCP set_config tool, same capability in the same process, demanded the PIN.

• The PIN guards MCP only. It is a pairing code you read off the startup banner, scoped to the management tools. It is not an API key.
• server.api_key is the real credential and is null by default. Set it and it covers /v1, /api, /mcp and the watchdog; the PIN keeps working on the two MCP endpoints beside it.
• The shipped bind is 0.0.0.0 on all three listeners. The Setup tab's Network exposure row turns amber and required the moment any listener is exposed with no key — checking all three, because server.host on loopback with gui.host on 0.0.0.0 used to read green while the panel was wide open.
• A cross-origin browser request is not "this machine", even on loopback. With cors_origins: ["*"] any page you visit could preflight PATCH /api/config at 127.0.0.1:1234 and arrive looking local — so the origin comparison includes the port, and Origin: null counts as foreign. CORS governs what a page may read, never whom the server trusts. The panel's websocket has a host-only version of the same gate, because the panel is reached through the port it was served from.
• A remote browser on a keyless install gets reads and inference, 403s on box changes, and the PIN withheld — otherwise anything on the LAN could read the PIN off an open endpoint and use it. The PIN was theatre exactly when it mattered.
• Images are fetched under an SSRF guard that blocks loopback, link-local, private, ULA and CGNAT space — the 100.64/10 range, where every mesh-VPN peer lives — and resolves once, connecting to the vetted address with the original Host and SNI.
• Nothing leaves the box unasked. The only outbound calls are Hugging Face for models, GitHub for the pinned llama-server build and its update check, the opt-in StudioForge release check, and image URLs a request names; self-update reports "not configured" without a network call until you set update.repo, and a unit test pins that.

Two limits stated rather than hidden: a peer-address check trusts whatever is on loopback, which behind a reverse proxy is the proxy, so put the proxy behind server.api_key; and there is no auth beyond one shared key — no accounts, no rate limiting. The house rule from the OpenClaw write-up still applies: loopback plus an authenticating proxy, or a mesh VPN, never the raw internet.

The honest part

Written down plainly so you can decide before you install it:

• Windows is the reference platform — it is what the tray, the job-object VRAM guard and the per-process GPU counters were built against. Linux is supported and CI runs both, and is less battle-tested; the source-build path has its command construction tested but has never been exercised end to end here. macOS is not supported: no CUDA.
• NVIDIA only. The planner reads NVML, the engine is a CUDA build, and quant affinity is expressed in compute capabilities.
• The VRAM estimate is an estimate. Weights land within 2% of file size and KV is exact from the per-layer geometry, but the compute buffer is a calibrated fraction, tuned once at startup, clamped to 0.03–0.15 and held in memory only — so a bad calibration is undone by a restart. Two calibration histories on this box were contaminated and are now ignored entirely.
• Multi-GPU splitting is proportional, not measured. It does not model interconnect bandwidth, and mixing generations runs at the slower card's pace.
• Concurrency estimates are arithmetic. The estimator assumes slots sit half full and derates MoE models by a flat half, and --ctx-checkpoints is not modelled at all. Both errors point toward fewer slots, which is the safe direction — but run the parallel benchmark before trusting 8.
• The speed estimates use nominal vendor figures — 5090 at 1792 GB/s and 209 fp16 TFLOPS, 3090 at 936 and 71, none measured here. There are exactly two calibration anchors, both on this rig, both at one slot: a dense 31B measured 39.4 tok/s against an estimate of 36.1, and a 122B MoE measured 37.3 against 47.4. Nothing is validated at four or eight slots, or on a dense model above 31B.
• Reserving a GPU for another program constrains my planner only. Nothing enforces it against the other program, and nothing stops it taking the memory first.
• One instance per data directory, and the lock covers the data directory rather than the model library — two instances with different data directories over one library are still two writers.
• No licence has been chosen. There is deliberately no LICENSE file, and pyproject.toml says so in a comment — Get it has what that means in practice.
• There is no third-party security audit. The claims above describe what the code does; I wrote both. Read the source — that is why it is a download and not a service.

2,503 unit tests pass and 17 skip, in 327 seconds on this box; CI runs the same suite on Windows and Ubuntu with the GPU probe forced to a null backend. A second suite loads real weights onto real GPUs and is deselected by default and gated behind an environment variable — belt and braces, after the orphan incident above. Not done, or not switched on: the 8-slot micro-batch A/B, named sampler presets and mypy in CI; app self-update is written but stays off until you set update.repo yourself.

Gotchas

The honest list — things that actually bit, in rough order of how much time they cost:

• LM Studio on 1234 must be quit first. Both cannot hold the port; the preflight names the holder rather than printing a bind traceback, and server.port moves it. The library is fine to share.
• Two llama.cpp flags do not mean what they look like. --ctx-size is the budget across all slots, not per slot; --fit defaults to on upstream, alongside --n-gpu-layers auto — both above, under the planner.
• Reasoning models return an empty reply under --reasoning-format auto. Same prompt, only the flag changed: content 0 characters and reasoning_content 316 under auto; content 323 under none. reasoning_content is not in the OpenAI schema, so a standard client reads an empty string and concludes the model said nothing. I run the 31B on deepseek because my clients read that field; everything else gets none.
• llama.cpp vX.Y.Z prereleases carry no Windows CUDA asset. One tagged v0.1.2 sat above two ordinary bNNNN builds with no prebuilt archives at all, so the Server tab offered an update behind a button that could only ever fail. Tags are filtered to ^b\d+$ now.
• Never tree-kill the tray's root with a browser tab open on the panel. Under a venv launcher stub the watchdog is the server's grandchild, and one restart once ended with the server dead, the watchdog dead and nothing spawned. Use the tray menu, the panel, or sfctl recover --restart.
• pkill -f llama-server from another tool kills your backends. bench-llm does exactly that between runs — above, under the harness clients.
• The MCP PIN is not an API key, and a keyless server still needs a placeholder credential in some clients — above, under dsh.
• OpenClaw's MCP key is mcp.servers, nested under mcp. The flat mcpServers map is not a key its schema knows — above, under OpenClaw.
• Two measurement shortcuts will lie to you. Repeating one prompt in a speculative benchmark times the prompt cache (+751% against +0.4%), and a layers × heads × context KV formula is 4× out on a Qwen3.5 — both above, under the planner. /props also reports speculative.types: "none" while drafting, so read timings.draft_n off a real completion.
• Vision models get no prompt-cache benefit. llama.cpp disables cache reuse for multimodal models itself, and each image is budgeted at 1,024 tokens unless the mmproj metadata says otherwise — enough that an 8k window is mostly pictures.

Get it

The zip below is the whole thing: the tagged source tree, the tests, the docs, the launchers, the systemd units, plus a dist/ folder holding both wheels and the sdist so you can install without a build step. studioforge-2026-08.zip — release v1.26-08-23, 4,238,728 bytes (4.04 MiB), SHA-256:

84f4f828b5c75206236890f28e8651c96146a7bb39c14e21b13a0922a13f7d3f  studioforge-2026-08.zip

226 entries under one top-level directory, built with git archive from the annotated tag v1.26-08-23 at commit 0610446, so it can only contain tracked files — no config.yaml, no data/, no local overrides. The source is also at github.com/LaserLloyd/StudioForge. It needs Python 3.12+, uv, an NVIDIA driver from the 580 series up, and a folder of GGUFs. Licence: not yet chosen, so all rights are formally reserved — in practice, treat it like the rest of this site's downloads: free for personal use, and if you want it commercially, ask me.

If something breaks, email me — the address is on the About page — and send the shape of the failure rather than your config: the error.code, the numbers out of a 507, the last twenty lines of logs/models/<model>.log. Never the PIN or the key. And if you build the joint bin-packing planner, the named sampler presets, or an AMD path that actually works before I do, I would rather merge yours than write mine.

Where this leaves me

What I did not expect was how much of this turned out to be measurement rather than code. The planner is arithmetic anybody could write; what made it trustworthy was reading llama.cpp's own KV geometry instead of a formula, then measuring the slot knee at 2 when the estimator said 8. Nearly every decision in that log started as a number that disagreed with a belief. If you already run LM Studio on 1234, the whole experiment is a clone, one batch file, and pointing models.dir at the folder you already have — and if it does not earn its place in an afternoon, your old setup is untouched.

Related: DeepSeek Harness (dsh) (where this server first appears, as an unexplained provider block), bench-llm (where the rig's tokens-per-second numbers come from, and the tool that will kill your backends), My OpenClaw Setup (the post that named the problem this solves), and DisPatch (the chat app in front of it).

Downloads

• Download StudioForge source (zip) 4.0 MB

Free for personal use. If it saves you an afternoon, the coffee button's nearby.