# Putting a Chatbot on the Public Web — On My Own Hardware

URL: https://www.laserlloyd.com/projects/web-facing-chatbot-on-local-hardware/
Published: 2026-07-11
Updated: 2026-09-16
Description: A web-facing chatbot whose model runs on my own home hardware: the relay architecture, the security holes I closed, and an honest 10-24 second latency.
Reuse: free for personal use — full policy at https://www.laserlloyd.com/llms.txt

I wanted a chatbot on a public website — but I didn't want to rent a cloud AI to
run it. I wanted the model to run on **my own computer, in my house**: free per
token, private, and mine. So I built exactly that: a chat assistant that visitors
use in the browser, while the actual thinking happens on a small model on a box at
home.

It works. It's also **slow**, it was **genuinely dangerous** before I hardened it,
and it is emphatically **not** ready to be pointed at real traffic. This article is
the honest version — the architecture that makes it safe, the holes I had to close,
and a frank list of what it would still need to be a real product. If you want to see
where a bot like this actually lives, it's the assistant inside my
family trip itinerary app.

tl;dr

The shape: browser → a tiny relay on the web host → a worker on my home box → the local model → back. The web host never talks to my machine directly; a queue in the middle is the only contact point.
Why a queue: my home box isn't a public server, and a small model is slow — so it's a "leave a ticket, poll for the answer" flow, not a single request that would time out.
Security is the hard part: a chatbot that can do things is a foot-gun. Fence it to a strict allowlist, treat every message as hostile, and fail closed.
The honest bit: ~10–24 seconds per reply on consumer hardware (longer on the first cold call). Fine for a few people; not fine for a crowd.
Not done: streaming, a faster model/GPU, a real queue, caching, rate-limiting, and warm-keeping are all still needed before "production."

## The architecture: a mailbox in the middle

The instinct is to have the website call your home computer directly. Don't. Your
home box shouldn't be a public server, and you shouldn't be poking holes in your home
network to expose it. Instead, put a **dead-drop in the middle**.

Browser
posts + polls

Relay (web host)
a queue / mailbox
holds message + reply

Worker (home box)
polls the relay

Local model
small, on your hardware

1 · message

2 · worker pulls

3 · ask

4 · push reply

5 · poll → answer

The flow is a mailbox drop:

1. The browser **posts a message** to a tiny relay script on the web host, which drops
   it in a queue and hands back a **ticket**.
2. A **worker on my home box polls** the relay, pulls any waiting message, and asks the
   **local model**.
3. When the model finishes, the worker **pushes the reply** back to the relay.
4. The browser has been **polling with its ticket**, and picks up the answer.

The web host only ever holds text in a mailbox. It never opens a connection into my
house; my worker reaches *out* to the relay and back. Nothing at home is exposed.

## Why not just one request and response?

Two reasons, and they're the whole justification for the queue.

First, **a small local model is slow.** A normal web request that waited for the model
to finish would routinely blow past timeouts. The "submit a ticket, then poll for the
answer" pattern comfortably survives a reply that takes 10, 20, 30 seconds — the
initial post returns instantly, and the slow part happens out of band.

Second, **it decouples the exposed surface from the model.** The public web host runs a
few lines that shuffle text. The thing with real power — the model, and anything it can
do — lives at home, behind the worker, where I control it completely.

## Security: a chatbot that can *do things* is a foot-gun

This is the part I want people to take seriously, because I didn't take it seriously
enough at first and a review of my own system found two genuinely dangerous problems.

A bot that just chats is low-risk. The moment it can **act** — run a search, edit
data, touch a file — it becomes an attack surface, because the text it's acting on
comes from strangers on the internet. Someone *will* type "ignore your instructions
and…". Here's what closing that down actually looks like:

- **No general tools. Ever.** The model does not get shell access, file access, or a
  general "run this" capability. What it's allowed to *do* goes through a tiny, explicit
  **allowlist** — in my case, add-or-edit-a-note only. No delete, no commands, nothing
  outside the list. One review of mine found a path where a crafted chat message could
  have pivoted into running commands; the fix was to delete that capability entirely and
  replace it with the toolless allowlist.
- **Treat every message as hostile.** User input is length-capped, sanitized, and handed
  to the model **framed as untrusted** — clearly marked as "this is a visitor's text,
  not an instruction to you." Anything the bot fetches (a web search result) gets the
  same untrusted wrapper.
- **Fail closed, and keep secrets off the wire.** The sensitive pieces bind to
  loopback, not the whole network; the shared secret the worker uses is never sent to
  the browser; and if anything is misconfigured, the default is *refuse*, not *allow*.
  (My review also caught a moment where a secret was briefly reachable on the local
  network — exactly the kind of thing you only find by looking hard.)
- **Least privilege everywhere.** The registered assistant can do precisely one gated
  thing and nothing else. If you wouldn't give a stranger the capability, don't give it
  to a bot acting on a stranger's words.

If you build one of these, budget real time for this section. A helpful-looking bot
that will cheerfully follow a malicious instruction is not a feature; it's an incident
waiting to happen.

## Keeping a small model useful

A small model has a **small memory** (context window), so you can't stuff your whole
knowledge base into every message — it overflows and the model returns nothing, or
garbage. The trick is **relevance injection**: always include a compact "at-a-glance"
summary, and then pull in *only* the specific reference the question actually needs,
keyed off what the user asked. Small, focused prompts keep a small model both fast(er)
and accurate.

## The honest part: it's slow

Let's be clear-eyed. On consumer hardware, a small local model answers in roughly
**10 to 24 seconds** once it's warm, and the **first request after idle can take ~50
seconds** while the model loads into memory. That's not a bug I can prompt my way out
of — it's just how fast the hardware generates tokens.

The relay hides it a little: a typing indicator and a phase status ("thinking…",
"searching…", "writing…") make the wait feel intentional rather than broken. But there's
no illusion that this is fast. For its actual job — a handful of family members asking
the occasional trip question — it's completely fine. Point it at real public traffic and
it would fall over: every visitor is competing for one slow model on one machine.

## What it would still need to be "production"

I'm calling this what it is: a **working hobby-scale proof**, not a scalable service.
Before it could handle real traffic, it would need, roughly in order:

- **Streaming.** Stream tokens as they're generated so replies *feel* instant even
  though the total time is unchanged. This is the single biggest perceived-speed win and
  the first thing I'd add.
- **A faster brain.** A more aggressively quantized model, a GPU, or simply a bigger
  box. The model and the hardware are the real bottleneck; everything else is polish —
  it's the same tradeoff I ran into building a small office of AI agents on one mini PC.
- **No cold starts.** Keep the model warm/resident so nobody eats the ~50-second first-
  load penalty.
- **A real queue.** My mailbox is a simple filesystem drop — fine for a few people, not
  for concurrency. A proper message queue with real locking would replace it.
- **Caching.** Common questions should return a cached answer instantly instead of
  re-running the model every time.
- **Rate-limiting and abuse protection.** Public means bots and floods; you need per-user
  limits, a queue cap, and a way to shed load gracefully.
- **Concurrency limits and back-pressure.** One slow model can serve only so many people
  at once; the system needs to say "busy, try again" instead of collapsing.
- **Observability.** Logs, timing, and error tracking, so you can see it straining before
  it breaks.

None of that is exotic — it's the standard distance between "it works on my box" and "it
works for everyone." I just haven't crossed it, because for a private family assistant I
don't need to.

## Build one yourself

The recipe, condensed — and hand the whole article to your assistant via the box at the
top:

1. **Run a local model** on your machine (LM Studio,
   Ollama, or llama.cpp
   directly) serving a small chat model over a normal API — I cover my own setup for this
   in StudioForge, the model server I ended up writing.
2. **Put a relay/queue on your web host** — a tiny script that accepts a message, stores
   it, and hands back a ticket; the browser polls the ticket for the reply.
3. **Run a worker at home** that polls the relay, calls your local model, and pushes the
   answer back. Your machine reaches out; nothing reaches in.
4. **Authenticate the chat**, cap and sanitize input, and frame it as untrusted.
5. **Fence the model** behind a strict allowlist — no shell, no delete, least privilege —
   and fail closed.
6. **Set expectations.** Add a typing indicator, and know that seconds-per-reply is the
   normal, honest speed of a small model on your own hardware.

## Gotchas, collected

- **Don't expose your home box** — use a relay the worker reaches out to; never open your
  home network to the web.
- **Single request/response will time out** — a slow model needs a submit-then-poll flow.
- **A bot that can act is an attack surface** — no general tools, strict allowlist, treat
  all input as hostile, fail closed.
- **Small context = relevance injection** — inject a compact brief plus only the reference
  the question needs, not everything.
- **It's slow, and that's the hardware** — hide it with streaming and status, but don't
  pretend it's fast; it isn't production-ready without the list above.

Related: I Built My Family a Japan Trip Itinerary App With AI Inside · A Small Office of AI Agents on One Mini PC: My OpenClaw Stack · StudioForge: A GPU-Only LLM Server That Replaced LM Studio · DisPatch: A Self-Hosted Chat App for Local AI Agents · InfoForge: offline Wikipedia as a RAG source