Putting a Chatbot on the Public Web — On My Own Hardware

I wanted a chatbot on a public website — but I didn’t want to rent a cloud AI to
run it. I wanted the model to run on my own computer, in my house: free per
token, private, and mine. So I built exactly that: a chat assistant that visitors
use in the browser, while the actual thinking happens on a small model on a box at
home.

It works. It’s also slow, it was genuinely dangerous before I hardened it,
and it is emphatically not ready to be pointed at real traffic. This article is
the honest version — the architecture that makes it safe, the holes I had to close,
and a frank list of what it would still need to be a real product. If you want to see
where a bot like this actually lives, it’s the assistant inside my
family trip itinerary app.

The architecture: a mailbox in the middle

The instinct is to have the website call your home computer directly. Don’t. Your
home box shouldn’t be a public server, and you shouldn’t be poking holes in your home
network to expose it. Instead, put a dead-drop in the middle.

The flow is a mailbox drop:

1. The browser posts a message to a tiny relay script on the web host, which drops
   it in a queue and hands back a ticket.
2. A worker on my home box polls the relay, pulls any waiting message, and asks the
   local model.
3. When the model finishes, the worker pushes the reply back to the relay.
4. The browser has been polling with its ticket, and picks up the answer.

The web host only ever holds text in a mailbox. It never opens a connection into my
house; my worker reaches out to the relay and back. Nothing at home is exposed.

Why not just one request and response?

Two reasons, and they’re the whole justification for the queue.

First, a small local model is slow. A normal web request that waited for the model
to finish would routinely blow past timeouts. The “submit a ticket, then poll for the
answer” pattern comfortably survives a reply that takes 10, 20, 30 seconds — the
initial post returns instantly, and the slow part happens out of band.

Second, it decouples the exposed surface from the model. The public web host runs a
few lines that shuffle text. The thing with real power — the model, and anything it can
do — lives at home, behind the worker, where I control it completely.

Security: a chatbot that can do things is a foot-gun

This is the part I want people to take seriously, because I didn’t take it seriously
enough at first and a review of my own system found two genuinely dangerous problems.

A bot that just chats is low-risk. The moment it can act — run a search, edit
data, touch a file — it becomes an attack surface, because the text it’s acting on
comes from strangers on the internet. Someone will type “ignore your instructions
and…”. Here’s what closing that down actually looks like:

• No general tools. Ever. The model does not get shell access, file access, or a
  general “run this” capability. What it’s allowed to do goes through a tiny, explicit
  allowlist — in my case, add-or-edit-a-note only. No delete, no commands, nothing
  outside the list. One review of mine found a path where a crafted chat message could
  have pivoted into running commands; the fix was to delete that capability entirely and
  replace it with the toolless allowlist.
• Treat every message as hostile. User input is length-capped, sanitized, and handed
  to the model framed as untrusted — clearly marked as “this is a visitor’s text,
  not an instruction to you.” Anything the bot fetches (a web search result) gets the
  same untrusted wrapper.
• Fail closed, and keep secrets off the wire. The sensitive pieces bind to
  loopback, not the whole network; the shared secret the worker uses is never sent to
  the browser; and if anything is misconfigured, the default is refuse, not allow.
  (My review also caught a moment where a secret was briefly reachable on the local
  network — exactly the kind of thing you only find by looking hard.)
• Least privilege everywhere. The registered assistant can do precisely one gated
  thing and nothing else. If you wouldn’t give a stranger the capability, don’t give it
  to a bot acting on a stranger’s words.

If you build one of these, budget real time for this section. A helpful-looking bot
that will cheerfully follow a malicious instruction is not a feature; it’s an incident
waiting to happen.

Keeping a small model useful

A small model has a small memory (context window), so you can’t stuff your whole
knowledge base into every message — it overflows and the model returns nothing, or
garbage. The trick is relevance injection: always include a compact “at-a-glance”
summary, and then pull in only the specific reference the question actually needs,
keyed off what the user asked. Small, focused prompts keep a small model both fast(er)
and accurate.

The honest part: it’s slow

Let’s be clear-eyed. On consumer hardware, a small local model answers in roughly
10 to 24 seconds once it’s warm, and the first request after idle can take ~50
seconds while the model loads into memory. That’s not a bug I can prompt my way out
of — it’s just how fast the hardware generates tokens.

The relay hides it a little: a typing indicator and a phase status (“thinking…”,
“searching…”, “writing…”) make the wait feel intentional rather than broken. But there’s
no illusion that this is fast. For its actual job — a handful of family members asking
the occasional trip question — it’s completely fine. Point it at real public traffic and
it would fall over: every visitor is competing for one slow model on one machine.

What it would still need to be “production”

I’m calling this what it is: a working hobby-scale proof, not a scalable service.
Before it could handle real traffic, it would need, roughly in order:

• Streaming. Stream tokens as they’re generated so replies feel instant even
  though the total time is unchanged. This is the single biggest perceived-speed win and
  the first thing I’d add.
• A faster brain. A more aggressively quantized model, a GPU, or simply a bigger
  box. The model and the hardware are the real bottleneck; everything else is polish.
• No cold starts. Keep the model warm/resident so nobody eats the ~50-second first-
  load penalty.
• A real queue. My mailbox is a simple filesystem drop — fine for a few people, not
  for concurrency. A proper message queue with real locking would replace it.
• Caching. Common questions should return a cached answer instantly instead of
  re-running the model every time.
• Rate-limiting and abuse protection. Public means bots and floods; you need per-user
  limits, a queue cap, and a way to shed load gracefully.
• Concurrency limits and back-pressure. One slow model can serve only so many people
  at once; the system needs to say “busy, try again” instead of collapsing.
• Observability. Logs, timing, and error tracking, so you can see it straining before
  it breaks.

None of that is exotic — it’s the standard distance between “it works on my box” and “it
works for everyone.” I just haven’t crossed it, because for a private family assistant I
don’t need to.

Build one yourself

The recipe, condensed — and hand the whole article to your assistant via the box at the
top:

1. Run a local model on your machine (LM Studio, Ollama, llama.cpp) serving a small
   chat model over a normal API.
2. Put a relay/queue on your web host — a tiny script that accepts a message, stores
   it, and hands back a ticket; the browser polls the ticket for the reply.
3. Run a worker at home that polls the relay, calls your local model, and pushes the
   answer back. Your machine reaches out; nothing reaches in.
4. Authenticate the chat, cap and sanitize input, and frame it as untrusted.
5. Fence the model behind a strict allowlist — no shell, no delete, least privilege —
   and fail closed.
6. Set expectations. Add a typing indicator, and know that seconds-per-reply is the
   normal, honest speed of a small model on your own hardware.

Gotchas, collected

• Don’t expose your home box — use a relay the worker reaches out to; never open your
  home network to the web.
• Single request/response will time out — a slow model needs a submit-then-poll flow.
• A bot that can act is an attack surface — no general tools, strict allowlist, treat
  all input as hostile, fail closed.
• Small context = relevance injection — inject a compact brief plus only the reference
  the question needs, not everything.
• It’s slow, and that’s the hardware — hide it with streaming and status, but don’t
  pretend it’s fast; it isn’t production-ready without the list above.