InfoForge: Offline Wikipedia as a RAG Source for My Local AI Agents

I asked a local 8B model who wrote Kokoro and when it was published. It answered “Natsume Sōseki, 1914,” and then it did the part I actually cared about: it named the article it read that in. The model does not know that fact. It looked it up, in a 52.69 GB copy of Wikipedia sitting on a machine in my house, with the internet irrelevant to the transaction.

Why bother, when the model already “knows things”

Because it doesn’t, and the failure mode is the ugly kind. A small model asked for a date, a chemical property, or a place name will produce a fluent, confident, plausible answer with no signal whatsoever about whether it is true. It is not lying — it has no mechanism for distinguishing recall from generation. That is fine for prose and catastrophic for facts.

Retrieval fixes it, but the usual retrieval story assumes a cloud search API: your question goes to a company, that company logs it, and your assistant’s usefulness is now a function of somebody else’s uptime and terms of service. In a house where the AI stack answers questions from my kids, both halves of that bother me.

Wikipedia solves both problems at once. It is the single densest general-knowledge corpus that exists, it is legally and practically downloadable in full, and it fits on a disk that costs less than a night out. The Kiwix project has done the hard part for years — packaging whole wikis into the compressed, indexed ZIM format for offline use in schools, ships and places with no connectivity. I am just pointing an AI at it instead of a browser.

What you end up with

A question in, an answer out, and a citations block naming exactly which Wikipedia articles the answer was built from. Something like:

$ infoforge "Who wrote Kokoro and when was it published?"

Kokoro was written by Natsume Sōseki and published in 1914.

Sources:
  - Kokoro

Unremarkable to look at. The point is what is not happening: no network request left the building, nothing was logged to a provider, and the model was structurally prevented from answering from memory. If the search had come back empty, the command would have printed a failure and exited non-zero rather than filling the silence.

How it works

Five stages, none of them clever, which is the recommendation.

Keyword extraction. The question is not the query. “Who wrote Kokoro and when was it published?” is a bad search string; Kokoro is a good one. A few dozen lines of stopword-stripping and phrase-keeping does this well enough that I never replaced it.

Search. Kiwix’s full-text search is a real search index, not a grep, and it is the reason this works at all. Across 19.19 million entries it returns ranked titles in 3 to 30 milliseconds. There are no embeddings anywhere in this system, no vector database, no chunk store to rebuild when the dump updates. Wikipedia’s editors already did the chunking; they call them articles.

Fetch and truncate. The top matches come back as full articles in about half a millisecond each. Truncation is the only part with judgement in it: lead sections carry most of the answerable facts, so the budget is spent from the top down, with enough of each article to be self-contained and not so much that four of them bury the instruction that follows.

The grounded prompt. The articles go into the prompt under an instruction to answer only from those sources and to cite the titles used. This is the whole trick, and it is a prompt, not a guarantee — which is why the next section exists.

The model. A local reasoning model (DeepSeek-R1 8B, on the GPU server in the house) writes the answer. Warm, the entire round trip is about 8 seconds, and roughly 7.9 of those are the model. Retrieval is free; thinking is not.

Grounding as a refusal contract

Here is the design decision I would defend hardest, and it is not a technical one.

A prompt that says “use only these sources” is a request. Models comply with it most of the time and quietly ignore it some of the time, especially when the sources are thin and the model happens to have a memory that fits. If the system’s only defence is that sentence, then “grounded” is a vibe.

So the grounding is enforced outside the model. When search returns nothing usable, the tool does not hand a bare question to the LLM as a fallback — it fails, and it fails with a non-zero exit code, which is the one signal every calling program is already built to notice. And the AI agents that use it are ruled to report a failed lookup rather than substitute an answer they remember. A refusal propagates; it does not get smoothed over one layer up.

The result is an assistant that will tell you it couldn’t find something. That reads as a weakness on a feature list and it is the opposite. A model that says “I couldn’t find it” is worth more than a bigger model that confabulates, because the first one you can build on and the second one you have to fact-check — and if you have to fact-check every answer, you have not saved any work at all.

Privacy is an architecture, not a policy

Every big assistant has a privacy page. This one has a topology instead: the question goes from the terminal to a search index on the same LAN to a model on the same LAN and back. There is no provider in the path who could log it, no key that identifies me, no terms of service that can change next quarter. My kid can ask it something embarrassing and the transaction is as private as looking it up in a book on a shelf, because functionally that is what it is.

Nobody has to trust me about this, either, which is the good part. It is verifiable by unplugging the internet and watching it keep working.

The unglamorous 90%

The pipeline was a weekend of pleasant work. The actual hard part was three container flags.

The Wikipedia server runs as a rootless container on an immutable Linux desktop with SELinux enforcing, and every one of those adjectives cost me an evening:

• Rootless user mapping. The container has to run as root inside its own namespace, which maps to my unprivileged user outside it. Get this wrong and the server starts, reports itself healthy, and serves nothing — the process simply cannot read the file it was pointed at.
• SELinux relabelling. A volume mounted into a container on an SELinux system needs the mount option that relabels it, or the read is denied at the kernel by a policy that has never heard of your container. The error surfaces as a missing file, which sends you looking in exactly the wrong place.
• Explicit file paths. Auto-discovery of the ZIM directory did not behave the way the docs implied inside the container’s view of the filesystem. Naming the file explicitly took thirty seconds once I stopped assuming the problem was elsewhere.

Three flags. Roughly 90% of the elapsed time. This is the honest shape of most self-hosting work, and it is exactly the part that gets left out of the write-ups — which is how people conclude they’re bad at this when they are merely doing it for the first time, alone, with the same three flags in front of them.

Then it died, and nobody noticed

It worked. It was fast. I tested it, wrote down the numbers, and moved on.

A week later I went to use it and it was gone. The machine had rebooted at some point, the container had not come back, and nothing had told me — because there was no service unit, no monitoring, and no health check. Worse: the skill file that would have told my AI agents this tool existed had been drafted and never installed. So for that entire week, every agent in the house answered from memory when it could have looked things up, and had no idea it was missing an option.

The lesson is short enough to keep:

A capability nothing autostarts and nothing knows about is indistinguishable from a capability you never built.

The fix was a 12-line service unit and one markdown file. The unit makes the server survive a reboot. The markdown file is a shared skill in the agent stack — the mechanism my agents use to discover what they can do — telling them the tool exists, how to call it (a shell command, or plain HTTP if they prefer), and that a grounded answer must carry its citations.

The pipeline was 700 lines and the good story. Thirteen lines of boilerplate were what made it real. I keep relearning this one.

Wired into the agent stack

I run a small fleet of local AI agents for household and business work, and InfoForge is now a shared capability rather than a script I remember. An agent that needs a fact runs the command or hits the endpoint, gets an answer with citations attached, and passes both along. The citations matter more than they look: when an agent tells me something and names the article, I can check it in one click, and checking is what turns a plausible assistant into a usable one.

It also composes with everything else that stays in the house. The same GPU server answers InfoForge queries, drafts email and runs chat; the knowledge layer just became one more thing that never needed an account.

Gotchas

• Disk before download. ~53 GB for the no-images build, and roughly double if you want pictures. Check free space first; a half-written ZIM is a slow way to learn this.
• The dump has a date. Mine is June 2026. Wikipedia does not stop. Anything more recent than your dump does not exist as far as this system is concerned — which is fine if you know it, and a trap if you forget.
• Search quality is your keyword extractor. Nearly every bad answer I have seen traces to a bad query, not a bad model. Fix the extraction before you reach for a bigger model.
• Truncation is a real choice. Too little context and the answer isn’t there; too much and the model drifts off the instruction. Tune it against actual questions you care about, not synthetic ones.
• Test the failure path deliberately. Ask it something Wikipedia genuinely has no article on and confirm you get a refusal and a non-zero exit — not a fluent paragraph. If you only test questions that work, you have tested the easy half.
• Write the service unit on day one. See above. Ask me how I know.

Where this leaves me

The house has an offline encyclopedia with a search index in front of it and an AI that is only permitted to speak from it. It cost a download and a weekend. It is faster than the internet, it works during an outage, it cannot be discontinued, and it does not have a business model that involves me.

Next on the list: more ZIMs in the same rack — the medical and technical collections Kiwix packages, a wiki or two of my own — because the retrieval layer does not care what the corpus is. And a health check, because I have now learned that particular lesson at full price.

Downloads

• Download InfoForge project source (zip) 80 KB

Free for personal use. If it saves you an afternoon, the coffee button's nearby.