InfoForge: Offline Wikipedia as a RAG Source for My Local AI Agents

I asked a local 8B model when the novelist Natsume Sōseki was born. It answered “9 February 1867,” and then it did the part I actually cared about: it named the articles it read that in. The model does not know that fact. It looked it up, in a 52.69 GB copy of Wikipedia sitting on a machine in my house, with the internet irrelevant to the transaction.

Why bother, when the model already “knows things”

Because it doesn’t, and the failure mode is the ugly kind. Ask a small model for a date, a chemical property or a place name and it will give you a fluent, confident, plausible answer with no sign of whether it is true. It isn’t lying. It has no way to tell recall from generation. That is fine for prose and a disaster for facts.

Retrieval fixes it, but the usual retrieval story assumes a cloud search API: your question goes to a company, that company logs it, and your assistant now depends on somebody else’s uptime and terms of service. In a house where the AI stack answers questions from my kids, both halves of that bother me.

Wikipedia solves both problems at once. It is the densest general-knowledge corpus there is, you can legally download all of it, and it fits on a disk that costs less than a night out. The Kiwix project did the hard part years ago: it packages whole wikis into the compressed, indexed ZIM format for schools, ships and places with no connection. I am just pointing an AI at it instead of a browser.

What you end up with

A question in, an answer out, and a numbered citations list naming the Wikipedia articles the answer was built from. This is a real run from 2026-09-16 (the server address is loopback):

$ infoforge ask "When was Natsume Soseki born?"
infoforge: warning: skipping disambiguation page: Natsume
infoforge: warning: skipping disambiguation page: Soseki
infoforge: grounding on 5 article(s)
According to the provided sources, Natsume Soseki was born on 9 February 1867. ...

Citations:
[1] Natsume Soseki — http://127.0.0.1:8080/content/wikipedia_en_all_nopic/Natsume_Soseki
[2] Natsume Sōseki — http://127.0.0.1:8080/content/wikipedia_en_all_nopic/Natsume_S%C5%8Dseki
[3] Beongcheon Yu — ...

That took 4.4 seconds. And this is what happens when there is nothing to ground on:

$ infoforge search "qzxvbnm plorkwidget fnargle"
infoforge: error: no sources found for 'qzxvbnm plorkwidget fnargle'
$ echo $?
3

No paragraph, no guess, and an exit code any calling script can check. ask gives the same exit 3, plus a hint to try plainer keywords or run infoforge doctor. The full exit table is short: 0 ok, 2 bad config or usage, 3 nothing to ground on, 4 no sources cited (only with --strict), 5 every model endpoint unreachable. In the exit-5 case the tool still prints the articles it found, so the lookup isn’t wasted.

How it works

Five stages, none of them clever, and I’d recommend keeping it that way.

Keyword extraction. The question is not the query. Kiwix ANDs every search term, so “Who wrote Kokoro and when was it published?” matches almost nothing, while Kokoro is a good query. The core of the extractor is a stopword filter that keeps the original word order:

def keyword_query(question: str) -> str:
    tokens = _TOKEN_RE.findall(question)
    keywords = [
        t for t in tokens
        if t.lower() not in _STOPWORDS and t.lower() not in _WEAK_VERBS
    ]
    return " ".join(keywords) if keywords else question.strip()

v2 adds two things on top: capitalised phrases from the question are tried as article titles first, and the combined hits are re-ranked so that disambiguation pages, bare years and “List of…” pages sink.

Search. Kiwix’s full-text search is a real search index, not a grep, and it’s the reason this works at all. When I built this, it returned ranked titles from 19.19 million entries in 3 to 30 milliseconds. On 2026-09-16, on a busier day on the same machine, the same kind of query took 90–730 ms. My guess is that the index had dropped out of the page cache, but either way you won’t notice it next to the model’s time. There are no embeddings anywhere in this system, no vector database, and no chunk store to rebuild when the dump updates. Wikipedia’s editors already did the chunking. They call the chunks articles.

Fetch and trim. kiwix-serve returns an article over loopback in about 1 ms warm. I measured 0.7–1.1 ms with curl on three articles I had just read, and 6–9 ms the first time. My first draft said “0.5 ms” without saying that it was a warm number. Trimming is the only step that takes judgement. Cutting from the top lost the answer too often, because the sentence you need is often deep in a long article. So each article keeps a short lead (a quarter of its budget) plus a window around the passage that best matches the question’s rarest words.

The grounded prompt. This is the whole trick, and it is four sentences long. The rules go first, then the numbered sources, then the question:

You are a question-answering assistant grounded ONLY in the source texts provided below.
Answer the user's question using only information found in these sources. If the sources do not contain the answer, say so clearly.
When you use information from a source, cite the article title (for example: "according to <title>").
Do not invent facts, titles, or URLs that are not present in the sources.

Source 1: <title>
<trimmed article text>
...
Question: <the question>

Temperature is 0.3 rather than 0. The code comment says 0.0 made some local builds loop on repetitive source text. It is a prompt, not a guarantee, which is why the next section exists.

The model. A local reasoning model (DeepSeek-R1 8B, on the GPU server in the house) writes the answer. The Q4_K_M file is 5.0 GB, and its KV cache adds about 0.15 GB per 1,000 tokens of context at f16, so the weights plus a 32k-token window come to roughly 10 GB before the compute buffer. The reply is stripped of its <think> blocks before printing. My original warm measurement was about 8 seconds for the whole round trip. Three runs on 2026-09-16 took 4.4, 5.2 and 9.7 seconds. Retrieval is free; thinking is not. There is no local fallback any more: I removed the mini PC’s model server on 2026-09-16. If the GPU server doesn’t answer, the tool exits 5 and still prints the articles it found.

Grounding as a refusal contract

This is the design decision I would defend hardest, and it isn’t a technical one.

A prompt that says “use only these sources” is a request. Models follow it most of the time and quietly ignore it some of the time, especially when the sources are thin and the model happens to remember something that fits. If that sentence is the system’s only defence, “grounded” is just a feeling.

So part of the grounding is enforced outside the model, in three layers:

1. No sources, no model call. If search returns nothing, the tool never sends a bare question to the LLM as a fallback. It exits with code 3, the one kind of signal every calling program already checks.
2. Sources that don’t answer. The prompt tells the model to say so, and in that case the run still counts as a success. An honest “not in these sources” is the product working.
3. An answer that cites nothing. The tool prints a warning that the answer should be treated as ungrounded. With --strict, it exits with code 4 instead.

On top of that, the agents that call it are told to report a failed lookup rather than substitute an answer they remember. A refusal gets passed up the chain. Nobody one layer up gets to paper over it.

The result is an assistant that will tell you it couldn’t find something. On a feature list that looks like a weakness, and it’s the opposite. A model that says “I couldn’t find it” is worth more than a bigger model that makes things up, because you can build on the first one and you have to fact-check the second. If you have to fact-check every answer, you haven’t saved any work.

Privacy you can check

Every big assistant has a privacy page. This one has a network path instead: the question goes from the terminal to a search server on the same machine, then to a model on my home network, and back. Nobody in that path could log it, no key identifies me, and there are no terms of service to change next quarter. A question asked here is as private as looking something up in a book on a shelf, because that is effectively what happens.

Nobody has to take my word for it, either. You can check it by unplugging the internet and watching it keep working.

The unglamorous 90%

The pipeline was a weekend of pleasant work. The hard part was three container flags.

The Wikipedia server runs as a rootless Podman container on an immutable Linux desktop with SELinux enforcing, and each of those three conditions cost me an evening. This is the command that works:

podman run -d --name infoforge-kiwix \
  --user root \
  -p 127.0.0.1:8080:8080 \
  -v "$HOME/zims:/data:z" \
  ghcr.io/kiwix/kiwix-serve \
  -k /data/wikipedia_en_all_nopic.zim

• --user root. Under rootless Podman, root inside the container maps to your own unprivileged user outside it. The image’s default user lands in the sub-UID range, can’t get through your home directory, and gets Permission denied on /data.
• :z on the volume. On an SELinux system the bind mount has to be relabelled, or the kernel denies the read even for container root. The error looks like a missing file, which sends you looking in exactly the wrong place.
• Explicit ZIM paths (plus -k). Without a file argument, the image’s start script launched kiwix-serve with no ZIM at all and died with “Arguments did not match expected patterns”. -k (--skipInvalid) lets the server start even when one file is still downloading.

On a Docker host without SELinux or rootless user mapping you probably need none of this. Binding to 127.0.0.1 is a v2 change. The first version listened on the whole LAN, which a knowledge server with no authentication has no reason to do.

Three flags took roughly 90% of the time. That’s the honest shape of most self-hosting work, and it’s exactly the part the write-ups leave out. That’s how people conclude they’re bad at this, when really they’re just doing it for the first time, alone, with the same three flags in front of them.

Then it died, and nobody noticed

It worked. It was fast. I tested it, wrote down the numbers, and moved on.

A week later I went to use it and it was gone. The machine had rebooted at some point, the container hadn’t come back, and nothing told me, because there was no service unit, no monitoring and no health check. Worse, the skill file that would have told my AI agents the tool existed had been drafted and never installed. For that whole week, every agent in the house answered from memory when it could have looked things up, and none of them knew there was another option.

The lesson is short enough to keep:

A capability nothing autostarts and nothing knows about is indistinguishable from a capability you never built.

The fix was a 12-line user service unit (comments trimmed here) and a one-page skill file:

[Unit]
Description=InfoForge kiwix-serve (offline knowledge base for grounded answers)
After=default.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=%h/.local/bin/infoforge serve start
ExecStop=%h/.local/bin/infoforge serve stop
ExecReload=%h/.local/bin/infoforge serve recreate
TimeoutStartSec=180

[Install]
WantedBy=default.target

oneshot with RemainAfterExit is deliberate. serve start returns as soon as the container is up, but the service should read as “active” for as long as the container is meant to run, so systemctl --user stop has something to stop. Enable it with systemctl --user enable --now infoforge.service. Since serve start is idempotent, a reload simply recreates the container when the set of ZIM files changes.

The skill file is a shared note that my agents read to find out what they can do. It says three things: the tool exists, here is the command, and a grounded answer must carry its citations.

The health check I wished for is now infoforge doctor. It walks the chain in order (config, ZIM folder, container, HTTP, a test search, each model endpoint) and prints PASS or FAIL for each link (output trimmed):

$ infoforge doctor
  [PASS] config          parsed <config path>
  [PASS] zims            1 enabled: wikipedia_en_all_nopic.zim
  [PASS] container       infoforge-kiwix: running, serving wikipedia_en_all_nopic.zim
  [PASS] kiwix http      http://127.0.0.1:8080 responds
  [PASS] search          3 hit(s) for 'water' in wikipedia_en_all_nopic
  [PASS] llm primary     <gpu server>/v1 reachable, model ...DeepSeek-R1-0528-Qwen3-8B... (model listed)

doctor: all checks passed

The first version was one Python script of about 700 non-blank lines, and it was the good story. The service unit and the skill file were what made it real, and I keep relearning that.

Wired into the agent stack

InfoForge is now a shared tool for the local agents I run, not a script I have to remember. An agent that needs a fact runs infoforge ask (or --json for a machine-readable reply), gets an answer with citations, and passes both along. The citations matter more than they look: when an agent names the article, I can check it in one click, and being able to check is what makes a plausible assistant usable.

Gotchas

• Disk before download. The no-images build is ~53 GB, and the version with pictures is roughly double that. Check free space first; a half-written ZIM is a slow way to learn this.
• The dump has a date. Mine is June 2026, and Wikipedia doesn’t stop. As far as this system is concerned, anything newer than your dump doesn’t exist. That’s fine if you remember it and a trap if you forget.
• Search quality is your keyword extractor, and your title lookup. The first version answered “Who wrote Kokoro and when was it published?” with “Natsume Sōseki, 1914” and cited the article. When I re-ran that question on v2 (2026-09-16), the title slot went to an album spelled KoKoro, and the novel’s article never made the top five. The model then did the right thing: it listed the other Kokoros it had been given and asked which one I meant. Asked “Who wrote the novel Kokoro?”, it said the sources didn’t contain the answer. No hallucination, but no answer either. Nearly every bad answer I’ve seen comes from a bad query, not a bad model, so fix retrieval before you reach for a bigger model.
• Trimming is a real choice. Too little context and the answer isn’t there; too much and the model drifts away from the instruction. Tune it on questions you actually care about, not synthetic ones.
• Test the failure path on purpose. Ask for something Wikipedia has no article on and confirm you get exit code 3, not a fluent paragraph. If you only test questions that work, you’ve tested the easy half.
• Write the service unit on day one. See above.

Where this leaves me

The house has an offline encyclopedia with a search index in front of it and an AI that is only allowed to answer from it. It cost a download and a weekend. It works during an outage, it can’t be discontinued, and there’s no business model behind it that involves me.

Next on the list: more ZIMs in the same setup (the medical and technical collections Kiwix packages, and a wiki or two of my own), because the retrieval layer doesn’t care what the corpus is. v2 already has a zimify command for turning a folder of documents into a ZIM. The other item is the Kokoro miss above, which is a title-matching bug and not a model problem.

Related: my local AI agent stack, StudioForge (the GPU server the model runs on), the mini PC that hosts InfoForge, and what the agents actually cost (InfoForge itself is free).

Downloads

• Download InfoForge project source (zip) 80 KB

Free for personal use. If it saves you an afternoon, the coffee button's nearby.