DeepSeek: Running Locally — a 4-Step Guide (No Experience Needed)

Install Ollama onto your system. Ollama is the backend that runs LLMs — it doesn't include a Graphical User Interface (GUI, basically the web page you interact with). That comes later.

Install via the following link:

https://ollama.com/download/windows

Once complete, test that it's working:

• Open a command prompt (type "cmd" into the search bar to open Command Prompt)
• In the command prompt, type "ollama" and press enter
  • You should see a list of commands pop up — that means Ollama is working. You can close the command prompt window; Ollama runs in the background:
  • If the response is "'ollama' is not recognized as an internal or external command, operable program or batch file" then your installation has failed. Try again.

C:\Users\you>ollama
Usage:
  ollama [flags]
  ollama [command]

Available Commands:
  serve       Start ollama
  run         Run a model
  pull        Pull a model from a registry
  list        List models
  ps          List running models
  rm          Remove a model
  ...

This step is done when: you can open Ollama from your command prompt

If you don't already have Docker, set it up from the following directions (no modifications needed, run as is):

https://docs.docker.com/desktop/setup/install/windows-install

This step is done when: you can open Docker on your computer:

Open WebUI is just an interface that lets you talk to Ollama. There are multiple interfaces out there; this is the one I use.

Follow the directions from the quick start site:

https://docs.openwebui.com/getting-started/quick-start

• At the time of this writing, all that was needed was to run the following (note: this is because I have an Nvidia GPU):
  • docker pull ghcr.io/open-webui/open-webui:main
  • docker run -d -p 3000:8080 --gpus all -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:cuda
    • This second command only works for people with Nvidia GPUs (CUDA is Nvidia-only)
    • Other directions are available at the link above

Once complete, you should see a new container in your Docker program:

Click on the Port to launch the web user interface.

• Once it comes up, create a login
  • This is running locally, so it's just how you authenticate yourself on your own machine. It does not create an online account.
  • Here's the documentation from Open WebUI on creating an account: Link

This step is done when: you can open the GUI (by clicking on the "Port(s)" in the Docker container) and have gotten into Open WebUI

Note: you should not have a model loaded yet

Parameters are the model's size — 7 billion, 14 billion, 32 billion, and so on. Uncompressed, each billion parameters takes about 1 GB of vRAM, so an 8 GB card can run roughly a 7B model at full quantization. File size scales the same way: a 7B model is around 7 GB, a 40B model around 40 GB. Bigger is also smarter — a 7B model can seem dumb where the 40B version of the same model can seem almost human.

Quantization is compression, measured in bits — typically 16, 8, 6, or 4. Fewer bits means a smaller file with some loss of quality: at 16-bit, 1B parameters is about 1 GB of model; at 4-bit, about 0.6 GB. As a rule of thumb, 16/8/6-bit keeps the model smart at full size; 5- or 4-bit is good enough, medium-sized, and fast; 2- or 3-bit is very small but may not stay coherent. If you want more detail, here's an excellent visual guide to quantization.

On Windows, press Ctrl + Shift + Esc to open Task Manager, click Performance, then click your GPU. Your vRAM is the Dedicated GPU Memory figure — the total is what you have, and your available vRAM is that total minus what's currently in use. So if total = 24 GB and used = 1.2 GB, you have about 22.8 GB available. If you have multiple video cards (like me), add their vRAM together to get your total.