Running a genuinely useful language model on your own desktop used to require a research lab’s budget. That changed fast. By mid-2026, a $500 graphics card and roughly 90 minutes of setup are enough to run a 13-billion-parameter model that answers in real time, never sends a single token to a third-party server, and costs nothing per query once the hardware is paid off.
That shift explains why run llm locally has turned into one of the most searched phrases among developers and hobbyists this year. Cloud API bills for GPT-5.5 or Gemini 3.1 Pro-class usage add up quickly for anyone running batch jobs, coding agents, or high-volume chat features. A local setup flips that math: pay once for hardware, then run inference without limits.
This guide walks through the entire process end to end. You’ll pick hardware that won’t bottleneck on VRAM, install drivers and Ollama, download and quantize a model, wire up a browser-based chat interface with Open WebUI, and benchmark the result in tokens per second. It closes with a complete, reproducible Docker stack you can copy directly into your own project.
Two numbers matter more than anything else before you spend a dollar. First, 16GB of VRAM is the threshold that separates “runs a 7B model adequately” from “runs a 13B model comfortably with room for longer context.” Second, budget cards like the 16GB RTX 4060 Ti (around $500) now sit at the center of this market, largely because Nvidia’s CUDA ecosystem still covers the large majority of local AI tooling. AMD hardware works too, just with noticeably thinner software support.
Developers run models locally for three main reasons. Privacy tops the list, since code, medical notes, or internal documents never leave the machine. Cost comes next: a single RTX 4090 pays for itself within months for anyone routing thousands of daily calls through GPT-5.5 or Gemini 3.1 Pro instead. Latency and offline access round out the list, which matters on flights or in facilities with no reliable connection.
The model side has caught up too. Open-weight families like Llama 4, Qwen 3.5, and Mistral now ship versions tuned specifically for consumer GPUs, and quantization formats like GGUF shrink a model’s memory footprint by 50 to 75 percent with a small, often barely noticeable, quality tradeoff. Cheaper VRAM plus smarter compression is what makes this tutorial realistic on a few hundred dollars of hardware instead of a rented data-center GPU.
None of this was practical even two years ago. Early local-LLM setups meant hand-compiling inference code, patching Python dependencies, and hoping a model fit in memory by trial and error. The tooling has since consolidated around a handful of runners, GGUF became the default distribution format for quantized weights, and GPU vendors started shipping enough VRAM at consumer price points to make 13B-class models a realistic target rather than an enthusiast stunt. The rest of this guide assumes you’re starting from zero and walks through every step in order, from confirming your hardware can do the job to running a benchmark that proves it’s working.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Prerequisites: Hardware and Software Checklist
Gather these before starting. Version numbers below reflect what’s current and stable as of June 2026. Where a tool ships frequent point releases, use whatever the latest stable build is at install time.
- GPU: Nvidia card with 8GB VRAM minimum, 16GB strongly recommended (RTX 4060 Ti 16GB, RTX 4070, RTX 4080, or RTX 4090). AMD RX 7000/9000-series cards with 16GB+ work with ROCm but need extra configuration.
- CPU: 6 cores minimum, 8+ cores recommended for smoother CPU-offloaded layers.
- System RAM: 16GB minimum, 32GB recommended, 64GB+ for 30B-class models.
- Storage: NVMe SSD with at least 100GB free. Model files run 4-40GB each depending on size and quantization.
- Operating system: Windows 11 (23H2 or later), Ubuntu 22.04 LTS or 24.04 LTS, or macOS 14+ on Apple Silicon.
- GPU driver: Latest Nvidia Game Ready or Studio driver supporting CUDA 12.x.
- CUDA Toolkit: 12.x (latest release), Nvidia GPUs only.
- Docker: Docker Engine 24.0 or newer, plus the NVIDIA Container Toolkit if you plan to run models inside containers.
- Python: 3.10 or newer, only needed if you plan to call models programmatically.
- Ollama: latest stable release, the model runner this tutorial builds around.
None of this requires a data center. It requires a decent GPU, about 90 minutes, and the 13 steps below.
Step 1-2: Do the VRAM Math and Choose Your GPU
Every local LLM decision starts with VRAM, not raw compute. A model’s parameter count times its quantization bit-width, divided by 8, gives a rough memory footprint in gigabytes. A 7-billion-parameter model quantized to 4-bit (Q4) needs roughly 4-5GB just to load the weights, plus overhead for context and the KV cache. A 13B model at Q4 needs around 8-9GB. That overhead is why a “13GB model” rarely runs comfortably on a 12GB card. Leave 2-4GB of headroom.
This is also where most first-time builders waste money. A GPU with more CUDA cores but less VRAM will still choke on a model that doesn’t fit in memory, forcing partial CPU offload and tanking your tokens-per-second. VRAM capacity, not shader count, is the number to optimize for. The table below reflects typical 2026 street pricing and what each card handles well.
Here’s the math worked through on a real example. Llama 3.1 8B at Q4_K_M quantization ships as roughly 4.9GB of weights. Add a KV cache for an 8,192-token context window, plus a couple gigabytes of runtime overhead for CUDA itself, and actual usage lands closer to 7-8GB. That’s why an 8GB card runs this model but leaves almost no headroom for a second application, while a 12GB card runs it with room to spare for a longer conversation. Scale that same logic up to a 13B model (roughly 8-9GB of weights at Q4) and the gap between “technically fits” and “fits comfortably” becomes the whole reason 16GB cards dominate this segment of the GPU market.
| GPU | VRAM | Approx. Street Price | Comfortable Model Size | Notes |
|---|---|---|---|---|
| RTX 3060 (used) | 12GB | ~$300 | 7B (Q4/Q5) | Cheapest usable entry point |
| RTX 4060 | 8GB | ~$300 | 7B (Q4) | VRAM-constrained above 7B |
| RTX 5060 | 8GB | ~$370 | 7B (Q4) | Best budget 1080p/AI hybrid |
| RTX 4060 Ti 16GB | 16GB | ~$500 | 13B (Q4/Q5) | Current budget sweet spot |
| RTX 4070 | 12GB | ~$600 | 13B (Q4) | Balanced mid-range pick |
| RTX 4070 Ti | 12GB | ~$800 | 13B (Q4) | Faster than 4070, same VRAM cap |
| RTX 4080 | 16GB | ~$1,200 | 30B (Q4) | Strong all-rounder for larger models |
| RTX 4090 | 24GB | ~$1,600 | 70B (Q4, tight) | Top single-GPU performer |
| RTX 5070 Ti / RX 9070 XT | 16GB | Varies by region | 13B-30B (Q4) | 2026 gaming/AI hybrid pick |
Professional or multi-model builds tend to specify 16+ core CPUs and 64-128GB of system RAM to avoid bottlenecks entirely, but that’s overkill for a single 7B-13B model running on a dedicated GPU. Nvidia’s RTX line remains the safer buy because its CUDA ecosystem covers the vast majority of AI tooling out of the box. AMD cards run models fine through ROCm or Vulkan backends, but expect more manual configuration and slower day-one support for new model architectures. If you’re pricing this out alongside a full build, check current GPU prices before committing, since the market has moved fast this year.
Step 3: Check Your CPU, RAM, and Storage
VRAM gets the spotlight, but the rest of the system still needs to keep up. Ollama and llama.cpp-based runners load the full model into system RAM first before pushing layers to the GPU, so a machine with 16GB of RAM and an 8GB model will thrash. Budget at least 2x the model file size in system RAM as a floor, and double that if you plan to run a browser UI, a database, or a second model alongside it.
CPU choice matters most when layers spill over from GPU to CPU, which happens whenever a model doesn’t fully fit in VRAM. A modern 6-8 core chip handles this without much drama for 7B-13B models. AMD’s AM5 platform running Ryzen 9000-series (Zen 5) chips is the current default for 2026 AI-oriented builds, while Intel’s LGA1851 Ultra-series parts get picked when the NPU or QuickSync matters for other workloads on the same machine. If you’re assembling a new system from scratch rather than repurposing an existing PC, the general steps in our guide to building a gaming PC apply directly, with the GPU choice swapped for whichever card from the table above fits your VRAM target.
Storage is the easiest piece to get wrong by underestimating it. A single 8B model at Q4 quantization runs 4-5GB, but most people end up testing three or four models before settling on one, plus keeping Docker images, embeddings, and chat logs. Budget 150-200GB free on an NVMe drive if you plan to experiment, not the bare 100GB minimum. NVMe also loads models measurably faster than SATA SSDs, which matters every time you swap models.
Power and cooling deserve a mention too, since they’re easy to overlook on a repurposed machine. A GPU running sustained inference draws closer to its rated power limit than it would during typical desktop use, so an old 500W power supply paired with a 16GB card is worth double-checking against that card’s actual power draw before assuming it’ll hold up under a long benchmarking session. Case airflow matters less for short chat sessions but becomes relevant once you start running longer batch jobs or leaving a model loaded for hours as a background service.
Step 4-5: Install GPU Drivers and the CUDA Toolkit
Start by confirming your current driver and GPU are visible to the system. Open a terminal and run:
nvidia-smiIf that command isn’t found, install or update your Nvidia driver first, then reboot. On Windows, grab the latest Game Ready or Studio driver from Nvidia directly. On Ubuntu, the quickest path is the distro’s own driver manager:
sudo ubuntu-drivers autoinstall
sudo rebootOnce nvidia-smi returns a driver version and GPU name, install the CUDA Toolkit (12.x, latest release) if you plan to compile anything from source or use Python libraries like PyTorch directly. If you’re only running Ollama, you can actually skip a manual CUDA install entirely, since Ollama ships its own bundled CUDA runtime. Installing the full toolkit is still worth doing if you plan to benchmark with other frameworks later.
Step 6-7: Install Ollama and Pull Your First Model
Several tools can run a local model. Ollama wins for this tutorial because it handles quantization, model management, and GPU offload automatically through a single CLI and REST API, with no manual configuration files required. Here’s how it stacks up against the other common options.
| Tool | Interface | GPU Support | Best For | Docker Support |
|---|---|---|---|---|
| Ollama | CLI + REST API | Nvidia CUDA, AMD ROCm (partial), Apple Metal | Fast setup, scripting, automation | Yes, official image |
| LM Studio | Desktop GUI | Nvidia CUDA, Apple Metal, Vulkan | Non-technical users, model browsing | No official image |
| llama.cpp | CLI, embeddable library | Nvidia, AMD, CPU-only, Apple Metal | Maximum control, embedding in apps | Community images |
| text-generation-webui | Browser GUI | Nvidia CUDA, AMD (partial) | Power users, plugins and extensions | Yes, community-maintained |
| Open WebUI | Browser GUI (ChatGPT-style) | Depends on backend (Ollama, vLLM) | Best chat interface layered on Ollama | Yes, official image |
Install Ollama on Linux or macOS with the official install script:
curl -fsSL https://ollama.com/install.sh | shOn Windows, download the installer directly from ollama.com and run it like any other application. Once installed, pull your first model. Llama 3.1 8B is the standard starting point since it balances quality and size well for an 8-16GB card:
ollama pull llama3.1:8bThat single command downloads the model and its Q4_K_M-quantized weights, checksums them, and registers the model locally. Expect a 4-5GB download on a typical connection.
Llama 3.1 8B is the safest default for a first install, but it’s not the only reasonable choice, and which model you settle on should depend on what you’re actually using it for. Anyone doing general chat and writing tasks gets similar results from Mistral 7B, which trades a little bit of Llama’s polish for a noticeably smaller memory footprint at the same quantization. Developers leaning on a model for code generation and debugging tend to get better results from a coding-tuned variant, since a general chat model will happily generate plausible-looking but subtly broken code. Anyone with 24GB of VRAM to spare and a preference for reasoning-heavy tasks should also look at how GLM-5.2 and DeepSeek V4 compare on coding benchmarks before settling on a default, since both ship distilled versions specifically sized for local use. Pulling a second or third model to compare costs nothing but disk space and download time, and Ollama lets multiple models sit installed side by side without conflict.
Step 8-9: Run the Model and Confirm GPU Offloading
Start a chat session directly from the terminal:
ollama run llama3.1:8bType a prompt and confirm you get a response. The real check, though, is whether the GPU is actually doing the work rather than the CPU quietly carrying the load. While the model is generating a response, open a second terminal and run:
nvidia-smiYou’re looking for GPU utilization climbing well above idle and memory usage matching roughly what the model needs. Here’s an example of what that output looks like mid-generation:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M.|
|================================+======================+======================|
| 0 NVIDIA GeForce RTX 4070 | 00000000:01:00.0 On | N/A |
| 30% 62C P2 145W / 200W | 9821MiB / 12282MiB | 97% Default |
+-----------------------------------------------------------------------------+97 percent GPU utilization with memory usage close to the model’s footprint means offload is working correctly. If utilization stays near 0 percent while your CPU fan spins up instead, jump ahead to the troubleshooting section below. For anyone scripting against the model instead of chatting manually, Ollama exposes a REST API on port 11434 by default:
import requests
response = requests.post(
"http://localhost:11434/api/generate",
json={
"model": "llama3.1:8b",
"prompt": "Explain quantization in one paragraph.",
"stream": False
}
)
print(response.json()["response"])That’s the same API Open WebUI, LangChain, and most local-first agent frameworks talk to under the hood, which is why standardizing on Ollama early pays off later.
Step 10: Install Open WebUI for a ChatGPT-Style Interface
The terminal is fine for testing, but a browser-based chat interface makes a local setup usable day to day, especially for anyone other than the person who installed it. Open WebUI is the most widely deployed front end for Ollama, and it runs cleanly in a single Docker container alongside Ollama itself.
docker run -d --name open-webui \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:mainOnce the container finishes pulling and starting, open http://localhost:3000 in a browser, create a local admin account on first launch, and the model you pulled earlier should already appear in the model dropdown. This step is optional if you’re only calling models through the API or CLI, but it’s the difference between a personal project and something you’d hand to a coworker.
Step 11: Pick the Right Quantization Level
Quantization is what makes any of this fit on consumer hardware at all. Full-precision (FP16) weights for a 7B model run around 14GB, more than most gaming GPUs have available. GGUF quantization compresses those weights down to 4, 5, or 8 bits per parameter, trading a small amount of precision for a large reduction in memory footprint. Q4_K_M is the default most people should start with, since it keeps the bulk of the model’s quality while cutting size by roughly 70 percent compared to FP16. Q5 and Q6 variants exist for anyone with VRAM to spare who wants output closer to the full-precision original.
| Model Size | Recommended Quant | Min VRAM | Min System RAM | Example Models |
|---|---|---|---|---|
| 3B | Q4_K_M | 4GB (or CPU-only) | 8GB | Llama 3.2 3B-class, small Phi models |
| 7B-8B | Q4_K_M | 6-8GB | 16GB | Llama 3.1 8B, Mistral 7B |
| 13B | Q4_K_M | 10-12GB | 16-32GB | Older Llama 2 13B-class models |
| 30B-34B | Q4_K_M | 20-24GB | 32-64GB | Qwen 32B-class, Yi 34B-class |
| 70B | Q4_K_M | 40-48GB or dual GPU | 64GB+ | Llama 3.1 70B-class models |
| 70B+ (dual GPU) | Q4_K_M / Q5_K_M | 48GB+ combined | 128GB | Large dense or MoE 70B+ models |
To pull a specific quantization instead of the default, specify the tag directly, for example ollama pull llama3.1:8b-instruct-q5_K_M. If you’re unsure which to pick, start at Q4_K_M and only move up once you’ve confirmed the smaller quant actually leaves your GPU underused.
Step 12-13: Benchmark Tokens Per Second and Tune GPU Layers
Tokens per second is the single number that tells you whether a setup is actually usable. Ollama’s verbose mode prints exact timing data for every generation, which is the fastest way to compare quantization levels or GPUs on your own hardware:
ollama run llama3.1:8b --verboseAfter a response finishes, you’ll see a stats block similar to this (illustrative example, your own numbers will vary by GPU, quantization, and prompt length):
total duration: 8.42s
load duration: 612ms
prompt eval count: 24 tokens
prompt eval duration: 340ms
eval count: 186 tokens
eval duration: 7.44s
eval rate: 25.0 tokens/sAs a rough guide, 15-30 tokens/s on a Q4-quantized 7B-8B model feels close to real-time reading speed for most people. Below 5 tokens/s, a chat interface starts feeling sluggish and layers are likely spilling onto the CPU. If that happens, you can force more layers onto the GPU explicitly (or fewer, if you’re running out of VRAM) using a custom Modelfile:
FROM llama3.1:8b
PARAMETER num_gpu 33
PARAMETER num_ctx 8192
PARAMETER temperature 0.7
SYSTEM """
You are a concise technical assistant. Keep answers under 200 words unless asked for more detail.
"""Build and run it with ollama create my-model -f Modelfile followed by ollama run my-model. The num_gpu parameter controls how many of the model’s layers get offloaded to the GPU versus the CPU, and num_ctx sets the context window, which directly trades off against VRAM headroom. To watch GPU load continuously while you tune these values, leave a monitor running in a spare terminal:
watch -n 1 nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu --format=csvComplete Working Project: A Self-Contained Local AI Stack
Everything above works individually, but the most useful end state is a single stack that starts with one command and survives a reboot. The following Docker Compose file combines Ollama and Open WebUI, with the GPU passed through correctly and both services set to restart automatically. This is the same pattern worth reaching for once you’ve confirmed the manual steps work and want something durable enough to leave running in the background, rather than a terminal session you have to babysit.
Treat this as a starting template rather than a finished product. Once it’s running, the natural next additions are a second pulled model for comparison, a reverse proxy with authentication if you expose it beyond your own machine, and a scheduled task that periodically re-pulls the latest image tags so both containers stay current without manual intervention.
The docker-compose.yml Stack
version: "3.8"
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- webui_data:/app/backend/data
depends_on:
- ollama
volumes:
ollama_data:
webui_data:Bringing the Stack Up and Testing It
Save that file as docker-compose.yml, make sure the NVIDIA Container Toolkit is installed so Docker can see the GPU, then start everything with a single command:
docker compose up -d
docker exec -it ollama ollama pull llama3.1:8bOpen http://localhost:3000, create an account, and start chatting. Both containers restart automatically on reboot thanks to the unless-stopped policy, so this stack survives power cycles without any manual intervention. From here, swapping in a different model is just a matter of pulling it and selecting it from the Open WebUI dropdown, no configuration changes required.
Common Pitfalls to Avoid
- Buying for CUDA cores instead of VRAM. A faster GPU with less memory will still choke on a model that doesn’t fit, forcing slow CPU offload regardless of how fast the chip is otherwise.
- Ignoring the quantization tag when pulling a model. Pulling a full-precision or Q8 variant by accident on an 8GB card is the single most common cause of “out of memory” errors on a first attempt.
- Skipping the driver update before installing CUDA. An outdated Nvidia driver paired with a newer CUDA Toolkit causes version mismatches that are confusing to diagnose after the fact.
- Leaving other GPU-heavy applications open while benchmarking. A browser with hardware acceleration, a game, or a second AI tool running in the background will quietly eat VRAM and skew your tokens-per-second numbers.
- Running Ollama and LM Studio at the same time. Both try to bind port 11434 or claim the GPU exclusively, leading to confusing failures that look like a broken install.
- Assuming AMD hardware works identically to Nvidia. ROCm support has improved, but expect slower day-one compatibility with new model releases and more manual setup overall.
- Underestimating storage needs. Most people test three or four models before settling on one, and Docker images plus model files add up faster than the bare minimum suggests.
Troubleshooting Guide
- “CUDA out of memory” error: Drop to a smaller quantization (Q4 instead of Q5/Q8), reduce
num_ctx, or close other GPU-using applications before retrying. - Ollama not detecting the GPU, falling back to CPU: Confirm
nvidia-smiworks outside Docker first. If you’re in a container, verify the NVIDIA Container Toolkit is installed and the compose file’sdeploy.resourcesblock is present. - Extremely slow generation, under 5 tokens/s: Check
nvidia-smiduring generation. Near-zero GPU utilization means the model is running on CPU, usually because it doesn’t fit in available VRAM. - Open WebUI can’t connect to the Ollama backend: Double-check
OLLAMA_BASE_URL. Inside Docker Compose it should point to the service name (http://ollama:11434), notlocalhost. - Model download stalls or fails partway: Re-run the same
ollama pullcommand. Ollama resumes partial downloads rather than restarting from zero. - “nvidia-smi: command not found”: The Nvidia driver isn’t installed or didn’t load correctly. Reinstall the driver and reboot before doing anything else.
- Docker container can’t see the GPU at all: Install the NVIDIA Container Toolkit and restart the Docker daemon. On Linux, this is a separate package from the GPU driver itself.
- Model output is garbled or repeats itself endlessly: Lower the
temperatureparameter, confirm you pulled an “instruct” or “chat” tagged variant rather than a base model, and check that the quantization file didn’t get corrupted during download. - “Port 11434 already in use”: Another Ollama instance or a conflicting service is already bound to that port. Stop it with your OS’s process manager before starting a new one.
- GPU passthrough not working inside WSL2 on Windows: Update to the latest WSL kernel, confirm you installed the Windows-side Nvidia driver (not a Linux driver inside WSL), and verify with
nvidia-smifrom inside the WSL shell itself.
Keeping Your Local AI Server Secure
Running a model locally removes the privacy risk of sending prompts to a third-party API, but it introduces a smaller, easier-to-manage risk of its own: Ollama’s API listens on port 11434, and by default that’s reachable from anywhere on your local network, not just from the machine it’s running on. That’s convenient if you want to call it from a laptop on the same Wi-Fi network, but it also means a misconfigured router or an overly permissive firewall rule can expose an inference endpoint to the open internet without any authentication in front of it.
A few habits close most of that gap. Bind Ollama to 127.0.0.1 instead of 0.0.0.0 if you only ever call it from the same machine, which stops it from answering requests from other devices entirely. If you do need network access, for example from Open WebUI running on a different machine, put a reverse proxy in front of it with basic authentication rather than exposing port 11434 directly, and confirm your router isn’t forwarding that port to the public internet. On a home server, a simple firewall rule limiting inbound connections on 11434 and 3000 to your own local subnet closes the door on the most common misconfiguration. None of this is complicated, but skipping it is the difference between a private local model and an open one that any device on your network, or beyond it, can query for free.
Container isolation helps too. Running Ollama and Open WebUI inside Docker, as the working project below does, keeps both processes sandboxed from the rest of the host system by default, which limits the blast radius if either service is ever compromised through a future vulnerability. Keep both images updated by re-pulling the latest tag periodically, since model runners are actively developed software and older builds miss both performance improvements and security patches.
Advanced Tips for Power Users
Once the basic stack runs reliably, a few upgrades make a meaningful difference. Multi-GPU setups split larger models across two cards using tensor parallelism, which is how a 70B model becomes practical on two 24GB cards instead of one. Speculative decoding pairs a small, fast draft model with a larger target model, letting the large model verify several tokens at once instead of generating one at a time, often boosting throughput by 30-50 percent without any quality loss.
Custom Modelfiles are worth building out beyond the basic example above. Baking a system prompt, temperature, and context length into a named model means every application calling it gets consistent behavior without repeating configuration in client code. For anyone running multiple models simultaneously, a lightweight router in front of Ollama’s API can direct short queries to a fast 3B model and complex ones to a 13B or 30B model, cutting average latency without sacrificing quality on the queries that need it.
Finally, KV cache quantization (available as an experimental flag in recent Ollama and llama.cpp builds) compresses the memory used by conversation history itself, which matters most once context windows stretch past 8,000-16,000 tokens. It’s a smaller win than model quantization, but it stacks on top of everything else covered here and can be the difference between a context window that fits and one that doesn’t on a tightly-sized GPU.
If this machine is going to run as a permanent home server rather than a project you start manually, set it up to survive reboots without you at the keyboard. On Linux, that means enabling the Docker daemon itself to start on boot (sudo systemctl enable docker) so the unless-stopped restart policy in the compose file below actually kicks in after a power cycle. On Windows, Docker Desktop’s own “start on login” setting covers the same need. Either way, test it by rebooting once during setup rather than discovering the gap the first time your power goes out.
Frequently Asked Questions
Do I need a GPU to run an LLM locally?
No, but it changes the experience substantially. Models in the 3B-7B range run on a modern CPU alone, just 2-5x slower than the same model on a GPU. Anything above 7B on CPU-only hardware tends to feel too slow for interactive chat, though it’s still fast enough for batch jobs where you’re not waiting on the response in real time.
What’s the minimum VRAM to run a 7B model?
6-8GB at Q4 quantization, with 8GB giving more comfortable headroom for longer context windows. Drop below 6GB and you’ll need a smaller model or a more aggressive quantization like Q3 to avoid CPU offload.
Can I run a 70B model on a single consumer GPU?
It’s tight but possible on a 24GB card like the RTX 4090 at aggressive Q4 quantization with a limited context window. Dual-GPU setups handle 70B models far more comfortably and leave room for a longer context window without swapping to disk.
Is AMD or Nvidia better for local LLMs?
Nvidia remains the safer default because of CUDA’s broad tooling support. AMD GPUs work through ROCm, but with less day-one compatibility for new model releases and generally more manual setup, so budget extra troubleshooting time if you go that route.
How much does it cost to build a local AI PC in 2026?
A capable entry-level build centers on a 16GB RTX 4060 Ti at around $500 for the GPU alone. Paired with a mid-range CPU, 32GB of RAM, and an NVMe drive, a full budget-focused system lands well under $1,500 total, and less if you’re repurposing an existing PC and only buying the GPU.
What’s the difference between Ollama and LM Studio?
Ollama is CLI and API-first, which makes it better for automation, scripting, and Docker deployments. LM Studio is a desktop GUI aimed at users who prefer browsing and downloading models visually without touching a terminal. Many people end up using both, Ollama for anything scripted and LM Studio for quickly trying a new model.
Does quantization hurt model quality?
Slightly, but Q4_K_M and Q5_K_M quantization are tuned to minimize the impact. Most users can’t reliably tell the difference between Q5 and full precision in everyday use, while Q4 shows minor degradation mainly on complex reasoning and multi-step math tasks rather than everyday conversation.
Can I run local LLMs on a laptop?
Yes, provided it has a discrete GPU with 8GB+ VRAM or an Apple Silicon chip with enough unified memory. Thin-and-light laptops without a discrete GPU can still run 3B-7B models on CPU, just slower, and battery life takes a noticeable hit during sustained generation.
Thirteen steps sounds like a lot written out, but most of it is one-time setup. Once the GPU is installed, the driver is current, and Ollama and Open WebUI are running as a Docker stack, adding a new model or trying a different quantization level takes one command, not a fresh walkthrough. The 16GB VRAM threshold, the $500 price point on cards like the RTX 4060 Ti, and the 15-30 tokens/s benchmark range covered above are the three numbers worth remembering if you take nothing else from this guide. Everything else is configuration you set once and rarely touch again.
Related Coverage
- How to Build a Gaming PC in 12 Steps, 90 Min [2026]
- Gaming GPU Prices Surge: RTX 5090 Nears $3,000 [2026]
- Nvidia Skips New Gaming GPUs, Breaks 30-Year Streak [2026]
- Llama 4 vs Qwen 3.5 vs Mistral: Best Open LLM [2026]
- GLM-5.2 vs DeepSeek V4 vs Kimi K2.6: 62% SWE Pro [2026]
- How to Enable XMP/EXPO RAM: 12 Steps, 40 Min [2026]
For more hardware tutorials and GPU benchmarks, visit our AI chips and hardware hub.
Sources and further reading: Ollama, llama.cpp on GitHub, Hugging Face’s GGUF documentation, LM Studio, Puget Systems’ hardware primer for local LLMs, and Nvidia’s GeForce RTX 40-series product pages.
