How to Set Up vLLM: 12 Steps, 90 Min [2026]

Every new open-weight model release comes with the same follow-up question: how do you actually run this thing at scale? Ollama handles a single chat session on a laptop just fine, but production traffic is a different problem entirely. You need to batch dozens of concurrent requests, split a 70-billion-parameter model across several GPUs, and expose an API your application can hit without falling over. That gap is what vLLM fills. Built originally at UC Berkeley and now maintained as an open-source project with contributors across the industry, vLLM has become one of the default choices for teams who want to self-host large language models instead of routing every request through a third-party API. This tutorial walks through installing vLLM, launching an OpenAI-compatible server, tuning it for your hardware, and deploying it with Docker and Kubernetes, in 12 steps you can finish in about 90 minutes.

·

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

What Is vLLM, and Why Are Teams Self-Hosting With It?

The vLLM project on GitHub describes itself plainly:

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.

vLLM Project, GitHub

That’s a modest way to describe what has become one of the most widely adopted serving layers for open-weight models. Instead of loading a model into a Python script and generating one prompt at a time, vLLM runs as a server. It accepts many requests concurrently and schedules GPU work so the hardware rarely sits idle between them.

The core trick is called PagedAttention. Standard transformer inference reserves a large, contiguous block of GPU memory for each request’s key-value cache, which is the running memory of everything the model has read so far in that conversation. Most of that reserved memory goes unused because requests rarely hit their maximum length. PagedAttention borrows an idea from operating-system virtual memory: it splits the KV cache into small, non-contiguous blocks and pages them in as needed. The original vLLM research reported throughput gains of up to 24 times over Hugging Face Transformers on the workloads it tested (and up to 3.5 times over Hugging Face Text Generation Inference), with the exact number depending on model size and request pattern.

vLLM traces back to a research project at UC Berkeley’s Sky Computing Lab, where PagedAttention was first described in a paper presented at the ACM Symposium on Operating Systems Principles in 2023. The project has since grown well beyond a single research lab, with an open governance model and a release cadence that outpaces almost every other inference engine in the space. That pace is worth keeping in mind as you work through this tutorial: flags and defaults do shift between versions, so treat any specific number here as a snapshot rather than a permanent fact, and check the official docs if something doesn’t match what you see on screen.

Kwon, who helped build vLLM and discussed its design in a recent conference talk, put it simply:

vLLM is an inference engine. So what it does is to take open source large language models that you can download from Hugging Face … to run it efficiently on data center hardware.

Kwon, conference talk on vLLM

That framing separates two jobs people often lump together. Training and fine-tuning happen elsewhere, often on rented clusters or through a lab’s own infrastructure. vLLM’s job starts once you already have model weights and need to serve them to real users without renting a hosted API for every single call. If you’re weighing self-hosted open models against frontier options like the ones we tracked in our Claude Opus 4.8 leaderboard coverage, the tradeoff usually comes down to cost at volume, data control, and how much engineering time you can spend on infrastructure instead of product work.

Self-hosting isn’t the right call for every team. If your traffic is bursty and low-volume, a hosted API from a frontier lab is usually cheaper once you count engineering time. But for teams with steady, high-volume inference traffic, or data residency requirements that rule out sending prompts to a third party, running your own serving layer on your own GPUs changes the cost and control equation. vLLM is the tool most of those teams reach for first.

vLLM vs Ollama vs LM Studio: Which Local Inference Engine Fits Your Stack?

vLLM is not the only way to run an open-weight model outside a hosted API. Ollama and LM Studio solve a related but different problem: getting a model running quickly on a single machine with minimal setup. We covered the Ollama path in detail in our guide to running a local LLM with Ollama. The short version of how the three compare:

Aspect	vLLM	Ollama	LM Studio
Primary use case	Production-grade serving at scale	Quick local runs, one user at a time	Desktop experimentation with a GUI
Interface	CLI plus an OpenAI-compatible API server	CLI plus a REST API	Desktop GUI plus an optional local server
Model format	Hugging Face safetensors	GGUF	GGUF
Request handling	Continuous batching via PagedAttention	Simple, largely single-request focused	Simple, largely single-request focused
Multi-GPU support	Built-in tensor parallelism	Limited	Limited
Quantization options	AWQ, GPTQ, FP8 and more	GGUF quant levels (Q4, Q5, Q8, etc.)	GGUF quant levels (Q4, Q5, Q8, etc.)
License	Apache 2.0	MIT	Free proprietary app
Best for	Self-hosted production APIs	Fast local testing, laptop use	Non-technical local experimentation

If you’re prototyping a feature on your own machine, Ollama or LM Studio will get you a working chat interface in minutes. Once that prototype needs to serve concurrent traffic from real users, vLLM is the more natural next step, and this tutorial assumes that’s where you’re headed.

None of this makes Ollama or LM Studio the “wrong” choice. They’re optimized for a different job. A solo developer testing prompts, or a non-technical team member who wants a local model for personal use, gets far less value from vLLM’s batching and multi-GPU machinery than from Ollama’s one-command simplicity. The decision point is traffic, not skill level: the moment more than one request needs to hit the model at the same time, on hardware you control, vLLM’s design starts paying for itself.

Prerequisites: Hardware, Drivers and Software Versions for vLLM

Confirm these before you start. Skipping this step is the single most common reason a first vLLM install fails partway through.

• Operating system: Linux is the best-supported and most common target, with Ubuntu 22.04 or newer being typical. Windows users should plan on WSL2 or Docker rather than a native install.
• Python: version 3.10 or newer is required, and vLLM’s own documentation recommends 3.12 or newer for the smoothest experience.
• GPU: an NVIDIA GPU with CUDA 12.9 or CUDA 13.0 support, per vLLM’s current install matrix. 16GB or more of VRAM is a practical minimum for small models in the 1 to 8 billion parameter range.
• Package manager: pip works, but vLLM’s documentation now recommends uv for faster, more reliable dependency resolution.
• Disk space: at least 20 to 50GB free for model weights and caches, more if you plan to keep several models on hand.
• Optional: a Hugging Face account and access token if you plan to download gated model repositories.
• Optional: Docker with the NVIDIA Container Toolkit installed if you want to follow the containerized path in Step 9.

If you’re still deciding what hardware to buy for local inference rather than working with what you already have, our comparison of the Nvidia DGX Spark against the Mac Studio is a useful reference point for how far a given memory budget actually goes.

How Much VRAM Does Your Model Actually Need?

Before picking a model in the next step, it helps to do the arithmetic instead of guessing. The rule of thumb is simple: multiply the parameter count by the bytes used per parameter, then add headroom for the KV cache and activation memory. A model loaded at 16-bit precision uses roughly 2 bytes per parameter, an 8-bit quantized version uses roughly 1 byte, and a 4-bit quantized version, like AWQ or GPTQ, uses roughly half a byte. None of these numbers are exact, since framework overhead and your chosen context length both add on top, but they get you close enough to avoid downloading a model that was never going to fit.

Model Size	16-bit Weights Only	8-bit Quantized	4-bit Quantized (AWQ/GPTQ)
1.5B parameters	~3GB	~1.5GB	~0.8GB
7-8B parameters	~16GB	~8GB	~4GB
13-14B parameters	~28GB	~14GB	~7GB
34B parameters	~68GB	~34GB	~17GB
70B parameters	~140GB	~70GB	~35GB

Treat these as weights-only figures, not a full sizing plan. The KV cache adds more on top of every number in that table, and how much more depends on your --max-model-len setting and how many concurrent requests you expect to serve at once. A 7B model that just barely fits at 16-bit precision with no room for the KV cache will fail under real traffic even though the initial load succeeds. Leave at least 20 to 30 percent of your GPU’s memory unallocated for that reason, which is exactly what the --gpu-memory-utilization flag from Step 6 controls.

Step 1: Create a Python Virtual Environment for vLLM

Keep your vLLM install isolated from other Python projects on the same machine. vLLM pulls in a specific, fast-moving stack of PyTorch and CUDA-linked dependencies that can conflict with unrelated packages if you install everything into one global environment.

python3 -m venv vllm-env
source vllm-env/bin/activate
python -m pip install --upgrade pip

On Windows inside WSL2, the same three commands work unchanged once you’re inside the Linux subsystem’s shell.

Step 2: Install vLLM and Verify GPU Access

vLLM’s documentation recommends installing with uv for speed, though a plain pip install works too. Install uv first, then let it resolve the right PyTorch backend automatically.

pip install uv
uv pip install vllm --torch-backend auto

Before you spend time downloading a multi-gigabyte model, confirm your GPU driver and vLLM install are both working.

nvidia-smi
python -c "import vllm; print(vllm.__version__)"

nvidia-smi should print your GPU name, driver version and current memory usage. If that command isn’t found, stop here and fix your NVIDIA driver installation before continuing. The second command should print a version string with no import errors. If you see a CUDA-related error at this step already, jump ahead to the troubleshooting table later in this guide, since that specific failure is a documented, common one.

Step 3: Download an Open-Weight Model From Hugging Face

vLLM downloads models directly from the Hugging Face Hub the first time you reference them by repository ID, caching the weights locally so later restarts skip the download. For this tutorial, use a small instruction-tuned model that fits comfortably on a single consumer GPU. If you’d rather compare options before committing to a download, our breakdowns of Phi-4 Mini vs Gemma 3 vs Llama 3.2 and DeepSeek V4 vs GLM-5.2 vs Qwen cover the open-weight models most teams are actually choosing between right now.

pip install "huggingface_hub[cli]"
huggingface-cli login   # only needed for gated model repositories

huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct --local-dir ./models/qwen2.5-1.5b

You can also skip the manual download step entirely and let vllm serve pull the model on first launch, which the next step does. Downloading it separately first is mainly useful when you want to see the file sizes and confirm disk space before the server process starts.

Step 4: Launch the vLLM OpenAI-Compatible API Server

With a model chosen, one command starts a full API server. vLLM’s own quickstart documentation shows this exact pattern:

Run the following command to start the vLLM server with the Qwen2.5-1.5B-Instruct model:

vLLM Documentation, Quickstart

vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000

By default, according to the same documentation, “it starts the server at http://localhost:8000.” The first launch takes longer than later ones because the model still needs to download and load into GPU memory. A successful startup ends with log lines that look like this:

INFO:     Started server process [48213]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Leave this process running in its own terminal window or background it with a process manager. Everything from here on assumes the server is up.

Step 5: Send Your First Chat Completion Request

Test the server with a plain curl request before wiring up any application code. The endpoint shape matches OpenAI’s Chat Completions API, so anything already written against that format works here once you change the base URL.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in one sentence."}],
    "max_tokens": 100
  }'

A working server returns a JSON payload shaped like this:

{
  "id": "chatcmpl-a1b2c3d4",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "PagedAttention manages the model's key-value cache in small, non-contiguous blocks, similar to how an operating system pages virtual memory, so GPU memory is used more efficiently."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 14, "completion_tokens": 33, "total_tokens": 47}
}

If you get a connection-refused error instead, the server likely hasn’t finished loading yet. Wait for the “Application startup complete” line from Step 4 before retrying.

Step 6: Tune GPU Memory Utilization and Batching

vLLM ships with defaults tuned for dedicated GPUs, not shared ones. NVIDIA’s own vLLM release notes flag this directly: vllm serve uses aggressive GPU memory allocation by default, reserving close to the entire card, which causes problems on systems with shared or unified GPU memory. The fix is to set --gpu-memory-utilization lower, commonly around 0.7, so the process leaves room for the rest of the system.

vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.7 --max-model-len 8192

These are the flags worth understanding before you tune anything else:

Flag	Default	What It Controls	Recommended Setting
–gpu-memory-utilization	Close to 1.0	Fraction of GPU memory reserved for weights and KV cache	0.7 to 0.9; lower on shared or unified-memory systems
–tensor-parallel-size	1	Number of GPUs the model is sharded across	Match your GPU count on multi-GPU nodes
–max-model-len	The model’s built-in maximum	Maximum sequence length, prompt plus completion	Lower it if you don’t need the full context window, to save VRAM
–quantization	None	Quantization method, such as awq, gptq or fp8	Set to match the format of the model you downloaded
–dtype	auto	Weight precision, such as float16 or bfloat16	Leave on auto unless you hit precision-related errors
–port	8000	Port the API server listens on	Change only if 8000 is already in use
–api-key	None	Requires a bearer token on every request	Always set this before exposing the server beyond localhost

Continuous batching, the mechanism that lets vLLM interleave multiple in-flight requests instead of processing them one at a time, works automatically in the background. There’s no flag to turn it on. Your job at this step is mainly making sure the server has enough memory headroom to actually take advantage of it under real load.

Think of these settings as a chain rather than independent dials. Raising --max-model-len gives every request more room for context, which leaves less memory for the KV cache, which shrinks how many requests can batch together at once, which lowers overall throughput even though nothing about the GPU itself changed. If your use case genuinely needs long context, that tradeoff is worth making. If it doesn’t, an unnecessarily high --max-model-len is one of the most common reasons a vLLM deployment underperforms its hardware.

Step 7: Enable Multi-GPU Tensor Parallelism

Once a model is too large for one GPU’s memory, tensor parallelism splits its layers across multiple GPUs on the same machine. Set --tensor-parallel-size to the number of GPUs you want to use, and vLLM handles the sharding automatically.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9

The number you pass has to evenly divide the model’s attention heads, so common values are powers of two: 2, 4 or 8. GPUs connected with a fast interconnect like NVLink see far less overhead from this than GPUs communicating only over PCIe, so factor your hardware’s actual topology into how much throughput you expect before committing to a specific server configuration.

Step 8: Quantize Your Model to Cut VRAM Usage

Quantization reduces the numeric precision of a model’s weights, shrinking memory footprint at some cost to output quality. AWQ and GPTQ are two popular post-training quantization methods that compress weights to roughly 4 bits, and both are widely used to fit larger models onto smaller GPUs. FP8 keeps more precision than either and has gained ground on newer GPU generations with native FP8 support in hardware.

Community-quantized variants are commonly published under a matching repository name with an -AWQ or -GPTQ suffix. Point vLLM at one and set the matching flag:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ --quantization awq

Quantization is not free. Expect some drop in output quality, particularly on tasks that need precise reasoning or exact formatting. Test your specific use case before committing a quantized model to production, rather than assuming the quality hit will be negligible.

Step 9: Containerize vLLM With Docker

Running vLLM inside Docker sidesteps most of the driver and dependency headaches from a bare-metal install, and it’s the path most production deployments actually use. Install Docker and the NVIDIA Container Toolkit first, then run the official image.

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-1.5B-Instruct

The volume mount matters more than it looks. Without it, the container re-downloads the model from Hugging Face every time it restarts, since the cache directory would otherwise live only inside the disposable container filesystem. The --ipc=host flag avoids a shared-memory error that PyTorch’s multiprocessing can throw inside a default Docker container.

Add a restart policy once you move past manual testing, since a crashed container should come back on its own rather than waiting for someone to notice.

docker run --runtime nvidia --gpus all \
  --restart unless-stopped \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-1.5B-Instruct --api-key your-secret-key

Notice the --api-key flag added at the end. Anything you run outside a fully trusted, private network should carry one from the first container run, not as a follow-up step you mean to get back to later.

Step 10: Put vLLM Behind a Reverse Proxy With HTTPS

vLLM’s own server speaks plain HTTP. Exposing that directly to the internet means unencrypted traffic and no protection beyond whatever API key you’ve set. A lightweight reverse proxy fixes both. Caddy is a common choice because it handles TLS certificates automatically with almost no configuration.

your-domain.com {
    reverse_proxy localhost:8000
}

That three-line Caddyfile is enough to get a working HTTPS endpoint in front of your vLLM server, including automatic certificate renewal. Nginx works too if your team already standardizes on it, but expect to write more configuration by hand, including your own certificate management through something like Certbot.

Treat the reverse proxy as more than a TLS box to check off. It’s also the natural place to add rate limiting, so one misbehaving client can’t monopolize your GPU queue, and to restrict access by IP range if the server only needs to be reachable from your own application infrastructure rather than the open internet. Neither of those belongs inside vLLM itself, since the serving engine’s job is running the model, not acting as a firewall.

Step 11: Monitor Throughput, Latency and GPU Usage

vLLM exposes a Prometheus-formatted metrics endpoint alongside the API itself, which is what most teams wire into an existing Grafana dashboard rather than building custom logging from scratch.

curl http://localhost:8000/metrics | head -20
nvidia-smi dmon -s u

The first command shows the raw metrics vLLM tracks internally, including request counts and latency histograms. The second gives you a quick, continuously updating terminal view of GPU utilization without installing anything extra, useful for a fast sanity check while you’re still tuning the flags from Step 6.

Step 12: Deploy vLLM to Kubernetes for Auto-Scaling

Once vLLM is running reliably in Docker, wrapping it in a Kubernetes Deployment gets you restart policies, rolling updates and autoscaling on top of what you already built. The main thing that differs from a typical web service deployment is requesting GPU resources explicitly through the NVIDIA device plugin.

resources:
  limits:
    nvidia.com/gpu: 1
env:
  - name: HF_HOME
    value: /root/.cache/huggingface

Pair that resource block with a persistent volume for the Hugging Face cache so pods don’t re-download model weights on every restart, and a readiness probe against the server’s health endpoint so traffic doesn’t route to a pod that’s still loading.

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 120
  periodSeconds: 30

The generous initialDelaySeconds values matter here. Model loading can take well over a minute depending on size and disk speed, and a probe that fires too early will report the pod as unhealthy and trigger a restart loop before the model ever finishes loading in the first place. If you’re deploying specifically on AWS, our Amazon EKS setup guide covers the cluster side of this in more depth. Scale on request latency or queue depth rather than raw CPU metrics, since GPU-bound workloads don’t show CPU pressure the way typical web services do.

Complete Working Project: A FastAPI Chat Service on Top of vLLM

With the server running, here’s a small, complete project that wraps it in your own API layer instead of exposing vLLM’s raw endpoint directly to client applications. This is the pattern most teams land on in practice, since it lets you add logging, rate limiting or auth without touching vLLM itself.

pip install fastapi uvicorn openai

# app.py
from fastapi import FastAPI
from pydantic import BaseModel
from openai import OpenAI

app = FastAPI()
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

class Prompt(BaseModel):
    message: str

@app.post("/chat")
def chat(prompt: Prompt):
    completion = client.chat.completions.create(
        model="Qwen/Qwen2.5-1.5B-Instruct",
        messages=[{"role": "user", "content": prompt.message}],
        max_tokens=300,
    )
    return {"reply": completion.choices[0].message.content}

uvicorn app:app --port 9000

That’s a working microservice in under twenty lines, and it demonstrates the point of running an OpenAI-compatible server in the first place: the openai Python package talks to your self-hosted vLLM instance with no changes beyond the base_url. If you’re building something more involved, like a retrieval-augmented system on top of this same server, our RAG pipeline tutorial picks up right where this one leaves off.

Common Pitfalls When Self-Hosting vLLM

Most first attempts at running vLLM hit one of these problems. Check this list before you assume something is broken with your hardware.

• Picking a model that doesn’t fit your VRAM. Check the parameter count and precision before downloading anything. A 70B model at 16-bit precision needs well over 140GB of VRAM before you’ve even started serving requests, which rules out most single-GPU setups without quantization or sharding.
• Downloading a GGUF file by mistake. GGUF is the format Ollama and llama.cpp use. vLLM expects the Hugging Face safetensors format instead, so a model repository built for GGUF users simply won’t load.
• Leaving the server open with no API key. vLLM will bind to every network interface with no authentication unless you explicitly set --api-key. That’s fine on a private network and a real problem the moment that port reaches the public internet.
• Not pinning package versions. vLLM ships new releases at a fast pace, moving from version 0.6 to well past 0.20 in roughly eighteen months. An unpinned upgrade can silently change default behavior or break compatibility with your installed PyTorch build.
• Forgetting how aggressive the default memory setting is. On a shared machine, or one running other GPU processes, the near-100% default for --gpu-memory-utilization can starve everything else or crash on startup.
• Skipping the reverse proxy. Running vLLM’s raw HTTP server directly on the internet without TLS in front of it is a common shortcut during testing that quietly becomes a production habit.
• Benchmarking on a cold server. The first few requests after startup include one-time costs like CUDA graph compilation and cache warming. Judging your real throughput from those first requests will make vLLM look slower than it actually is once it’s warmed up.

Troubleshooting vLLM: 8 Common Errors and Fixes

Match your error message against this table before searching further. Most installation-stage failures are one of these eight.

Error or Symptom	Likely Cause	Fix
CUBLAS_STATUS_INVALID_VALUE on CUDA 12.9+	Conflicting system CUDA libraries on LD_LIBRARY_PATH	Unset or remove the system CUDA path from LD_LIBRARY_PATH, or reinstall with a matching PyTorch and CUDA wheel
CUDA out of memory on startup	–gpu-memory-utilization set too high, or model too large for available VRAM	Lower –gpu-memory-utilization, for example to 0.7, or switch to a smaller or quantized model
Server hangs at “Loading model weights”	Incomplete model download or a disk I/O bottleneck	Re-download with huggingface-cli, and confirm free disk space and network throughput
ImportError or version mismatch between torch and vllm	Installed a vLLM build compiled against a different PyTorch or CUDA version	Reinstall with uv pip install vllm –torch-backend auto, or match versions from the official install matrix
“Unsupported architecture” GPU error	Installed wheel wasn’t built for your GPU’s compute capability	Confirm your GPU is on the supported list, and install the CUDA build matching your driver version
Connection refused on localhost:8000	Server still loading, or bound to a different host or port	Wait for the “Application startup complete” log line, and double-check your –host and –port flags
401 or 403 responses from API requests	–api-key set on the server but not sent by the client	Include an Authorization: Bearer header on every request that matches your –api-key value
Garbage or truncated output	–max-model-len set too low, truncating available context	Raise –max-model-len, or shorten your prompts to fit the current limit

Advanced Tips for Running vLLM in Production

Once the basic setup works, these are the changes that actually move throughput and reliability numbers for teams running vLLM at real scale.

• Turn on automatic prefix caching for repeated system prompts. If many requests share the same long system prompt or few-shot examples, prefix caching avoids recomputing that shared prefix every time, which matters most for RAG and agent workloads carrying long, repeated context.
• Use guided decoding when you need structured output. vLLM supports constraining generation to a JSON schema or regex pattern, which is far more reliable than asking a model nicely to return valid JSON and hoping.
• Separate prefill-heavy and decode-heavy traffic on larger clusters. Prompt processing and token generation have different performance characteristics, and teams running at scale sometimes split them onto different node pools to keep both stages efficient.
• Serve multiple fine-tuned variants from one base model with LoRA adapters. Instead of deploying a full separate copy of the model for every fine-tune, vLLM can load lightweight LoRA adapters on top of one shared base model.
• Watch queue depth, not just GPU utilization, when deciding when to scale. A GPU can show 100% utilization while requests still queue for seconds, so latency and pending-request count are better autoscaling signals than raw utilization alone.
• Warm the server before you route real traffic to it. Send a handful of throwaway requests right after startup so CUDA graph compilation and cache warming happen before a paying customer’s request is the one waiting on them.

Frequently Asked Questions About Running vLLM

What GPU do I need to run vLLM?

vLLM runs on NVIDIA GPUs with CUDA support as its primary, best-supported path, with AMD ROCm and other backends also available. For small models in the 1 to 8 billion parameter range, a single consumer GPU with 16 to 24GB of VRAM, like an RTX 4090, is typically enough once you apply quantization. Larger models need data-center GPUs or several GPUs split with tensor parallelism.

Is vLLM free to use commercially?

Yes. vLLM is released under the Apache 2.0 license, which permits commercial use, modification and redistribution. You still need to separately check the license of whatever model weights you load, since a model’s license is independent from the serving engine’s license.

Can I run vLLM without a GPU, using CPU only?

vLLM has experimental CPU support on some backends, but it’s built around GPU acceleration, and PagedAttention’s memory-management benefits are far smaller without one. For anything beyond light testing, plan on a GPU.

How is vLLM different from Ollama or llama.cpp?

Ollama and llama.cpp are built around the GGUF model format and optimized for running a single model comfortably on one machine, including laptops without a dedicated GPU. vLLM expects Hugging Face-format weights, focuses on serving many concurrent requests efficiently through continuous batching, and is the more common choice once you’re serving real production traffic rather than a personal chat session. If you’re just getting started with local AI on a laptop, our Ollama setup guide is the faster starting point.

Does vLLM support Windows?

Native support is strongest on Linux. Windows users typically run vLLM inside WSL2 or a Docker container rather than installing it directly, which is also the path most production deployments use regardless of host operating system.

How much VRAM does a small model need with vLLM?

It depends on the model’s parameter count, the precision you load it at, and how much context length you reserve for the KV cache. A quantized 7 to 8B model can often fit in under 10GB of VRAM, while the same model at full 16-bit precision needs closer to 16GB before you even account for the KV cache and batching headroom.

Can I use vLLM with a model that isn’t listed on Hugging Face?

Yes, as long as the weights are in a format vLLM’s model loader supports, generally Hugging Face’s safetensors format, and the architecture is one vLLM has implemented. Point vllm serve at a local directory path instead of a Hugging Face model ID.

Is the vLLM API really a drop-in replacement for the OpenAI API?

For the core chat and completion endpoints, yes, that’s the explicit design goal, which is why the official Docker image is even named vllm-openai. Existing code written against OpenAI’s Python SDK usually only needs its base URL changed to point at your vLLM server.

Do I need to restart vLLM every time I want to serve a different model?

With a single vllm serve process, yes, one server instance loads one model at a time. Teams that need to serve several different models at once typically either run multiple vLLM processes on different ports and GPUs, or use LoRA adapters on top of one shared base model when the variants are close enough to share a foundation, as described in the advanced tips above.

What happens if I send more requests than my hardware can handle?

Requests queue rather than failing outright, up to whatever limits your deployment sets. Continuous batching means vLLM keeps admitting new requests into the running batch as capacity frees up, so throughput degrades gradually under heavy load instead of falling over immediately. That said, a queue that keeps growing faster than it drains is a sign you need more GPU capacity, not a problem to tune your way out of indefinitely.

At this point you have a self-hosted model behind an authenticated, HTTPS-terminated, OpenAI-compatible API, tuned for your GPU and ready to scale beyond a single machine if traffic grows. The next decisions are less about vLLM itself and more about what you build on top of it: which models you standardize on, how you monitor cost per request instead of just uptime, and whether a Kubernetes-based deployment or a simpler managed GPU host makes more sense for your team’s size. Revisit the flags in Step 6 as your traffic pattern becomes clearer. The defaults that work for a five-person internal tool rarely stay right once real users show up.

Related Coverage

• How to Run a Local LLM With Ollama: 13 Steps, 90 Min
• How to Build a RAG Pipeline: 12 Steps, 90 Min
• DeepSeek V4 vs GLM-5.2 vs Qwen: 10x Price Gap
• Phi-4 Mini vs Gemma 3 vs Llama 3.2: 128K vs 32K
• Nvidia DGX Spark vs Mac Studio: 128GB vs 512GB RAM
• How to Set Up Amazon EKS: 12 Steps, 90 Min
• More AI and Machine Learning coverage