What is Quantization? — AI Glossary

What it is: Quantization is a technique for shrinking an AI model by storing its internal numbers (weights) at lower precision — 8-bit or 4-bit instead of the standard 16-bit or 32-bit. The result: the same model runs on less powerful hardware, often with little quality loss.
Who it is for: Developers running AI models on local hardware (Mac, PC, Raspberry Pi) and anyone curious why a 70-billion-parameter model fits on a laptop.
Best if: You want to run large open-weight models like Llama or Mistral locally and need to fit them in limited GPU memory.
Skip if: You only use cloud-based AI assistants — quantization happens on the provider’s side, invisible to you. Want one practical AI workflow every morning? Subscribe to our free daily newsletter.

What is quantization?

Quantization is a model-compression technique that stores the internal numbers (weights) of a neural network at lower numerical precision. Instead of representing each weight as a 16-bit or 32-bit floating-point number, quantized models use 8-bit, 4-bit, or even 2-bit integers. This dramatically reduces the model’s memory footprint — and surprisingly, often without major quality loss.

A typical example: Llama 3.1 70B at full precision needs about 140 GB of GPU memory, far beyond any consumer GPU. The same model quantized to 4-bit needs about 40 GB and runs on a high-end consumer setup. Quantized to 2-bit (more aggressive), it can fit on a single 32 GB Mac with Apple Silicon.

Why does quantization matter?

Quantization is why running powerful open-weight AI on a laptop is feasible at all in 2026. Without it, you’d need data-center hardware to use any frontier-quality model. With it, hundreds of thousands of developers and hobbyists run capable AI locally — for privacy, cost control, or offline use.

The technical surprise that drove quantization adoption: model quality degrades much less than naive math would suggest. A well-quantized 4-bit model might score within 1-2% of the full-precision version on most benchmarks. That’s an extraordinarily good trade-off given the 4x reduction in memory.

How is quantization done?

Three main approaches you’ll see referenced:

• Post-training quantization (PTQ) — take an already-trained model and convert its weights to lower precision. Fast and common. Examples: GPTQ, AWQ, GGUF.
• Quantization-aware training (QAT) — train the model with quantization in mind from the start. More work but produces better low-bit results.
• Mixed precision — some weights (the ones most affected by precision loss) stay high-precision while others get aggressively quantized.

For end users: if you download a model from Hugging Face or use a tool like Ollama, llama.cpp, or LM Studio, you’ll see quantization labels like “Q4_K_M” or “4-bit AWQ.” These describe the quantization scheme. Lower bits = smaller file but slightly worse quality.

Related terms

• Llama
• Hugging Face
• Open Weights
• Knowledge Distillation

Learn more on Beginners in AI

• Ollama — Run Local AI Models
• AI Glossary

Sources and further reading

• Hugging Face — Quantization Documentation
• GPTQ: Post-Training Quantization for Generative Pre-trained Transformers

Last reviewed: May 2026. AI terminology evolves quickly — verify specifics on the official source pages above.

You may also like

• Ollama — Run Local AI Models
• AI Glossary
• Llama
• Hugging Face
• Open Weights
• Knowledge Distillation