Do you really understand why local LLMs work? Unraveling the mechanics of 'AI Quantization'

When I first ran a local LLM, there was one thing that bothered me.

Wait, Llama 3 70B means 70 billion parameters, right? Why does that run on a regular laptop?

At the time, I felt like I had heard that 'something called quantization compresses it,' so I just accepted it and moved on. But honestly, the mechanism remained vague.

Quantization is a technology that converts AI model data into a rougher form to make it lighter.that's what it is.

So, I recently looked into it properly, and it was more interesting than I thought. It's not just 'compression'; there's a more deliberate strategy involved. Today, I'd like to look at that together.

🔢 First of all, what is this talk about 'numerical precision'?

Roughly speaking, the contents of an AI model are a massive chunk of numbers. Parameters called 'weights' of a neural network are lined up, ranging from billions to tens of billions.

Usually, these numbers are stored in formats like FP32 (32-bit floating point) or FP16 (16-bit). In short, it's about 'how many decimal places to express precisely,' and the more digits, the more accurate it is, but it consumes more memory.

For example, if you save the Llama 3 70B (70 billion parameter) model in FP16, it is about 140GB in calculation. It far exceeds the VRAM of a typical GPU, right?

That's where 'quantization' comes in.

⚖️ The basics of quantization: reducing precision to make it lighter

Quantization is a technology that dramatically reduces the size of a model by converting FP16 (16-bit) numbers into INT8 (8-bit integers) or INT4 (4-bit integers).

Roughly, here is the image.

• FP16 (16-bit) → Number of representable values: 65,536

• INT8 (8-bit) → 256

• INT4 (4-bit) → 16

You might think, 'Wait, only 16 options? Is that okay?' I thought so too.

But in reality, most of the model's weights are concentrated in a specific range, and if you map them efficiently to integers, the degradation in precision is not as large as you might think. Experiments over the last few years have shown that there is almost no degradation with 8-bit, and even 4-bit is within a practical range for many tasks.

🛠️ Main quantization formats: GGUF, GPTQ, AWQ

If you've ever tried a local LLM, you might have seen file names like these.

These terms, 'GGUF' and 'Q4_K_M', represent the quantization format and method. Here is a summary of the main ones.

GGUF (A versatile format for CPUs)

This is the format used by the open-source project llama.cpp, and its biggest feature is that it allows LLMs to run on CPUs alone without needing a GPU. This is what Ollama and LM Studio use.

In 'Q4_K_M', 'Q4' stands for 4-bit quantization, 'K' refers to the mixed-precision strategy (K-quants) that varies the bit count between important and less important layers, and 'M' stands for Medium. It carefully quantizes by adjusting the importance for each layer.

If you are trying out a local LLM for the first time, starting with Q4_K_M is probably the safest bet in terms of balance.

GPTQ (The standard for GPUs)

This method uses gradient information to minimize quantization error and is optimized for inference on GPUs. It has a longer history in the ecosystem than GGUF. However, the fact that quantization takes time and it is GPU-exclusive are drawbacks, and it is recently being challenged by AWQ, which is described below.

AWQ (Becoming the new standard for GPUs)

AWQ is positioned similarly to an evolution of GPTQ, and its unique concept is 'distinguishing between weights that significantly affect accuracy and those that do not.' It protects only the high-importance parameters with high precision while aggressively quantizing the rest. It tends to have higher accuracy than GPTQ at the same bit count, and the quantization processing speed is also faster. Since 2024, it has rapidly gained popularity as a practical choice for GPUs.

🎯 'Which one to choose' depends on your environment

How you choose quantization depends heavily on your PC environment.

• No GPU, or want to try it on CPU only → GGUF Q4_K_M

• Have a GPU and enough VRAM → Consider AWQ or GPTQ

• Need no quantization (highest accuracy) → Full model in BF16 or FP16 (requires a lot of VRAM)

The important thing is not to feel too strongly that you are 'sacrificing accuracy.' At around 8 bits, you often cannot feel the difference from the full model in daily Q&A or text generation. Even at 4 bits, it is 'good enough' for many tasks.

I remember when I first tried a quantized model, I thought, 'Oh, it answers just fine.' That is also the fun part of local LLMs.

🤔 Miccell's perspective: I think quantization is the very 'democratization of AI'

Without quantization, LLMs would have remained exclusive to the cloud. We could have ended up in a world where only a few companies with expensive GPU servers could use them.

But thanks to advancements in quantization technology, even a laptop on hand can now run 70B-class models. You can run AI locally without connecting to the internet or sending data to some server.

This isn't just 'amazing' in a simple sense; it's a pretty big deal from the perspectives of privacy, cost, and the fact that it increases the number of people who can access it.

If you follow the technical details, the world of quantization is still evolving rapidly. For example, there is research into 1-bit quantization (BitNet), exploring ways to make models work properly even with extremely reduced weight precision. The challenge of how much we can trim down continues.

📝 Summary: Knowing about quantization changes how you see local LLMs

To summarize today's discussion briefly, it looks like this.

• Quantization is a technology that reduces the file size of AI models by lowering the numerical precision (bit count) of the model.

• While precision drops slightly, 8-bit is practically fine for almost all uses, and 4-bit is usable in many cases as well.

• GGUF is a versatile format that runs on CPUs, while AWQ and GPTQ are for GPUs.

• Without quantization, the option of local LLMs would not exist.

For those who have run Llama 3 on Ollama, that comfort is only possible because of quantization. When you think about it that way, don't you feel like 'once you understand the mechanism, you feel a bit more attached' to it?

I do (laughs).

For those who have actually tried quantized models, please let me know in the comments which format you found easiest to use.

🛠️ Production Notes

For this article, it was quite difficult to decide 'where to start' when talking about quantization.

The moment I brought up terms like FP32 and INT8, it seemed like it would become 'too difficult,' but I think it became a bit easier to read by starting from my own experience of 'not understanding why a 70B model could run on a laptop'...

Also, regarding the three topics of GGUF, GPTQ, and AWQ, I could have easily made a comparison table, but since note doesn't support tables and given the tone of this article, I chose the direction of 'understanding through the flow.' How does it feel when you read it back?

Miccell - Once you understand the mechanism, the world becomes more interesting.