I tried running gemma3 with quantization. Is it acceptable even on an N100 CPU!? ~CPU inference with llama-cpp (GPU is blazing fast...)~

Hello, this is Rcat.
This time, I'm talking about large language models.
Until now, whenever a new model came out, I would try it out by running inference directly on the GPU, butthis time I'm going to trya quantized model.

Next time, I will
turn the test script I used this time into a local AI server.

Introduction

Terms of Service

Please check the terms of service in advance when using information or works.

About comments

Please check the terms of service guidelines before commenting.
Comments that do not comply will be deleted.

Overview

What's this all of a sudden?

I recently found an edge AI app for Android smartphones called Edge Gallery. This app runs LLMs using the smartphone's CPU, and I was so surprised that I had to try it out.
As a result,although it's somewhat slow, it's perfectly usable...

When I looked into why it runs on a smartphone, I found that the model is significantly reduced through quantization. Even on a regular PC, I was limited to running 3B models on the GPU, buta 4B model runs on a smartphone.

So, I thought,if I do the same thing on a PC, it should be at a level where it's perfectly usable,so I decided to give it a try.

What is quantization?

Quantization is a technique to reduce the size of a model by reducing the bit count of its parameters. By sacrificing some accuracy, this is expected to significantly reduce memory usage and improve inference speed.

For example,until now, GPUs used 16-bit floating point,butwhat I'm testing this time is reduced down to 4-bit integers.
In terms of bit count, that's 1/4.Furthermore,since it's converted to integers, the computational cost is also significantly reduced.

Trying it out

Library used

I will use llama-cpp this time. Since it's a Python library, I can use it with the same feel as before.

Model used

I will usethe same foundation as Geminideveloped by Google,the open-source gemma-3.
I was impressed by how it actually ran on the smartphone app, so I wanted to try using it on my PC as well.
Also, since I have Gemini do all my usual automation tasks, I have high hopes that a model from the same family will return answers similar to what I'm used to.

Since there was no official GGUF format compatible with llama-cpp, I will use models converted by other providers.
This time, I chose from the following.

https://huggingface.co/unsloth/gemma-3-4b-it-qat-GGUF/tree/main

I wasn't sure how to choose a model, so I asked an AI and received the following answer.

たくさんのファイルがあって迷ってしまいますよね！はい、モデル本体は通常1つのGGUFファイルで構成されています。 他のファイルは、量子化のレベルが異なるバリエーションや、関連する設定ファイル、あるいはマルチモーダル機能のための補助ファイルなどです。

このリストの中から、ノートPCのCPUで使うことを前提に、性能とファイルサイズのバランスが最も良いと思われるGGUFファイルを選んでいきましょう。

あなたが選ぶべきGGUFファイル
おすすめは、以下のファイルです。

gemma-3-4b-it-qat-Q4_K_M.gguf
サイズ: 2.49 GB
理由:
前回の説明で推奨した Q4_K_M 量子化レベルです。これは、ファイルサイズとモデルの出力品質のバランスが非常に優れていると広く認識されています。
qat (Quantization-Aware Training) モデルなので、量子化による性能劣化が最小限に抑えられていることが期待できます。
ノートPCのCPUでも、このサイズであれば十分に動作する可能性が高いです。
他のファイルの解説と選ばない理由
gemma-3-4b-it-qat-BF16.gguf (7.77 GB):

これは「BF16」（BFloat16）形式で、量子化されていない、オリジナルの精度に近いモデルです。最も高品質ですが、ファイルサイズが非常に大きく、ノートPCのCPUで動かすには重すぎます。メモリも大量に必要とします。
gemma-3-4b-it-qat-Q2_K.gguf (1.73 GB), Q3_K_S.gguf (1.94 GB), Q3_K_M.gguf (2.1 GB) など、より小さいファイル:

これらはさらに強く量子化されており、ファイルサイズは小さく、読み込みも速くなります。しかし、その分、モデルの出力品質（応答の正確性や流暢さ）が低下する可能性があります。まずは Q4_K_M を試してみて、もしパフォーマンスがどうしても足りない場合に、これらのより軽量なモデルを検討すると良いでしょう。
gemma-3-4b-it-qat-Q5_K_M.gguf (2.83 GB), Q6_K.gguf (3.19 GB), Q8_0.gguf (4.13 GB) など、より大きいファイル:

これらは Q4_K_M よりも精度が高い可能性がありますが、その分ファイルサイズも大きくなり、ノートPCのCPUで動かす際のメモリ消費と推論速度に影響が出やすくなります。まずは Q4_K_M を試してみて、もし品質に不満があり、かつPCの性能に余裕があると感じた場合に、これらのより高精度のモデルを検討すると良いでしょう。
gemma-3-4b-it-qat-IQ4_NL.gguf (2.36 GB), gemma-3-4b-it-qat-IQ4_XS.gguf (2.26 GB) など IQ で始まるファイル:

これらは「I-Quantization (Importance Quantization)」と呼ばれる、比較的新しい量子化手法です。一部のテストでは Q4_K_M と同等か、特定の状況下でより良いパフォーマンスを示すこともあります。Q4_K_M でうまくいかない場合や、さらに試してみたい場合に選択肢となりますが、まずは最も標準的で安定している Q4_K_M から始めるのが無難です。
UD- で始まるファイル (例: gemma-3-4b-it-qat-UD-IQ1_M.gguf):

これは「User Defined」または特定の用途に最適化された量子化バージョンである可能性があります。通常の用途では推奨されません。
mmproj-*.gguf ファイル (例: mmproj-BF16.gguf):

これらは「マルチモーダルプロジェクター」の重みファイルです。Gemma 3は画像とテキストの両方を扱えるマルチモーダルモデルですが、llama-cpp-python でテキスト生成のみを行う場合は、通常このファイルは不要です。画像を扱うような複雑なアプリケーションを構築しない限り、ダウンロードする必要はありません。
.gitattributes, README.md, config.json, imatrix_unsloth.dat:

これらはモデルのメタデータ、説明、設定ファイル、または量子化に関する補助データであり、モデル本体のGGUFファイルではありません。ダウンロードする必要はありません。
結論
ダウンロードすべきファイルは、gemma-3-4b-it-qat-Q4_K_M.gguf です。

So, I will download the files I was told about. Also, since I want to run some benchmarks, I ultimately compared the following files.

• gemma-3-4b-it-qat-Q4_K_M.gguf
  The medium model that was most recommended

• gemma-3-4b-it-qat-IQ4_NL.gguf
  A model quantized based on parameter importance

• gemma-3-4b-it-qat-Q3_K_M.gguf
  A model reduced to 3-bit.

• gemma-3-4b-it-qat-UD-Q2_K_XL.gguf
  Trying out 2-bit as well

• gemma-3-4b-it-qat-UD-Q4_K_XL.gguf
  4-bit from the same series as the 2-bit

• gemma-3-4b-it-qat-UD-Q3_K_XL.gguf
  Same as above

Source code

Since this is my first time using it, I threw it at Gemini for now, then added a little bit myself.
This function supports blocks and streams (where text is displayed on the screen as it is generated) and returns the time taken for processing.
During streaming, it can also return the time taken until the first token is output.

Data to be used

The purpose of this activity is to perform the post analysis used in the following article on my own computer, so I will use comment information obtained from bulletin boards.
There are too many posts, and I've been hitting the API over 1000 times a day recently, so if I don't take some of the load myself, I'll seriously get restricted...

By the way, the input and output look like this.
I just feed the comments to the AI and have it evaluate whether they are useful in three stages.
It's great that SNS information is fast, but over 70% of the posts are irrelevant... That's a waste of time, so I'm using AI to make it more efficient like this.

By the way, Gemini 2.0 Flash Lite responds in about 5 seconds.

Operation test

I am using a CPU for inference, specifically a Ryzen 5 3600.

The test source looks like this.
I defined the comments at the top, but I've omitted them since they would be mosaicked anyway.
I took 6 comments from 3 bulletin boards each and checked the inference time and accuracy for each. 'Answer' is the value when I flagged it based on my own criteria.
The time is the average calculated after running inference on the 3 bulletin boards.

Prompt

I will keep this private here.
To explain briefly, the instructions were to divide the post into three stages and output it in JSON.

Results

The results were as follows.
Lowering the bits doesn't really make it faster, and sometimes it's even slower.At this rate, isn't Q4 MK just fine...
As for the accuracy... I suppose my instructions were bad. However, since it doesn't change much between models, I feel like you should just choose the one that runs fast.

{
  "gemma-3-4b-it-qat-IQ4_NL.gguf": {
    "time": "29.4",
    "first": "7.7",
    "score": "24.074074%"
  },
  "gemma-3-4b-it-qat-Q3_K_M.gguf": {
    "time": "25.5",
    "first": "7.0",
    "score": "24.074074%"
  },
  "gemma-3-4b-it-qat-Q4_K_M.gguf": {
    "time": "26.5",
    "first": "4.3",
    "score": "24.074074%"
  },
  "gemma-3-4b-it-qat-UD-Q2_K_XL.gguf": {
    "time": "32.8",
    "first": "10.3",
    "score": "22.222222%"
  },
  "gemma-3-4b-it-qat-UD-Q3_K_XL.gguf": {
    "time": "32.9",
    "first": "9.8",
    "score": "22.222222%"
  },
  "gemma-3-4b-it-qat-UD-Q4_K_XL.gguf": {
    "time": "29.3",
    "first": "6.0",
    "score": "25.925926%"
  }
}

By the way,when it drops below 3 bits, it feels like it lacks reading comprehension based on the output.It ignores instructions likewithin 10 charactersand so on.

The notable output of 2-bit looks like this.

Since what I'm expecting looks like this

Additional experiment

If it doesn't change much, I tried to see what would happen with a larger one.
The speed dropped here. The output is as expected, so the reading comprehension has improved, but since the time until the first token appears has become longer, I don't feel like I would dare to use it.

{
  "gemma-3-4b-it-qat-Q4_K_M.gguf": {
    "time": "26.6",
    "first": "4.4",
    "score": "24.074074%"
  },
  "gemma-3-4b-it-qat-Q5_K_M.gguf": {
    "time": "33.5",
    "first": "9.1",
    "score": "25.925926%"
  },
  "gemma-3-4b-it-qat-Q6_K.gguf": {
    "time": "33.9",
    "first": "8.0",
    "score": "25.925926%"
  }
}

What about on a GPU?

It's llama-cpp, but it seems you can install the GPU version by adding a little something during installation.
Click here for details

Build

It seems there is a pre-built one, but... surprisingly, CUDA 12.6 was installed, so I couldn't use it.
So I will build it myself.

For those who don't have CUDA or don't know about it, install from here
*Of course, this is for NVIDIA GPUs.
There seems to be one for ROCm too, so maybe it works on AMD GPUs??

It didn't go well halfway through, so I looked up various things, and finally, I typed the command below and succeeded. The tmp part might not be necessary, but even if you do it, it's only during the session, so it's fine.

set FORCE_CMAKE = 1
set CMAKE_ARGS=-DGGML_CUDA=on
set TEMP=C:\temp
set TMP=C:\temp
pip install llama-cpp-python --force-reinstall --no-cache-dir

The build took about 30 minutes and the installation was finished.

Execution results

It output 1000 characters in 15 seconds... This is an RTX2060...
If a Linux server has a GPU, it's at a level where some tasks could be completely migrated locally...
I don't have Thunderbolt, so I can't use an eGPU either, so I'm sunk...

Summary

This time, I tried a quantized LLM.
Until now, I had passed it by thinking that the accuracy would drop, but that's not the case at all.
Even if you lower the precision of the parameters, you can try a larger model for that amount, and since float has become int, it's not a burden at all even if it's large.

I've started wanting a local LLM server, but should I replace it with a PC with a GPU, or should I try my hand at the NPUs that have come out recently...
See you again.

Link Collection

Blazing fast LLM with Ryzen AI MAX+395 Radeon8060S

How to install Python

Installing Linux on a mini PC