How to use gguf-fit: Get the llama-server launch command in 2 lines

Last time, I wrote an article introducing a tool called gguf-fit. It was about a tool that determines the launch settings for llama-server just by reading a GGUF file, covering topics from discrepancies in quantization to actual measurements of the KV cache. I received feedback from readers saying it was "difficult."

Admittedly, you don't need all that information just to use it. This time, I will focus solely on how to use it.

The previous article

You don't need to read the previous article

To be clear, you don't need the previous article to read this one. I wrote about why gguf-fit can provide those numbers in the previous post, so please read it later if you are interested. Here, I will only write about "what to type to get what result."

What gguf-fit does for you

In a nutshell, it is a tool that tells you whether a GGUF file will fit on your GPU and how far you can extend the ctx (the upper limit of the conversation length it can remember) when you provide it with the file. It gives you the answer just by reading the contents of the file, without ever running the GPU.

I have many GGUF files with different quantizations, and I used to repeatedly launch and shut them down to check which ones would fit on my GPU. That is no longer necessary.

Installation

uv tool install git+https://github.com/zephel01/gguf-fit

If you don't have uv, you need to install it first, but this one line adds three commands.

Step 1: Read the file

Run this command pointing to the GGUF file you have on hand.

gguf-probe --json --out gguf.json /models/*.gguf

Replace `/models/*.gguf` with the location of your file. It will read all GGUF files in the folder and write them to a file called gguf.json. It finishes in a few seconds.

If you run it without the --json flag, a list will appear on the screen. If you actually read different quantizations of Qwen3.8-27B, it will look like this table (only 5 out of 12 types are extracted here).

file        GB     GiB   最多の型
Q3_K_S    12.57  11.71   Q3_K
Q4_K_M    17.11  15.93   Q4_K
Q5_K_M    19.83  18.47   Q5_K
Q6_K      22.88  21.31   Q6_K
Q8_0      29.05  27.05   Q8_0

You only need to look at the GiB column. If it is smaller than your GPU's memory (VRAM), there is a high probability that the file will fit.

Step 2: Get the launch command

Pass the gguf.json you just created and a part of the filename you want to load.

gguf-plan gguf.json --pick Q5_K_M

You don't need to specify the VRAM capacity. It automatically detects your GPU and calculates accordingly. When I run it on my RTX 5090 (31.4 GiB), it produces this output.

# ===== Qwen3.8-27B-Q5_K_M / ctx 131,072 / KV f16 =====
# estimate: model 18.47 + KV 8.63 + overhead 1.00 = 28.10 GiB / budget 31.4 GiB
# headroom 3.30 GiB
# native ctx = 262,144  / no rope scaling
# hybrid attention: only 17/65 layers hold KV = 68 KB/token
# KV f16 = 69.1 KB/token, measured here (gguf-calibrate), not derived from the GGUF

# --- llama-server ---
# found 4 MTP tensors -> adding --spec-type draft-mtp
llama-server -m /mnt/data/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q5_K_M.gguf \
  --port 8085 --device CUDA0 \
  -ngl 99 -fa on \
  --ctx-size 131072 --parallel 1 \
  --batch-size 2048 --ubatch-size 512 \
  --threads 16 \
  --spec-type draft-mtp

You just need to copy this as-is, paste it into your terminal, and run it. You don't need to understand what '-ngl' or '-fa' stand for one by one. It decides everything, including the ctx number and whether to compress the KV, without even needing to launch the GPU once.

If your GPU is smaller, the ctx-size in the resulting command will be a smaller number. If the file is too large for your VRAM, it won't output a launch command at all and will display the reason instead.

A quick overview of the terms that appear

VRAM: The memory on your GPU. If you have a '24GB' card, it's roughly 24GB. If the model size is larger than this, it won't launch at all.

Quantization: The numbers inside the model have been reduced in precision to make them smaller. The smaller the number, like Q4, the lighter the file, but the coarser the quality.

ctx: The length the model can remember during a single conversation. The longer it is, the more VRAM it consumes.

KV Cache: Memory that increases as you extend the ctx. How much this can be compressed determines how far you can extend the ctx with the same VRAM. This is the part that gguf-plan calculates automatically.

If you want to know more

I wrote about why these numbers appear and how the actual measurements differ from the calculated values in my previous article. It also covers things like how a model with 65 layers only had KV for 17 layers, and a mistake I made in one of my calculations. If you only need to know how to use it, you don't need to read that.

Conclusion

gguf-fit is a tool written in Python, and it is available on GitHub under the MIT license.

https://github.com/zephel01/gguf-fit

If you try it out and the command it generates doesn't work, please let me know. I have only been able to test it on an RTX 5090, RTX 3090, and Radeon 8060S, so I honestly don't know how it will behave in other environments.

#gguffit #GGUF #llamacpp #LocalLLM #Beginner #Quantization #Python #PersonalDevelopment #uv #VRAM #BeginnerGuide