A GGUF of Deepseek v4 flash 0731 284B crushed down to 2bit produced the exact same results as the cloud-based original across all 16 questions

I loaded an 80.76 GiB file onto a mini PC and ran it through 16 questions from my custom benchmark. The content is a 284B total parameter model where only the expert portions have been crushed down to the 2bit level. I also sent the same 16 questions to the original model running on the cloud API.

The judgments matched for all 16 questions. Whether it solved them or failed to solve them, the results were identical, down to the question numbers. However, there was a 2-question difference between this and another GGUF created with the same recipe, and that difference was statistically insignificant.

Last time, I wrote about how benchmarks can easily lie if you don't standardize conditions when swapping backends on the same machine. At that time, I measured that the Resolved rate fluctuates by 5.0 points even in the same environment. I will use that figure as a yardstick to interpret the 12.5-point difference this time.

DeepSeek V4 Flash — Only 13B of the 284B are active

DeepSeek-V4-Flash-0731 was released on July 31, 2026, and the model card states it is the "official release replacing the preview version." 0731 is not a release number but a date snapshot name, and the official API `deepseek-v4-flash` has been replaced with this.

I confirmed the architecture matches via config.json and the GGUF header. 284B total parameters, 13B active, 43 layers. There are 256 routed experts, and only the top 6 are active per token. There is 1 shared expert and a context length of 1,048,576.

"Flash" is the smaller version focused on speed and cost. In the official news on April 24, V4-Pro was announced as 1.6T total / 49B active, and V4-Flash as 284B total / 13B active, presented as models of different sizes. The official benchmark is 82.7 on Terminal Bench 2.1. The license is MIT.

Two points have a subtle impact. The DSpark for speculative decoding is included in the weights. And the official source does not distribute a Jinja-style chat template. The model card instead points to a Python script in the encoding folder, and there is no chat_template key in tokenizer_config.json. Tool calling is in DSML format.

The filename is the specification itself

This is the filename of the GGUF I used this time.

DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

It is long, but once you understand the meaning, you can read the quantization recipe directly. The correspondence was clearly stated in the distribution source's README.

IQ2XXS   ffn_gate_exps / ffn_up_exps  → IQ2_XXS
w2Q2K    ffn_down_exps                → Q2_K
SExpQ8   共有エキスパート ffn_*_shexp  → Q8_0
AProjQ8  attn_q_a/q_b/kv/output_a/_b  → Q8_0
OutQ8    output.weight                → Q8_0
         token_embd / router          → F16
         norm 類                      → F32

IQ2_XXS is part of llama.cpp's I-quant series, a non-linear codebook type based on an importance matrix. In the official quantize README example, IQ2_XXS to IQ2_M is about 2.3–2.9 bpw. This is a model-dependent guideline, not an absolute value. imatrix is a mechanism that measures how much each weight contributes to the output by running calibration text beforehand and determines the distribution of rounding errors based on that importance.

The design philosophy was also in the README. Routed experts account for the majority of parameters, but each expert only handles a portion of tokens. Therefore, the router, projections, and shared experts are kept at Q8_0 to maintain behavior, while the experts are crushed to save size. The mechanism for crushing selectively is also in llama.cpp's quantize, where you can write tensor names as regular expressions in `--tensor-type`. By the way, I could not find any reliable source that proves "what breaks first when an MoE is dropped to the 2bit level."

The two compared versions — the only difference is the scale of imatrix collection

The two local versions I compared are from the repositories of antirez (Salvatore Sanfilippo, the creator of Redis) and Rednalreden (Maurice Hagemeijer, a different person). When I actually read the HF API file tree and the GGUF header, there was almost no difference. The quantization recipe was identical, and the Rednalreden README also explicitly states it is an "established DwarfStar Flash q2-imatrix recipe." The embedded chat template even had an identical md5, byte-for-byte.

The only real difference is the scale of the imatrix collection. The chunks_count is 202,100 for the antirez version versus 117,777 for the Rednalreden version. A difference of about 1.7 times.

The `chat-v2` in the filename is not a DeepSeek official version number, but a unique version number meaning the second edition of the template embedded in the GGUF by antirez. Since the official source does not distribute a template, the distributor wrote their own. The Rednalreden version just lacks chat-v2 in the filename, but the content is identical. The repository name `dwarfstar` is also not a quantization method, but the official name of antirez's custom inference engine, ds4.

I should be honest about one thing. The beginning of the antirez README states "these quantizations are for the DS4 inference engine only," and the Rednalreden Caveats state "this is not a llama.cpp compatible GGUF." Nevertheless, this measurement completed successfully with llama-server from llama.cpp b10157. The GGUF general.architecture is `deepseek4`. I have not confirmed whether llama.cpp added deepseek4 support or if the README is just being too conservative. I am presenting this as a fact.

Results — Cloud and 2bit local gave the same judgment for all 16 questions

I measured 1 trial each for 16 questions from my custom benchmark `llmbench`'s L7 grandmaster. I provide a bug report and source, verify the returned patch with a hidden test, and grade the Resolved judgment and code quality. The local machine is a GMKtec EVO-X2 (Ryzen AI Max+ 395 / Radeon 8060S / integrated memory), running a ROCm build of llama.cpp. Startup is fixed with `-ngl 99` / `n_ctx 16384` / `--parallel 1` / `--batch-size 512` / `--ubatch-size 512` / `--threads 8` / `--no-mmap`.

                       Resolved  Combined  品質平均  tok/s  平均待ち
クラウド 0731           13/16      73.5     80.9     59.2    64.4s
ローカル antirez版      13/16      72.2     77.7     12.6   271.1s
ローカル Rednalreden版  11/16      62.2     81.0     12.0   291.5s

Figure 1 is the judgment map for the 16 questions. The cloud version and the antirez version matched on all 16 questions. Zero discrepancies. Even the three questions that could not be solved, t101, t103, and t105, were the same. Since they failed in all three runs, this is likely a limitation of the task rather than the model.

The 2bit-class version, compressed to 80.76 GiB, returned the same answers as the cloud original for all 16 questions. On the left in Figure 2, the Combined average is 73.5 vs 72.2, a difference of 1.3 points.

A difference of two questions is not a difference.

Only the Rednalreden version failed t064 and t095, resulting in 11/16. A 12.5-point difference. This is where the previous benchmark becomes relevant.

When subjected to McNemar's exact binomial test (two-tailed), the cloud vs. antirez version shows 0 discrepancies, p = 1.00. Cloud vs. Rednalreden and antirez vs. Rednalreden both show 2 discrepancies (2 wins, 0 losses on one side), resulting in p = 0.50. This cannot be called a difference. In short, 12.5 points in 16 questions × 1 trial is just two questions.

Looking at the content makes it even more suspicious. The failure of t095 was `generation error: Read timed out (read timeout=600)`, a timeout of 600 seconds. It didn't give the wrong answer; it just didn't finish in time. t101, which failed in all three runs, also timed out in every execution.

The quality average (for solved questions only) was Rednalreden 81.0 > Cloud 80.9 > antirez 77.7, which is the reverse order of the Resolved rate. The builds that lost actually solved the questions they did solve more cleanly. The effect of the 1.7x difference in imatrix collection scale cannot be read from these 16 questions.

The real problem is the 12 tok/s side.

Figure 3 shows the average wait time per question. Compared to 64 seconds for the cloud, the local versions took 271 and 292 seconds. In tok/s, that is 59.2 vs 12.6 / 12.0, a difference of about 4.7 times. Waiting 4-5 minutes for one question, and over an hour for 16 questions. Even if the capability matches the original, this speed changes how you use it.

The first thing backed by numbers is rocWMMA's Flash Attention. You need to build the ROCm build with `-DGGML_HIP_ROCWMMA_FATTN=ON`. The official AMD blog, 'Trillion-Parameter LLM on an AMD Ryzen AI Max+ Cluster,' contains actual measurements showing generation increased from 3.46 tok/s to 8.30 tok/s at n_ctx=8192. This is primary information.

The main event is likely speculative decoding. Discussion #23659 in llama.cpp reports that Qwen3.5 122B-A10B generation on Strix Halo (gfx1151) with ROCm 7.2.3 increased from 19 tok/s to 26.86 tok/s. This is also primary information within the llama.cpp repository. While Unsloth's documentation generally cites 1.4–2.2x, I am reading this as secondary information. Above all, because V4 Flash includes DSpark in the weights there is no need to look for a separate draft model. I believe this is where the greatest potential for improvement lies.

A subtle pitfall is KV cache quantization. According to Discussion #22411 , symmetric KV cache quantization enables AMD HIP's fast fused Flash Attention path. If you make K and V asymmetric, like `-ctk q4_0 -ctv f16`, it falls back to a slower path without warning. Regarding ubatch, Issue #18725 (Strix Halo, Vulkan) reports that increasing it too much increases CONCAT overhead, with a process that took about 37 seconds at 512 taking about 60 seconds at 2048. The recommendation is 256–512. This time, it was already 512.

Conversely, there are things known not to work. I often see talk about tweaking BIOS VRAM allocation or GTT limits, but the official ROCm 'AMD Strix Halo system optimization' clearly states that 'Strix Halo memory is mapped, not physically partitioned.' It is a matter of whether the capacity fits or not, and it does not affect speed.

Increasing TDP was also almost useless. According to actual measurements in the Strix Halo Wiki (secondary information as it is a community wiki), Gemma 3 27B generation remained almost flat at about 6 tok/s from 55W to 120W. Only prompt processing speed increases. This is likely because generation is bandwidth-limited. I have not found any actual measurements showing `--n-cpu-moe` works on unified memory machines.

I will write that things are not settled when they are not. `GGML_HIP_NO_VMM` is called 'essential for gfx1151' in Discussion #20856 , while another blog says the opposite, claiming 'turning it OFF allowed access to a 96GB GTT pool.' Reports from individual benchmarks on the superiority of ROCm vs. Vulkan are also split, with Vulkan being slightly better for short contexts and ROCm's rocWMMA being superior for long contexts. Since results change based on the backend and version, nothing can be said from a single measurement.

However, there is a post on the Framework community forum claiming up to 32 tok/s (decode) for V4 Flash 284B on a single Strix Halo 128GB. There is no mention of the quantization type, so it cannot be compared directly, but it suggests that 12.6 tok/s is not the upper limit.

Can you call that number a difference?

What remained in my hands was not the bold conclusion that '2bit was enough.' It was the fact that they matched on all 16 questions, the fact that a difference of two questions could not be treated as a difference, and the fact that I had to wait 4-5 minutes.

The most dangerous moment was when I was about to line up 11/16 and 13/16 and write that 'the imatrix collection scale was effective.' p = 0.50. I had measured myself last time that the Resolved rate fluctuates by 5.0 points even in the same environment, yet the moment I saw 12.5 points, which is larger than that, I tried to read meaning into it. When numbers are large, you tend to see a difference.

Then, how many questions × how many trials would be enough to call it a difference? Is comparing the superiority of quantization builds by Resolved rate even suitable in the first place? Once the quality average comes out in reverse order, the conclusion can flip if you only look at one metric. Does your comparison table have the number of trials and statistical tests attached?

If anyone is running V4 Flash on Strix Halo, especially if you've managed to get speculative decoding using DSpark running on the llama.cpp side, please let me know in the comments how much of a speed boost you got. If it stays at 12 tok/s, I don't have the energy to run the next 16 questions.

#LocalLLM #DeepSeek #llamacpp #Quantization #GGUF #StrixHalo #ROCm #MoE #Benchmark #PersonalDevelopment #MachineLearning