I pitted the 9B and 4B versions of DeepSeek-V4-Pro-Qwen3.5 against each other across 40 tasks x 5 runs x 4 quantizations: The difference with double the parameters was only 3.1pt Combined

Hello, this is CoolZero.

On August 1st, Jackrong released two distilled models from DeepSeek-V4-Pro. Both 9B and 4B are MTP-version GGUFs. Since it is rare to see sibling models from the same pipeline with exactly double the parameter count released on the same day, I thought this was the perfect material to verify whether you can 'drop the 9B and get by with the 4B'.

I will state the conclusion first.In my actual measurements, comparing the best configurations, 9B-Q4_K_M scored 71.5 Combined, while 4B-Q5_K_M scored 68.4. The difference was only 3.1pt.As for the Resolved rate, 4B-Q8_0 reached 80.0% (32/40), perfectly tying with the best of the 9B. However, in terms of average success rate across 5 runs, the 9B-Q4_K_M achieved 78.0% compared to 74.5% for the best 4B, meaning the 9B was superior in stability.

And once again this time,it did not turn out that increasing the quantization bits leads to better results. The 9B dropped to 65.0% Resolved at Q6_K, while the 4B conversely fell to 62.5% at Q4_K_M. Despite being siblings from the same training flow, the shapes of their quantization curves were polar opposites.

Characteristics of these models

The nominal benchmark values are as follows.These are all values posted on the model cards, not my own measurements. Please read them separately from the actual measurement tables that appear later.

The 9B card also makes the claim of '-36.1% tokens per correct answer, +0.93pt accuracy' compared to Qwen3.5-9B.I will cross-reference how this token-saving claim held up in my actual measurements later in the speed section.

GGUF file sizes (HF posted values).

Note that while both repositories include 'mmproj-F32.gguf' (multimodal projector), only the 4B has a pipeline_tag of 'image-text-to-text', while the 9B is 'text-generation'. The handling of vision support is not consistent between the repositories. Since this is a pure code generation task, I did not use the projector.

Nothing is written on the card about MTP

Even though they call themselves 'MTP-GGUF',neither card provides any explanation on how to execute MTP, the presence of draft files, or the recommended number of drafts. The 4B card only contains a description of the evaluation conditions: 'MTP-enabled Q8 GGUF build using two draft tokens under llama.cpp speculative decoding'.

Looking at the file list, there are no independent draft files like 'mtp-*.gguf' in either repository, so I interpret it as a configuration where the MTP layer is bundled into the main GGUF (no confirmation from the card; confidence level MODERATE). This time, I launched it with '--spec-type draft-mtp', and it worked normally.

Verification environment

• Execution environment: llama.cpp (CUDA build, b10157-c6292cfb8)

• Hardware: EVO-X2 (AMD RYZEN AI MAX+ 395) + RTX5090 32GB

• Number of tasks: 40 questions (easy 5 / medium 5 / hard 10 / expert 12 / frontier 8)

• Number of runs: 5 times per task (measuring pass@k and success rate)

• Content: Python bug fixing and implementation tasks (function level, with deterministic tests)

• Sampling: Since there are no recommended values in the model card, I used the default values for llmbench (sample_temp 0.8 / max_tokens 24576 for multiple trials). top_p and top_k were not specified, using llama-server defaults

• Context length: 16,384 tok / parallel slots 1 / '-ngl 99' / batch 512 / threads 8

• Speculative Decoding: `--spec-type draft-mtp`

• Evaluation Metrics: Resolved Rate / Success Rate (5-run average) / pass@5 / Quality / Combined / Usability

I am using my own SWE-Bench-style benchmark (llmbench). I pass the bug report and source code to the LLM, apply the patch, and then use hidden pytest cases private to the LLM to determine if it is resolved. The difficulty tiers are easy / medium / hard / expert / frontier, and task IDs range from t001 to t040.

Combined = success_rate ×(0.5 + 0.5 × quality/100)× 100 (non-functional code scores 0). Quality is a weighted synthesis of lint density and maintainability metrics. GitHub: `zephel01/swe-bench` (MIT), reproducible with stdlib-only.

First, a disclaimer about the mismatch in context

Both models explicitly state on their cards that they do not include coding in their training data. We are throwing models tuned for math/STEM reasoning at a Python bug-fixing benchmark. The fact that they perform lower than coding-specialized models is a design choice, not a reflection of model quality. I want to state upfront that the numbers in this article are empirical measurements of how well math-distilled models can handle code fixes, and they do not invalidate the GSM8K/MMLU-Pro results on the model cards.

Results Summary

All 8 configurations were measured across 40 tasks x 5 runs, for a total of 1,600 trials.

tok/s is the median value. VRAM is the measured value for the llama-server process.

Conclusion: The overall champion is 9B-Q4_K_M (Combined 71.5). However, 4B-Q5_K_M is close behind at 68.4, a difference of only 3.1pt, while requiring only 4.4GB of VRAM. If you cannot fit the 9B model, 4B-Q5_K_M is practically sufficient for almost all use cases.

Looking only at the Quality column, 4B-Q4_K_M is the highest at 85.2, but this is the average of only the resolved tasks. Since it is an average of only the 25 tasks that passed, it appears higher because the difficult tasks were removed from the denominator. You should not compare models based on Quality alone.

9B vs 4B: Where did the 2x parameter difference matter?

Looking only at the summary, it seems '4B is enough,' but when lined up by task, the source of the difference becomes clear. I extracted only the tasks where the difference in pass counts was 4 or more between the 9B 4-quantization runs (20 trials total) and the 4B 4-quantization runs (20 trials total).

9B clearly wins in tasks that require 'reading the specification accurately and catching exceptions not explicitly written.' RFC4180 escape rules (t025), semver pre-release ordering (t023), right-associativity of exponentiation (t032), and inclusive vs. exclusive boundaries (t039). None of these can be solved without 'knowledge of the specification.' The 2x parameter count was effective here (Inference of causality, confidence MODERATE).

Conversely, the tasks where 4B wins are mostly straightforward implementation bug fixes. Misidentified return values (t012), state sharing bugs (t003), and counterexamples for greedy algorithms (t027). Here, the 9B advantage disappears.

The reversal in t007: 9B got 3/20, 4B got 13/20

The strangest one is t007. A medium-tier task where 'word_frequencies counts Word and word as different words.' 9B only passed this 3 out of 20 trials, while 4B passed 13 times. In fact, 9B's Q6_K and Q8_0 versions scored 0/5—they never passed once.

However, when opening the generated output, the 9B-Q6_K and 9B-Q8_0 t007 outputs failed for reasons unrelated to logic. The indentation was crushed to 1 space, and the block directly under the `for` loop was output without indentation, causing an IndentationError. pytest stopped at the collection stage. The logic itself was valid—using `re.findall(r'\b\w+\b', text)` to pick up words and lowercase them—and it is highly likely it would have passed if the indentation were correct (confidence MODERATE).

The 9B-Q4_K_M / Q5_K_M and the 4B configurations all used valid implementations that lowercase words after removing punctuation, and all saved trials passed 4 tests.The low performance of 9B on t007, at least as far as the saved trials show, is not a matter of 'intelligence.'

The true nature of the 9B-Q6_K collapse: Indentation collapsing to a single space

The phenomenon I found in t007 was not limited to t007. llmbench saves the generated code for one trial of each task in artifacts. This is the result of statically checking 41 files for each of the 8 configurations.

The 9B-Q6_K, which is at the bottom of the Combined rankings, had the most at 8 cases. The 9B-Q5_K_M and 4B-Q8_0, which are at the top of the Combined rankings, had zero. It corresponds perfectly with the rankings.

The reason 9B-Q4_K_M failed t024 (TTL entry lives slightly too long) 0/5 times was the same. I was puzzled because Q4_K_M was the only one failing completely on a task that the other three quantizations passed 3-5/5 times, but the saved output was a syntax error.

The score is mixed with cases where it didn't 'fail because it was too hard' but 'could not output the code it wrote in the correct format'. Since llmbench applies patches and runs pytest, if the indentation is broken, it gets 0 points regardless of how correct the content is.

Three caveats. Since artifacts only save one trial per task, it cannot be said to represent the cause of all 5 trials. Also, whether this collapse originates from the quantization itself or is an accidental occurrence under a sampling temperature of 0.8 cannot be separated with the current data. And since there are 14 failed questions for 9B-Q6_K, the 5 syntax errors alone cannot fully explain the bottom ranking.I will make this a homework assignment for a follow-up test with a different temperature.

Reversal phenomenon: 9B collapses at Q6_K, 4B collapses at Q4_K_M

This is another highlight of this article. The shape of the quantization curve was the exact opposite for 9B and 4B.

For 9B, Q4_K_M is the best, it drops as the bits increase, hits bottom at Q6_K, and then recovers slightly at Q8_0. For 4B, Q4_K_M is the worst, it jumps at Q5_K_M, and then stays high.

The drop in 9B-Q6_K is not small. With a Resolved rate of 65.0% (26/40), a success rate of 65.0%, and a Combined score of 59.5, it is the lowest among the 8 configurations. It failed 6 more questions than Q4_K_M. The 8 questions that only Q6_K newly failed are t003 (easy), t010, t021, t022, t027, t030, t035, and t036, which are scattered regardless of the bandwidth (instead, t024 and t029 recovered). It is symbolic that t003 in the easy band dropped from 4/5 to 2/5.

The 4B side's Q4_K_M also collapses across all bands. Medium was 2/5, expert was 8/12, and frontier was 3/8. Moreover, the total number of generated tokens was 136,073, the highest among the 8 configurations (9B-Q4_K_M was 82,684). It was the least cost-effective configuration, outputting 1.6 times the tokens and ending up with the lowest score.

Since these are 5-run averages, the difference between 9B's Q4 and Q6 (13.0pt in average success rate) cannot be explained by single-run luck. Part of the cause was found in the next chapter, but that alone could not explain everything.

What I can say is an operational conclusion. In this model family, 'just choosing high bits' is a losing strategy. For 9B, Q4_K_M, and for 4B, Q5_K_M. In each case, being closer to the minimum size was best.

Failed tasks by quantization

I will list tasks that did not pass the majority (3/5 or more) as resolved=False.

The 3 questions that failed in all 8 configurations were t020, t033, and t037. For t034, the only one that could get resolved was 4B-Q8_0 (3/5), and the remaining 7 configurations were resolved=False. Out of 40 trials, the number that passed, even adding one each from 9B-Q4_K_M and 4B-Q5_K_M, is 5.

Looking at the breakdown of failures, out of 1,600 trials, 443 were 'test failures', 21 were 'patch parsing failures' (breakdown: 20 file block detection failures, 1 invalid path), and 2 were timeouts. It is not dying due to empty output. The 4B variant of Agents-A1 that I verified the other day was failing 14-18 questions due to the phenomenon of hitting the output limit of 24,576 tokens and swinging and missing, but this 4B does not have that, and it is simply failing the tests. Even though '4B is low' is the same, the content is completely different.

Genealogy of the difficult tasks: t020 only passed 2 out of 40 trials

t020 'Calculator ignores operator precedence and parentheses' is a difficult task throughout this series. In a past article, I wrote that it 'failed in 9/10 models, a difficult task for this entire series'. That evaluation remains unchanged this time.

Out of 8 configurations × 5 runs = 40 trials, t020 only passed twice(once each for 4B-Q6_K and 4B-Q8_0). The 9B model failed all 20 trials across 4 quantizations.

I will list the tasks that have similarly been failing for a long time.

On the other hand, t019 (where the trie treats prefixes as registered words) was a stable task in past articles where "all models passed 10/10," but this time, the 9B model passed only 7 out of 20 trials, and the 4B model passed only 10. Whether the number of difficult tasks has increased or if this is a hole specific to this model family requires follow-up testing with other models (Confidence: LOW).

Conversely, there were only 3 tasks that passed in all 8 configurations × 5 runs = 40 trials: t001, t009, and t028.

Breakdown by Difficulty

This is the count of resolved (passed by a majority) tasks.

The breakdown becomes a bit clearer when looking at the average success rate (%).

The 4B model is stronger in the easy band (Q5_K_M and Q6_K are 100%). Due to the ceiling effect, the easy band no longer provides information. The difference appears in the expert and frontier bands, where 9B-Q4_K_M is the best at 77 / 65. The parameter advantage of the 9B model is clearly showing in the higher bands.

Since there is little point in running the 5 easy tasks, I will run follow-up tests for this family using a 60-task version, removing the easy tasks and adding 20 architect tasks. The ranking of the higher bands should become a bit clearer.

It was surprising that 4B-Q6_K topped all configurations at 88% in the hard band. However, since the same Q6_K is tied for last at 38% in the frontier band, I suspect the 88% in the hard band was likely helped by the placement of tasks that it happens to be good at within that band (Confidence: MODERATE).

Difference in "Reproducibility" as seen in pass@5

Looking at the pass@5 (≥1 success) column, which asks "did it pass at least once in 5 tries?", the range across the 8 configurations is only 87.5% to 95.0%. If you allow retries, any configuration can reach roughly the same level. The difference lies in the "probability of passing on the first try."

This "difference" column is directly the dependency on retries. 4B-Q4_K_M is 27.5pt. Even though it only hits 60% of the time if you throw it once, it reaches 90% if you throw it 5 times. In terms of operation, it can only be used on the premise that you "always run it to verify." 9B-Q4_K_M's 17.0pt is the lowest, making it the most resilient to a single-shot attempt.

Note that since I am running it 5 times at a sampling temperature of 0.8, most of this fluctuation is due to the temperature. If you lower the temperature, the success rate and pass@5 will converge, but you will lose the "lucky hits" where it only passes once out of 5 tries. For single-shot use cases, it is worth lowering the temperature to increase the success rate.

Usability: How much can you actually rely on it?

🟢 Autonomous = review almost unnecessary, 🟡 Auxiliary = can be entrusted if review is a prerequisite, 🔴 Impossible = cannot be entrusted.

It aligns almost perfectly with the Combined ranking, but there is one difference in how to read 4B-Q8_0. 🔴 Impossible is 8/40, the same number as 9B-Q4_K_M, meaning the "area where it cannot be entrusted at all" is the same size as the 9B's best. However, it only has 13 🟢 Autonomous, and 19 🟡 Auxiliary, which is the highest. It has a wide range of coverage if you run reviews, but it is a configuration where there is little work you can leave alone.

Practically speaking, the fact that the top 4 configurations share "8 to 9 out of 40 tasks are 🔴 Impossible" is the line for "review is a prerequisite if you put it in regular use." Even with 9B-Q4_K_M's 42% 🟢 Autonomous, humans still need to look at more than half of the work.

Speed: 4B is about 1.3x faster than 9B. However, discount the MTP boost.

tok/s is the median value for single-run execution (1 parallel slot). Since the tok/s per stream changes by an order of magnitude during parallel execution, do not use these numbers for estimating parallel environments.

Three points to note on how to read this.

1. All 8 configurations have MTP (Speculative Decoding) enabled. Models with MTP structurally produce faster numbers. Over 400 tok/s is a difference in implementation, not 'intelligence'.

2. It is not an MoE but a hybrid configuration of the Qwen3.5 lineage, so it is a different kind of boost from the 'fast because activation parameters are small' bias of MoE.

3. Since the standard dense version was not measured under the same conditions, how much MTP is contributing cannot be isolated in this data. I will re-test once the standard GGUF is released.

And,fast tok/s and finishing the job quickly are different things. 4B-Q4_K_M has 420 tok/s, which is 1.28x that of 9B-Q4_K_M, but the average latency is 9.6 seconds vs 8.3 seconds, meaning4B is slower. This is because it is outputting 1.6x more tokens.

Returning to the model card's claims: The 9B card claimed a '-36.1% tokens per correct answer compared to Qwen3.5-9B'. **In this actual measurement, at least compared to its sibling 4B, the 9B is clearly more token-efficient.** 9B-Q4_K_M solves more tasks using 61% of the tokens of 4B-Q4_K_M. The comparison target for the token-saving claim is Qwen3.5-9B, not 4B, so this is not a direct verification, but the direction does not contradict the card's claims.

Summary: Which one to choose in the end

• 9B Q4_K_M: Resolved 80.0% (32/40), success rate 78.0%, Combined 71.5, VRAM 6.4GB, ~327 tok/s. Top or tied for top in all metrics.The first choice if using this family. Fitting into 8GB VRAM is also a plus.

• 4B Q5_K_M: Resolved 77.5%, success rate 74.5%, Combined 68.4, VRAM 4.4GB, ~425 tok/s.The difference from 9B is 3.1pt in Combined. VRAM is 2GB less, and speed is 1.3x. Choose this for 6GB class GPUs or if you want to keep it resident.

• 4B Q8_0: Resolved is 80.0%, tied with 9B's best, but inferior in reproducibility with a 71.0% success rate. 🟡Assistance was the highest at 19/40.There is a reason to choose it if the operation assumes review, but if you are paying 5.8GB of VRAM, 4B-Q5_K_M is sufficient.

• 9B Q6_K: Combined 59.5, the lowest among the 8 configurations.I cannot find a reason to choose it.

• 4B Q4_K_M: Combined 57.8, highest token consumption. Speed is also not the fastest within 4B. No situation to actively choose it.

I will write down three lessons from this time.

First,even with sibling models from the same training flow, the optimal point for quantization is not inherited. 9B is Q4_K_M, 4B is Q5_K_M. Generalizations at the model family level, such as 'Q5 is the ironclad choice for this series', did not hold true for at least this family. You have to re-measure for each size.

Second,2x parameters only work in the higher bands. The easy band is better for 4B (100% vs 92%), and the difference shows up in expert and frontier. Moreover, it was concentrated in tasks requiring 'specifications as knowledge', such as RFC4180 escape rules or semver pre-release order. It is faster to calculate model selection backwards from the nature of the task rather than the number of parameters.

Third.Part of the score was not due to "intelligence," but to broken output formatting. In 8 out of 41 saved files, 9B-Q6_K had its indentation crushed to a single space, and 5 files scored 0 due to syntax errors. Even if the content is correct, pytest will not pass if the indentation is broken.When looking at benchmark scores, you will misjudge a model's true capability unless you count "could not solve" and "could not output" separately.

One last thing.If your environment can handle 9B, choose Q4_K_M (Combined 71.5); if it cannot, 4B-Q5_K_M (68.4, VRAM 4.4GB) is more than capable of competing. I found no reason to choose higher bits for this family.

The code and test results generated during verification have been archived. Please leave a comment if you want to see details on a specific task or want to try it in your own environment.

Model Information

• HuggingFace(9B): https://huggingface.co/Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF

• HuggingFace(4B): https://huggingface.co/Jackrong/DeepSeek-V4-Pro-Qwen3.5-4B-MTP-GGUF

• Base: Qwen3.5-9B / Qwen3.5-4B (architecture `qwen35`, total parameters 9.2B / 4.3B)

• Author: Jackrong. Distilled SFT from DeepSeek-V4-Pro (Max Effect) + RL via GSPO

• License: Apache-2.0 (both models)

• Recommended sampling: Not specified in model card(Used llmbench default sample_temp 0.8 for this test)

• Benchmark: `zephel01/swe-bench` (MIT)

Related Articles

• Agents-A1: Standard Q4_K_M is 92.5% in both runs, 4B variant dies with 14-18 out of 40 questions as "empty output" (2026-07-28)

• The 1-run champion was not the 5-run champion: Qwythos-9B-v2's new king is the MTP version Q6_K (2026-07-11)

• July 2026 Local LLM Benchmark Summary (2026-07-25)

• CoolZero's Article List: https://note.com/zephel01

#LocalLLM #llamacpp #Benchmark #Quantization #GGUF #DeepSeekV4Pro #Qwen35 #MTP #9B #4B #passk #CodingModel