The 1-run champion was not the 5-run champion: The new king of Qwythos-9B-v2 is the MTP version Q6_K

Hello, this is CoolZero.

Last time (v1), when I verified Empero AI's previous work, "Qwythos-9B-Claude-Mythos-5-1M," across 40 tasks, I crowned the MTP version Q8_0 as the champion. However, that article was based entirely on single-run (1run) measurements. One of the backbones of this series is "not to speak based on a single result," but I failed to uphold that principle there.

This time, it is its successor, Qwythos-9B-v2. And this time, with some reflection, I introduced 5-run main verification for the MTP version. As a result, the configuration that was the champion in v1 (Q8_0), while hitting 87.5% in a single run, dropped to 75.0% in the 5-run average. That is a single-run upward deviation of +12.5pt. Instead, the new king is the MTP version Q6_K—an average success rate of 81.0%, the only 5-quantization with a 100.0% pass@5, and zero tasks where all 5 runs failed.

This was a round where the lesson to myself, "don't speak based on a single run," was proven exactly as it is.

Characteristics of this model

GGUF size (HF nominal value. Not measured values):

The MTP version is **approximately +160 to 500MB** higher than the above for each quantization (due to the MTP head).

The "0% loop" and GGUF sizes via FTPO are strictly values posted on the model card. Treat these as separate from this actual measurement (Resolved, success rate, tok/s), and they are not mixed into the score table mentioned later. Since it is a 9B class, it runs on around 8GB of VRAM if using the q4 series.

Verification environment

• Execution environment: llama.cpp (CUDA build)

• Hardware: EVO-X2 + RTX5090 32GB

• Number of tasks: 40 questions (easy 5 / medium 5 / hard 10 / expert 12 / frontier 8)

• Number of executions: MTP version = 2-stage process of "1run screening -> 5runs main verification" / Standard version = 1run only (due to time constraints; treated as reference values)

• Content: Python bug fixing and implementation tasks (function level, with deterministic tests)

• Sampling: Official recommended temp 0.6 / top_p 0.95 (top_k 20 / repeat_penalty 1.05)

• Evaluation axes: Resolved rate / Success rate (5-run average) / pass@k / Quality / Combined / usability

I am using my own SWE-Bench-style benchmark (llmbench). I pass the bug report + source to the LLM, apply the patch, and determine resolution using hidden pytest not disclosed to the LLM. The difficulty tiers are easy / medium / hard / expert / frontier, and task IDs are t001 to t040.

• Combined = success_rate × (0.5 + 0.5 × quality/100) × 100 (non-working code is 0 points)

• Quality is a weighted synthesis of lint density + maintainability metrics + third-party LLM review

• GitHub: `zephel01/swe-bench` (MIT). Reproducible with stdlib-only

Result Summary (MTP version 5-runs verification)

The primary data for reliability is here. For reasons described later, I will also include the median during 1-run as a speed metric.

Conclusion: The MTP version Q6_K is the new champion. It takes the triple crown with an average success rate of 81.0%, pass@5 of 100.0%, and a Combined score of 74.9, and furthermore, it has zero tasks that failed all 5 runs. Q8_0, which was the king in v1, slowed down to an average of 75.0% over 5 runs.

Looking at Quality alone, Q8_0 (86.0) is slightly higher, but Combined is a metric that makes the success rate the main driver, based on the idea that "code that doesn't work is worth 0 points."Rather than the elegance of the written code, whether it passes at all—by that yardstick, Q6_K pulled 5.4 points ahead of Q8_0.

Highlight: v1 champion Q8_0 slowed down over 5 runs

This is the main point of this article. First, I will list the "1-run screening numbers" and "5-runs average" for each quantization of the MTP version.

If I had ended with a single run this time as well, I would have written that Q8_0 (1-run 87.5%) was "the champion again". However, when run 5 times, it sinks to 75.0%. Conversely, Q6_K did not stand out at 75.0% in 1-run, but it grew to 81.0% in the 5-run average. The single-run ranking (Q8_0 > Q6_K) was completely reversed in this verification.

Please pay attention to the magnitude of the difference. The +12.5pt for Q8_0 is clearly an upward fluctuation where it "happened to hit one good run out of five." Conversely, Q6_K and BF16 were actually unlucky in their single runs, and their true strength became visible over 5 runs. I avoid making definitive statements if it is within ±3pt as a range of luck, but +12.5pt is already a different story. This was the pitfall of the "single-run champion certification" that I fell into in v1.

As for the analysis, what I can clearly say is that this model has a large fluctuation in single runs, and the ranking based on 1-run cannot be trusted as is. The lesson is simple—The 1-run champion is not the 5-runs champion.

The difference in "reproducibility" seen in pass@5

As a different yardstick from the average success rate, I look at pass@5 (counted if it passes at least once out of 5 times). Here, the strength of Q6_K becomes even clearer.

Only Q6_K has a pass@5 of 100.0%, which means for all 40 tasks, it solved at least one out of 5 times. Q6_K is the only one with zero completely failed tasks. Other quantizations have some "wall that doesn't pass even if you try 5 times" somewhere (Q4_K_M has 3 such tasks).

However, this needs to be read calmly. pass@5 100% and an average success rate of 81.0% are different yardsticks. The former is "if you are lucky, you can solve any task," and the latter is "what percentage actually passes when you leave it to the model." The average success rate is what matters in actual operation, and please consider pass@5 as a bonus metric that shows the "breadth of potential." Nevertheless, "not having a bottleneck that never passes even once" is a merit that quietly counts for a 9B model.

Q6_K passed the bottleneck t020 only once in 5 times

A regular bottleneck in this series, t020[hard] "calculator ignores precedence and parentheses". This is a task that most models have failed in previous works as well.

The breakdown of t020 in this MTP version 5-runs:

Q6_K passed t020 once out of 5 times. This is a quiet contribution to the pass@5 100%. However, 1/5 is ultimately flaky (run-dependent), and it is not a reproducibility that allows one to say "Q6_K can solve t020." I will not exaggerate. Rather, the fact that even Q6_K fails 4 out of 5 times is the reason why t020 is a bottleneck, and it was alive and well this time, just like in the lineage of the series.

Too many flaky tasks: this much fluctuation at temp 0.6

The most prominent characteristic of this model is that it is extremely flaky (tasks that pass or fail within 5 runs).

For Q8_0, 21 out of 40 tasks—well over half—resulted in inconsistent runs. For reference, here is the breakdown of inconsistent runs for Q6_K and Q8_0:

• Q6_K (18 tasks): t004(3/5), t007(4/5), t010(2/5), t014(4/5), t015(4/5), t017(2/5), t019(4/5), t020(1/5), t021(3/5), t022(4/5), t029(3/5), t030(2/5), t033(2/5), t034(2/5), t035(4/5), t037(1/5), t038(4/5), t039(3/5)

• Q8_0 (21 tasks): t004(4/5), t007(4/5), t008(4/5), t015(4/5), t017(3/5), t020(1/5), t021(1/5), t022(4/5), t023(4/5), t025(3/5), t028(4/5), t029(1/5), t030(1/5), t032(2/5), t033(1/5), t034(4/5), t035(3/5), t036(3/5), t038(3/5), t039(4/5), t037(2/5)

What I want you to note is that this is the result at a relatively low temperature of 0.6. Even when running faithfully to the official recommendations, this is the range of fluctuation. The model card claims that "FTPO improved loops to 0% during low-temperature decoding" (value stated on the model card), but the elimination of loops and the stability of the output are separate issues, and my frank impression from actual measurements is that the reproducibility of solvable tasks is still shaky (I will avoid asserting causality. Confidence: MODERATE).

And this "high degree of flakiness" is the direct reason why single-run measurements are dangerous. If you rank a model by running it only once when half of the tasks change results based on luck, then of course an upward swing of +12.5pt can occur. The explanation that Q8_0 looked like the champion in v1 because it happened to hit the upper end of this fluctuation is the most convincing one.

Success count by difficulty (MTP version 5runs, based on resolved)

I will break down which bands made the difference.

Easy to medium are neck-and-neck across all quantizations (ceiling effect), and the difference is decided almost entirely in the frontier band. Q4_K_M sinks with 2/8 in the frontier, while Q6_K/BF16 are at 5/8, and Q8_0 is at 6/8. Looking only at this, Q8_0 looks good, but Q8_0 collapses in the expert band at 8/12 (Q6_K is 11/12), so the way to read this is that Q6_K is the one where strengths and weaknesses per band are well-balanced. Q8_0 is in a state where it is giving back in the expert band what it earned in the frontier.

Standard version (no MTP) is 1 run only: reference value where assertion is prohibited

Due to time constraints, the standard version could only be run once. Therefore, the following is a reference value, and I will not assert any rankings.

At first glance, Q5_K_M looks the best at 85.0%. However, what I want you to remember here is the fact that in v1 (the previous version), the standard Q5 was actually the lowest (72.5%). The same Q5 can be both the lowest and the highest just by being a single-run measurement—this is a perfect example of the "range of single-run fluctuation." If you look at this immediately after seeing the +12.5pt upward swing in the MTP version, it should be clear that you should not take the rankings of the standard version's 1-run results at face value. The true ranking of the standard version is homework until I can run 5 runs.

How to read tok/s (with important caveats)

First, tok/s is a measure of speed, not a measure of intelligence. In particular, this model has an MTP (Speculative Decoding) enabled version, and MTP structurally produces fast numbers (it is only fast because the look-ahead is correct; it does not improve accuracy). It is natural behavior for the dense standard version to be slow, and slow does not mean inferior.

(All are medians. `~` is an approximation)

I will write this honestly here. The MTP version's "~177-194 at 1run" and "~62-85 at 5runs" are too different in magnitude. The significant drop during 5runs may be due to differences in measurement conditions such as parallel execution, but it cannot be determined from the logs (Confidence: LOW). Therefore, in this article, I adopt the 1run value as the speed indicator, and I will not use the 5runs numbers for speed comparison, noting them as "possible condition differences." I will not line up the numbers and hype it up by saying "It's fast with MTP!"

Reading it with that premise, the MTP version generally provides a boost over the standard version (~177 vs ~103 for Q8_0), which is consistent with the trend seen in v1 that "the higher the precision of the quantization, the greater the MTP multiplier." The fact that only BF16 has sluggish growth at ~111 even in the MTP version is the same quirk as the previous work.

usability (can it actually be entrusted?)

🟢 Autonomous = almost no review required / 🟡 Auxiliary = can be entrusted if review is a prerequisite / 🔴 Impossible = cannot be entrusted.

MTP version (5runs):

The lowest failure rate is shared by Q6_K and BF16 (both at 17%). While Q6_K yields its autonomy rate to Q4_K_M (50%), it has the fewest 'untrustworthy' tasks, which is consistent with the Combined and pass@5 results.

Standard version (1-run, reference value):

The autonomy rates for the standard version appear higher across the board than the MTP version, but this is simply comparing 1-run (a single good run) with 5-runs (an average that smooths out fluctuations) and is not a fair comparison. I will reiterate that they can only be compared on equal footing once the 5-runs are completed.

Comparison with v1 (previous model)

Since this is a sequel, I will list the figures from the previous work. However, I must state beforehand that the previous v1 was measured entirely in single runs, so there are limits to a direct side-by-side comparison.

The MTP-Q8_0 of v1 was 'Resolved 85% / 253 tok/s'. Looking only at the numbers, it appears faster and more accurate than the v2 Q6_K (82.5% / ~188). However, this is a classic trap. The 85% of v1 is a figure that could include the same single-run upward fluctuation that this Q8_0 experienced, and it is not on the same footing as a 5-run average. While there is a difference in tok/s of 253 vs ~188, these are different models and different measurement runs, so it is not a comparison with standardized conditions.

Therefore, I will not write that 'v2 is slower/less accurate than v1'. What can be said is that there is progress in measurement design: 'v1 was a single run and cannot be directly compared, whereas v2 has been verified with 5-runs for the first time'. The takeaway this time is that we were finally able to measure with a reliable yardstick, rather than the superiority of the numbers.

Summary: Which quantization to choose

• If you prioritize accuracy and stability above all else → MTP version Q6_K. Average success rate 81.0%, pass@5 100.0%, Combined 74.9, no tasks failed all 5 times. The overall champion this time. If in doubt, choose this.

• If you want it to run as fast as possible / save VRAM → MTP version Q4_K_M. Fastest at ~194 tok/s in 1-run, with a success rate of 75.5%, which is within the practical range. However, this is only if you can tolerate the fact that there are 3 tasks that failed all 5 times.

• There is no reason to choose Q8_0 this time. It shines in 1-run, but has a 5-run average of 75.0% and is the most unstable with 21 flaky tasks. The v1 champion has abdicated.

• BF16 is not bad, but it is a lower-tier version of Q6_K. It puts up a good fight at 80.5%, but its size is 17.92GB (nominal), more than double that of Q6_K. It is also slow at ~111 tok/s. There is little reason to choose it actively.

• No comment on the standard version for now. Since it is only 1-run, I will not rank it. Q5_K_M looks like 85.0%, but there is a precedent where the same Q5 was the worst in v1, so I will treat this as a single-run fluctuation. 5-runs are homework.

Finally, two lessons to take away from this round.

1. The 1-run champion is not the 5-run champion. The author, who made Q8_0 the king in v1, witnessed a +12.5pt upward fluctuation for the same Q8_0 in v2. Single runs lie.

2. pass@5 100% and average success rate are different yardsticks. The breadth of 'every task can be solved at least once in 5 tries' is different from the ability of 'what percentage passes when left to run'. Q6_K is the king because it won in both, but if it were only one, it could not be called the king.

It is solid for the 9B class and is also a manageable size. If you have VRAM to spare, please start with the MTP version Q6_K, and if you want to save, start with the MTP version Q4_K_M.

The code and test results generated during verification have been archived. Please comment if you want to see details of a specific task or want to try it in your own environment.

Model Information

• HuggingFace: https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF

• Base: `Qwen/Qwen3.5-9B` (dense 9B) / Successor to the previous work Qwythos-9B-Claude-Mythos-5-1M

• License: Apache-2.0

• Recommended sampling: temp 0.6 / top_p 0.95 / top_k 20 / repeat_penalty 1.05

• MTP: Supported (`nextn_predict_layers=1`). GGUF files available for both standard and MTP versions

Related articles

• Previous (v1): "Verified the Qwen3.5-based 9B inference model 'Qwythos-9B-Claude-Mythos-5-1M' across 40 tasks (including MTP comparison)"

#LocalLLM #llamacpp #benchmark #quantization #GGUF #Qwythos #9B #MTP #passk #Qwen3