Resolved 90% but Q6_K is at the bottom: Verifying 4 types of Qwimi-3.6-27B-Coder-MTP quantization across 60 tasks (Q4/Q5/Q6 5runs)

Hello, this is CoolZero.

The subject this time is `trjxter/Qwimi-3.6-27B-Coder-MTP-GGUF`. It is a model distributed by individual developer trjxter (Rishi), who fine-tuned Qwen3.6-27B for coders using QLoRA and added MTP (Multi-Token Prediction = speculative decoding). The model card lists impressive figures like "+4pt over base in coding" and "processing time nearly halved," but these are the author's own benchmarks and proprietary scoring, with almost no third-party SWE-Bench-style verification to be found. So, I decided to put it to the test with my own llmbench.

I will give you the conclusion first.The main workhorse is Q5_K_M (19.5GB), which is confirmed as the overall champion with a 91.7% Resolved rate. And the main point of this article is not the quantization ranking itself, but the fact that the ranking from the 1-run screening was overturned in the 5-run test. Q6_K, which was at the bottom in the 1-run test, rose to 2nd place in the 5-run test, while Q4_K_M, which appeared to be in the middle, fell to the bottom. "Do not judge by a single result"—this has become the ultimate proof of the principle I have repeated throughout this series.

The nature of this model

GGUF size (decimal GB from HF measured bytes):

There are three points I want to clarify first regarding its nature.

First, the nature of the nominal values. The model card claims "coding +4.00pt in self-bench, HumanEval 80.00% -> 94.29%," but these are the author's own benchmark and scoring values (values listed on the model card, excluding SWE-Bench and LiveCodeBench). Moreover, the card itself is honest, admitting that performance in agent-based tasks is -4.00pt worse than the base. These nominal values remain just background information and are not mixed into the actual measurement tables at all.

Second, the context length. Qwimi's fine-tuning has only been trained and verified up to 16,384 tokens (the base supports 260,000 tokens). The GGUF card also explicitly states, "`--ctx-size` is set to a maximum of 16384. Anything beyond that is unverified." Be careful if you use it for long texts.

Third, the lack of third-party verification. There are 0 HF Discussions for both the GGUF and BF16 versions, and about 3,000 downloads. There is no mention on Reddit or X. As far as I can confirm, this article is almost the first attempt at third-party verification. That is precisely why I am treating the numbers with caution.

Verification environment

• Execution environment: llama.cpp (CUDA build)

• Hardware: EVO-X2 + RTX5090 32GB

• Number of tasks: 60 questions (easy 5 / medium 5 / hard 10 / expert 12 / frontier 8 / architect 20)

• Number of executions: Stage 1 = 1 time per task (screening) / Stage 2 = 5 times per task (measuring pass@k and success rate)

• Content: Python bug fixing and implementation tasks (function level, with deterministic tests)

• Sampling: (Under confirmation)

• Evaluation axes: Resolved rate / Success rate (5-run average) / pass@k / Quality / Combined / usability

llmbench is a self-made benchmark similar to SWE-Bench. It passes bug reports and source code, applies patches, and determines resolution using hidden pytest (not disclosed to the model). Combined is `success_rate ×(0.5 + 0.5 × quality/100)× 100`, and non-working code gets 0 points. GitHub is `zephel01/swe-bench` (MIT).

This time, I adopted a two-stage verification design. I ran 4 quantizations in 1 run to get an overview, and then conducted the main verification for 3 quantizations (Q4_K_M/Q5_K_M/Q6_K) with 60 tasks × 5 runs each (Q8_0 was not tested for 5 runs. I excluded it at my discretion because its speed is half that of the others and its 29GB VRAM occupancy puts pressure on the KV cache).

Stage 1: 4 quantizations × 1-run screening

Conclusion: The ranking at the 1-run mark is Q5 (90.0%) > Q8 (88.3%) > Q4 (86.7%) > Q6 (85.0%). Q6_K is in last place, and it was the only one in the medium band to produce a failure (t007).

At this point, I was about to write, 'It is strange that Q6_K is in last place. It is a reversal phenomenon where it is lower than Q4 despite having more bits.' However, fluctuations of ±3pt are common in a single run. So I proceeded to the second stage—and it was the right decision to do so.

Second stage: Main verification of Q4/Q5/Q6 with 60 tasks × 5 runs

Conclusion: Q5_K_M is the overall champion with 91.7% Resolved, 87.7% success rate, and a Combined score of 80.0. Second place is Q6_K, which was last in the 1-run test. Q4_K_M, which was in the middle in the 1-run test, fell to last place.

The highlight of this article: The 1-run ranking was overturned in 5 runs

It is striking when you line them up.

• 1-run ranking: Q5 > Q8 > Q4 > Q6 (last place)

• 5-runs ranking: Q5 > Q6 (rose to 2nd place) > Q4 (fell to last place)

Q6_K was in last place in the 1-run test, even committing the 'only failure in the medium band (t007).' However, in 5 runs, its success rate was 87.0%, closing in to within 0.7pt of Q5 (87.7%). t007 also passed 3/5 times in the 5-run test, so that failure was just a downward fluctuation. Conversely, Q4_K_M looked like it was in the middle in the 1-run test, but in the 5-run test, it fell about 5pt behind Q5/Q6 in success rate and had the most 🔴 failures with 11 tasks.

To be honest, at the first stage, I was about to write the conclusion that 'there is no reason to choose Q6' and 'Q4 is sufficient for practical use.' Both were based on the 1-run results, and both were overturned in the 5-run test. Single-shot screening is a tool for deciding candidates to drop, but it is not a tool for determining rankings. This is a demonstration that includes self-admonition, as the series itself, which advocates 'not speaking based on a single result,' almost spoke based on 1 run.

Q5 vs Q6: The distance between the champion and the group that restored its honor

Resolved is 91.7% vs 88.3%, a 3.4pt difference, but the average success rate is 87.7% vs 87.0%, a 0.7pt difference. Since it is within the ±3pt range, the honest reading is that 'they are almost tied in success rate.' Combined is also almost the same at 80.0 vs 79.8. Quality was actually highest for Q6 at 83.5 among the 3 quantizations (the difference is small).

Even so, I recommend Q5 as the main choice because the details consistently lean toward Q5. pass@k is 96.7% vs 95.0%, and 🔴 failures are 5 vs 7, with Q5 being the lowest. By difficulty, Q5 is 12/12 vs Q6 11/12 for expert, and Q5 8/8 vs Q6 7/8 for frontier. All are close, but Q5 is higher or tied in all of them. Although Q6 cleared its name and is almost tied in accuracy, there is no added value to justify the +3GB of VRAM and the speed decrease.

5-runs breakdown of usability (🟢Autonomous/🟡Assisted/🔴Failure):

In terms of the number of 🟢, Q4/Q6 are the highest at 35, but Q4 also has the most 🔴 at 11. Q5 has a modest 32 🟢, but only 5 tasks that are 'completely unmanageable.' If you want to minimize the number of tasks that are out of hand, Q5 is the choice.

Breaking through the difficult tasks: A breakthrough in t020, the only complete wall is t059

The common failures in the 3 re-run tests are t044 (plugin load order/circular detection), t046 (DI container lifetime), t047 (handling of subsequent tasks after a failed migration), and t059 (compensated summation of floating-point sums). However, there are nuances when looking at the content.

• 0/5 for all quantizations; the only complete wall is t059.

• t046 had a run where only Q4 passed 1/5, and t047 was 1/5 for Q4/Q5. They pass occasionally but are not stable.

• t044 passed and failed in the old measurements, but this time it was treated as a failure for all 3 quantizations at 1-2/5.

And one piece of news.Q6_K broke through the series-regular obstacle t020 (calculator operator precedence and parenthesis handling) in 3 out of 5 runs. A task that has consistently failed in past local LLM verifications has been (partially) broken for the first time. Q5 was 1/5 and Q4 was 0/5, so quantization choice matters, and 3/5 is still in the realm of flaky, but it is no longer an "impassable wall."

Correction of previous assessment: The "mystery of t021" did not reproduce

In the old measurements, there was a strange phenomenon where only Q5_K_M kept failing t021 (banker's rounding) in the expert band, and I had formed a "quantization noise hit-or-miss" hypothesis (confidence level LOW to MODERATE).

As a result of re-testing, this theory has disappeared. Q5 now passes in 3/5, while conversely, Q4 fails in 2/5, and Q6 passes in 4/5. The idea that "only Q5 is weak against t021" did not reproduce, so I am withdrawing my previous assessment.

The lesson learned is this: The compatibility between quantization and tasks fluctuates not only between quantizations but also between runs of the same quantization. Even 5 runs may not be enough to say that a specific quantization is weak against a specific task. I will handle causal stories at the single-task level with even greater caution in this series from now on.

tok/s: Stable in the re-test

In the initial measurements, the median tok/s fluctuated significantly depending on the measurement session, making it impossible to determine a speed ranking. In the 3 sets of 5 runs for the re-test, it is straightforwardly stable.

• Q4_K_M ~138.0 / Q5_K_M ~133.5 / Q6_K ~123.6 tok/s (median, consistent in the 120-140 range for all 3 sets)

This is a logical ranking by size. As a note of caution, models equipped with MTP (Speculative Decoding) structurally produce faster numbers, which is not proof of intelligence. With that in mind, for reference, my past actual measurements (non-MTP Qwen3.6-27B base model, same hardware, llama-bench tg256) were Q4 ~75 / Q5 ~68 / Q6 ~60 tok/s. The MTP version is about twice as fast. Since the measurement methods differ, these are reference values, but it is safe to say that MTP acceleration is effective. Q8_0 was consistently slow at ~48.8 in the first stage (5 runs not performed). These are single-stream tok/s; if run in parallel, they will drop further.

Summary

Here is the conclusion from hitting Qwimi-3.6-27B-Coder-MTP quantization with 60 tasks × 5 runs (Q4/Q5/Q6), stated in one line per quantization.

• Q5_K_M (19.54GB) = Confirmed as the primary choice. Resolved 91.7% - Overall top in success rate, Combined, and 🔴 minimum failures. If in doubt, choose this.

• Q6_K (22.43GB) = Reputation restored, but rarely needed. The success rate caught up to within 0.7pt of Q5, but there is no added value to justify the +3GB and speed reduction.

• Q4_K_M (16.81GB) = Underperformed this time. Success rate 82.7% - fell behind with 🔴11. A compromise option when VRAM is truly tight (pass@k 96.7% is tied with Q5, so it is reachable if retries are assumed).

• Q8_0 (29.05GB) = 5 runs not performed. It was 88.3% in 1 run, but the speed was ~49 tok/s, about 1/3 of the others. Again, I found no reason to choose this.

The biggest lesson this time was about verification design rather than the merits of quantization itself. The ranking of 1 run can be overturned by 5 runs. Both Q6's bottom ranking and Q4's sufficient practicality were illusions shown by a single run. Screening is just the entrance, and rankings should only be discussed after repeating the tests—I confirmed this principle of the series by having my own draft manuscript overturned.

The issue of all quantizations consistently failing t059 remains as homework.

The code and test results generated during verification have been archived. If you want to see details of a specific task or try it in your own environment, please leave a comment.

Model Information

• HuggingFace: huggingface.co/trjxter/Qwimi-3.6-27B-Coder-MTP-GGUF

• Base: Qwen/Qwen3.6-27B (dense, 27.8B). QLoRA SFT + MTP version by author trjxter (not official Qwen)

• License: Apache 2.0

• Recommended sampling: Base model Qwen3.6 official coding recommendations temp 0.6 / top_p 0.95 (top_k 20 / min_p 0.0). Since there is no unique recommendation table on the Qwimi card, the base model's official recommendations are adopted

• Note: Fine-tuning has only been verified up to 16K context (maximum 16384 recommended for `--ctx-size`)

Related Articles

• [RTX 5090 Measured] Comparing Qwen 9B, 27B, and 35B MoE across all quantizations | Local LLM Speed Guide 2026 (note article link)

#LocalLLM #llamacpp #benchmark #quantization #GGUF #Qwimi #Qwen3.6 #27B #MTP #RTX5090