Verified the 27B dense model 'Qwable-3.6-27b' from Qwen × Fable across 40 tasks (including MTP comparison) [RTX5090]

Hello, this is CoolZero.

Mia-AiLab released two 27B models around the same time.the MTP-less (standard) version andthe MTP-enabled version. They share the same base and training data, differing only in the presence of MTP layers. I wanted to verify for myself how MTP affects accuracy and speed, so I ran the MTP version with four quantizations (Q4/Q5/Q6/Q8) and the standard version with Q4.

Following the previous 35B MoE, this is the second installment in the Qwable series. Since the architecture changes to a 27B **dense** model, the speed and accuracy characteristics differ significantly.

Model characteristics

Qwable-3.6-27b-MTP (MTP-enabled version)

Qwable-3.6-27b (Standard version, no MTP)

Qwen3.6-27B is a dense model. Unlike the previous 35B MoE, all 27B parameters are used during inference.

GGUF size (measured values):

The MTP version is slightly larger than the standard version (due to the MTP layers).

Verification environment

• Execution environment: llama.cpp (CUDA build)

• Hardware: EVO-X2 + RTX5090 32GB

• Number of tasks: 40 questions (easy 5 / medium 5 / hard 10 / expert 12 / frontier 8)

• Content: Python bug fixing and implementation tasks (function level, with deterministic tests)

• Evaluation metrics:

  • Resolved rate (did pytest pass all tests?)

  • Quality (code quality: ruff/cyclomatic complexity)

  • Combined (overall score of the above)

  • Usability (🟢Autonomous / 🟡Assisted / 🔴Impossible)

Result summary

Conclusion: The MTP version Q5 offers the best balance of speed and accuracy. The standard version Q4 is high-accuracy but slow.

Why MTP is fast: The effect of speculative decoding

This is the most striking figure.

A 2x speed difference despite the same quantization. Models with MTP layers can utilize llama.cpp's speculative decoding feature. The MTP head looks ahead to predict the next token, and the main model verifies and accepts it, increasing effective throughput.

Since the standard version lacks this look-ahead feature, the pure 27B dense sequential generation speed (~74 tok/s) is the upper limit.

This speed difference is also reflected in other quantization comparisons. Even the MTP version Q8 reaches 119 tok/s, which is 60% faster than the standard version Q4 (74 tok/s).

Accuracy reversal: Standard version Q4 > MTP version Q4

While the MTP version is superior in speed, a reversal has occurred in accuracy.

The standard version Q4 outperforms by 7.5% in Resolved and 7.0 points in Combined. The speedup from speculative decoding comes at the cost of accuracy in some tasks in exchange for higher throughput.

Increasing to Q5 in the MTP version recovers accuracy to 95%, so 'MTP version Q5 is the best choice if you want to maintain accuracy while utilizing speed' is the best option.

The mysterious stall of MTP version Q6: Higher quantization does not always mean higher accuracy

Strange results have appeared within the MTP version.

Q6 is 7.5% lower than Q5. Specifically, here is what happened:

t022 (Interval Merge): Q5 solved it at the auxiliary level. Q6 failed to parse due to empty output after timeout.

t025 (CSV Detailed Parse): Q5 passed with auxiliary at quality 72. Q6 failed to parse due to empty output.

t034 (Event Bus): Q5 passed autonomously. Q6 failed to parse due to empty output.

These three tasks failed only in the MTP version Q6. This is a practical example that quantization bit count and accuracy do not increase monotonically, and it is possible that the Q6-specific weight rounding method had a negative impact on specific long-text output patterns.

Usability (How much can you actually rely on it?)

The MTP version Q5 and Q8 are the best with 5% 'fail' (2 tasks). The standard version Q4 has 8% failure (3 tasks); while it is slower, its accuracy is more stable than the MTP version Q4 (15%).

Tasks that failed across all quantizations

t020: Calculator (Order of operations and parenthesis handling)

Failed in all 5 quantizations. It also failed in the previous 35B version, making it a weakness of the entire Qwable series.

t027 (Coin count optimization): Auxiliary judgment in all quantizations (quality 24-32). The test passes, but the implementation quality tends to be low.

t034 (Unsubscribe during event bus dispatch): In the MTP version, only Q5 succeeded; others failed. The standard version also failed. It is the same pattern as the 35B version and is a task that the entire Qwable series struggles with.

Which is stronger: 27B dense or 35B MoE?

Comparison with the previous 35B MoE:

In terms of accuracy, the 27B outperforms the 35B. In terms of speed, the 35B MoE is overwhelmingly faster (230 vs 141 tok/s).

However, there is also a standard Q4 version for comparison.

For accuracy: MTP version Q5
For speed: 35B MoE Q6
For low VRAM/accuracy at lower speeds: Standard version Q4

Summary

• MTP version Q4: Unstable with 15% failure. However, it is the fastest at 146 tok/s.

• MTP version Q5: 95% Resolved, 5% failure, 141 tok/s. The best choice

• MTP version Q6: Lower accuracy than Q5 (12% failure). An example of the non-linear relationship between quantization and accuracy.

• MTP version Q8: Same accuracy as Q5 but 18% slower. Minimal benefits.

• Standard version Q4: 92.5% Resolved, outperforming the MTP version Q4 by 7.5%. However, it is half the speed (74 tok/s).

The greatest benefit of MTP is speed. It achieves nearly double the tok/s compared to the standard version, suggesting that speculative decoding is functioning. However, the MTP version Q4 loses to the standard version in accuracy, so you must increase it to Q5 or higher to ensure accuracy. At Q5 or higher, you achieve a balance of 'MTP speed benefits with sufficient accuracy'.

The code and test results generated during verification have been archived. Please comment if you want to see details on specific tasks or want to try it in your own environment.

Model Information

• Mia-AiLab/Qwable-3.6-27b (HuggingFace)

• Mia-AiLab/Qwable-3.6-27b-MTP (HuggingFace)

• Base: Qwen3.6-27B (Dense model, non-MoE)

• License: MIT

• Recommended sampling (Balanced): temp 0.6 / top_p 0.95 / min_p 0.02

• Recommended sampling (Coding): temp 0.2–0.4 / top_p 0.9

#LocalLLM #Qwen #Fable5 #Quantization #MTP #llamacpp #Benchmark #GGUF #27B