When I ran Qwen3.8-27B with 6-bit quantization, I ended up measuring 'bugs in my own benchmark' and 'temperature settings' before the model itself

Hello, this is CoolZero.

Qwen3.8-27B was released in early August. Seeing the model card claim SWE-bench Pro 61.7 and Opus4.6 Max's 53.4, I honestly thought, 'If a 27B local model can achieve that, what will happen to my 60 questions?'

To conclude, Resolved 87.5% (14/16). The same number I got from Claude Opus 5 (via Claude Code CLI) in July was achieved by the local 27B model.

However, I had to redo it twice to get there.

The numbers from the first 16 runs were not fit for public release. The cause was not the model, but a bug in my self-made benchmark extractor and a configuration error by the author using temperature 0.2 for a thinking model. Since I'm at it, I'll write about the whole process. I think this will be more useful for those making their own benchmarks than a quantization comparison.

First, the conclusion in 4 lines.

• **Due to a bug in the extractor, the complete implementation produced by the model was discarded.** When I re-ran the corrected version on 564 actual tasks, 14 items were salvaged from 'no code' to 'actual code', and of those, 9 items actually passed the hidden tests. Zero regressions. Tests went from 218 passed to 285 passed.

• **Temperature 0.2 was poison for the thinking model.** When I returned to the officially recommended temp 1.0, L7 aborts went from 15.3% (22/144) to 2.1% (2/96). With the same file and same harness, UD-Q6_K_XL went from 68.8% to 87.5% (+18.7pt)

• **Of the 16 L7 questions, 11 were passed by all 6-bit quantizations, and 1 (t105) failed all 6-bit quantizations.** Only the remaining 4 questions determined the ranking. At this scale, the difference in quantization is barely measurable

• The overall top is UD-Q6_K_XL (Resolved 87.5% / Quality 85.3 / Combined 81.1). However, at ctx 64k, it uses VRAM 29.4GB. If choosing for practical work, UD-Q4_K_XL (same 87.5% / 22.3GB / ~108 tok/s).

The nature of this model

Hybrid attention is a configuration that has continued since the Qwen3.6 series, but it uses `full_attention_interval: 4`, meaning it only uses full attention once every 4 layers. It is thanks to this structure that it can claim 1M tokens.

And **the bottom row of this table is the star of this article.** I ran 16 runs with temp 0.2, which I used for the Gemma series, without looking at this.

Model card listed values (reference values, not the author's actual measurements)

Source: Qwen/Qwen3.8-27B Model Card. These are nominal values and are different from the actual measurement table in this article.

What is noteworthy is that while GPQA Diamond is almost flat at 87.8 to 89.2, DeepSWE jumped from 13.3 to 42.2 and QwenSWEBench from 49.3 to 79.0, meaning only the agent/coding systems are jumping. It can be read as a generation shifted toward 'completing tasks to the end' rather than raw knowledge volume (Confidence: MODERATE).

GGUF size and actual VRAM

GGUF size is the byte count of the actual file recorded in results.json, and VRAM is the actual measurement of `vram_total - vram_free` immediately after starting `llama-server`.

**If you set ctx to 64k, only Q4_K_M (21.6GB) and UD-Q4_K_XL (22.3GB) will fit on a 24GB machine.** Q5_K_M barely touches 24.0GB, and UD-Q6_K_XL's 29.4GB leaves only 2.0GB remaining even on a 32GB 5090. If you allow long thinking with a thinking model, there is practically no room to increase bits might be the most practical discovery this time.

Note that if you use images/videos, `mmproj-F16.gguf` (approx. 0.93GB) is required separately. This benchmark is text-only, so it was not used.

What is Unsloth Dynamic (UD-*_K_XL)?

The unsloth card says "Unsloth Dynamic V3.0 (preview)." According to the Dynamic 2.0 documentation, while standard K-quant applies a uniform format to all layers, UD is a method that changes the quantization type for each layer and uses a different recipe for each model method. Its selling point is that it uses a proprietary dataset (over 1.5 million tokens) including chat templates for calibration, and evaluates degradation using KL divergence instead of perplexity (Confidence: HIGH, as stated in official documentation).

However, I cannot find any public documentation on what has changed in V3.0 (only the 2.0 page is available). I will write honestly here that it is a "preview version with unknown details."

In terms of size, it is consistent with the explanation. UD-Q4_K_XL is about 0.8GB larger than Q4_K_M, and UD-Q6_K_XL is about 3.0GB larger than Q6_K. It seems safe to understand that it is larger because more bits are allocated to important layers.

**I do not use any custom quantization in this article.** I am using the full-run unsloth version (`/mnt/data/models/Qwen3.8-27B-GGUF/`) and recording the model file's fingerprint (size / mtime / head+tail of sha256) in results.json.

Verification environment and two-stage design

• Execution environment: llama.cpp (CUDA build, `b10431-1692f9e50`)

• Hardware: EVO-X2 (AMD RYZEN AI MAX+ 395) + RTX5090 32GB. RTX3090 24GB is used only as a reference value for speed comparison

• Number of tasks in the first stage: 60 questions (easy 5 / medium 5 / hard 10 / expert 12 / frontier 8 / architect 20)

• Number of tasks in the second stage (main topic): 16 questions (L7 = grandmaster band only. Task IDs are non-sequential from t063 to t107)

• Number of executions: Each task 1 time (not 5 runs. Please be sure to read the reservation mentioned later)

• Content: Python bug fixing and implementation tasks (with deterministic tests). L7 includes migrations spanning multiple files, multiple bugs, and concurrent control

• Sampling: Official recommended temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / seed 42 (actually sent to payload, and effective values recorded in `environment.sampling`)

• Startup options: `-ngl 99 --ctx-size 65536 --parallel 1 --batch-size 2048 --ubatch-size 512 -fa on --threads 8 --spec-type draft-mtp`

• Output token limit: 49,152 tok

• Evaluation axes: Resolved rate / Quality / Combined / usability / tok/s

I will repost the explanation of llmbench as usual. It is a self-made benchmark similar to SWE-Bench, where bug reports and source code are passed to the LLM, a patch is applied, and then a hidden pytest (not disclosed to the LLM) is used to determine the resolution. Since pytest includes `-x`, it is aborted as soon as the first test fails.

• Combined = Success Rate × (0.5 + 0.5 × Quality/100) × 100(Non-functional code scores 0)

• Quality = A composite score re-normalized from ruff (weight 0.4) + maintainability metric radon (0.3). Effectively, ruff is about 57% and complexity is about 43%.

• usability: 🟢Autonomous = Success Rate ≥ 0.9 and Quality ≥ 80 / 🟡Assisted = Success Rate ≥ 0.6 / 🔴Impossible = Otherwise

• GitHub: `zephel01/swe-bench` (MIT). Reproducible with stdlib-only

Breakdown of this execution

**The 6 runs in bold are the main data for this article.** The 9 runs and 4 runs above them are used only for comparison as 'records from before the redo'.

A note on MTP. `--spec-type draft-mtp` is a feature that was merged into llama.cpp in May 2026, but Qwen3.8 is not yet listed on Unsloth's MTP support table(only up to Qwen3.6 is listed). However, each file of Qwen3.8-27B-GGUF has a size that is almost identical byte-for-byte to the 'MTP version' repository of Qwen3.6, so it is presumed that the MTP weights are included in the main body (Confidence: MODERATE, I have not directly verified the tensor names). Judging by the measured tok/s, it appears to be working, but I have not conducted a control experiment with MTP turned off.

Phase 1: 60 questions can no longer measure anything

Result Summary (60 questions, 1 run each, RTX5090)

**Conclusion: You should not discuss the superiority of quantization based on this table.** The range of the 6 runs is 95.0–100.0%, with a maximum difference of 3 tasks. Furthermore, the ctx, sampling, and harness versions are all different between the top 4 rows and the bottom 2 rows. I also dropped the Quality column because the old 4 runs did not have their Quality recalculated after re-extraction (mixing them together causes more harm).

This is the first time in the 60-question version of this series that Resolved 100% (60/60) was achieved. That in itself is news, but it is also a declaration that 'this difficulty level is finished'.

By difficulty: easy to frontier is a complete ceiling

(All values are after re-extraction)

**If we only look at the 40 questions (easy to frontier), 4 out of the 6 quantizations got a perfect score, and the remaining 2 only missed one question each.** The only room for differentiation is the 20 architect questions, and even there, the maximum difference is 2 tasks. **This is not a comparison of quantization.** Therefore, as a second phase, I added 16 L7 (grandmaster) questions.

Tasks failed in 60 questions (7-run aggregate, after re-extraction)

**t046 is the bottleneck for this model on the 60-question side.** It failed in 5 out of 7 runs, and only UD-Q5_K_XL and UD-Q6_K_XL passed it. t020 and t021 are regulars throughout this series (calculator, banker's rounding), so they are familiar faces in line with the lineage.

Phase 2 (Main Topic): L7 grandmaster 16 questions × 6 quantizations

From here on is the main data with **all runs under identical conditions (Unsloth version / ctx 64k / official recommended sampling / corrected harness / seed 42).**

Result Summary (L7 16 questions, 1 run each, RTX5090)

Conclusion: Resolved is tied for the top at 87.5% for 3 quantizations. Looking at Quality and usability, UD-Q6_K_XL is the sole leader, but since it requires 29.4GB at ctx 64k, the one to choose for practical work is UD-Q4_K_XL (tied for top, 22.3GB, fastest).

When sorted in Combined order, it is UD-Q6_K_XL 81.1 → UD-Q4_K_XL 80.0 → Q4_K_M 78.2 → Q6_K 75.0 → Q5_K_M 66.8 → UD-Q5_K_XL 66.7.The pattern where the 5-bit range forms a valley is common to both standard and UD. The idea that 'increasing bits improves accuracy' does not hold true this time either.

However, **since each is 1 run, 1 task = 6.25pt.** The difference between 87.5% and 81.2% is just one task, which is well within the range of luck.I will not conclude superiority or inferiority based on a difference of about ±6pt.

Stumble ①: The extractor was discarding the model's correct answers

Watching the first 16 runs, I noticed that 'the content of the failures was strange' and audited the benchmark itself. This is a definitive example.

`--- FILE: sniff.py ---` appeared twice in the `llm_output.txt` of the `t057` run (Q4_K_M, 60 questions).

Because `patch.py` was overwriting the same path with the last-write-wins rule, the saved `generated/sniff.py` became a 4-byte `...`, pytest returned `ImportError: cannot import name 'parse'`, and the result was recorded as a 'model failure'.

There are three main fixes.

1. Abolished last-write-wins for the same path (first-write-wins using `setdefault`) + added an `ast.parse`-based gate to reject `...` / `pass` / `<placeholder>` / syntax errors

2. Changed the code block regex to include line-start anchors and generalized language tags. The old regex only matched ` ```python `, missing the opening fence for ` ```json `, causing the fence handling to be off by one, picking up prose of thought as code

3. Sent `seed` / `top_p` / `top_k` / `min_p` in the payload (previously they were never sent, so the llama-server default of `seed=-1` = random every time was in effect)

Effect of the fixes (re-applied to 564 actual tasks, zero GPU time):

**The fact that it didn't pass even after 5 cases were rescued is proof that the fix is not creating false passes.** The test went from 218 passed → 285 passed (all existing tests passed without rewriting a single line, plus 67 new ones).

With this fix alone, the L7 score moved by 6.2 to 12.5pt. **This was also the true nature of the fluctuation that looked like 'running the same quantization twice results in a 6.2pt difference'.** After the fix, the re-run pair for Q6_K matches perfectly at 87.5% / 87.5%.

Stumble ②: Temperature 0.2 was poison for the thinking model

Even after fixing the harness, the quality of the L7 failures remained strange. **Nearly 20 out of 16 questions were 'ending because they reached the output limit'.**

At first, I thought 'the thinking budget is insufficient', so I doubled the limit from 24,576 to 49,152 and increased the ctx from 32k to 64k. **The result remained at 75.0% Resolved.** In fact, t101, which had passed with 11,026 tokens, used up all 49,152 tokens and failed. Doubling the budget just meant it looped longer.

Then I re-read the model card and finally realized. **The official recommendation for Qwen3.8's thinking mode is temp 1.0.** I had been using 0.2, which I used for the Gemma series, as is. I was still operating under the common sense of non-thinking models: 'lower the temperature for code generation'.

Termination rate reduced by a factor of seven

**I must specify here that two conditions are changing simultaneously.** The old run had a limit of 24,576 / ctx 32k, and the new run has a limit of 49,152 / ctx 64k. Just doubling the limit reduces the termination rate.

However, as mentioned above, the control run with only the limit doubled (still at temp 0.2) remained at 75.0%. The run with the same ctx and same limit but with sampling returned to the official recommendation was 87.5%. I judge the main cause to be the sampling side (Confidence: HIGH).

The degenerate loop has actually disappeared

I mechanically counted 'how many times the same line was repeated' and 'how many characters the longest line had' in `llm_output.txt`.

At temp 0.2

At official recommended settings

The degeneracy of 12 repetitions and 15,434 characters per line has disappeared almost entirely.

However, it has not been eradicated

Even with the official recommended sampling, one blatant case of degeneracy remained in the Q6_K for the 60-question side. Please look at the output for t053 (attribute value escaping).

data: dangerous for script, but not necessarily XSS? Could be. ...
Need think about URL scheme detection with escaped entities:
If href contains "javascript&#00000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
（以下、"0" が 7,930文字続いて出力終了）

It tried to write out the HTML numeric character reference ``, but it kept typing zeros and never returned. There are 0 FILE markers in the 20,053-byte output. The result is `patch parse failed: no file blocks found in output`.

QwenLM Issue #145 mentions that 'there are reports of falling into loops even with official recommended sampling' (Confidence: MODERATE, summary based on what I have read), butthe correct conclusion from actual measurements was 'it can be significantly reduced but not eradicated'.

The two cases that reached the limit on the L7 side have the same symptoms.

**The two cases that reached the limit are both RESOLVED.** This is because the revised extractor was able to pick up the completed draft that had been written mid-thought. Before the fix, both would have been recorded as 'model failures'.

Reversal phenomenon: 'Unsloth Dynamic loses to standard across all bit bands,' but it disappeared when I reverted the temperature

In the data before I redid it, I concluded that 'there is not a single bit band in L7 where UD outperforms standard.'This is the result of changing only the sampling with the same model file and the same harness.

It moved 25pt in the 6bit band alone.The assessment that 'UD-Q6_K_XL loses to standard by a large margin in the 6bit band' is highly likely to have beenjust a matter of UD-Q6_K_XL being more prone to degenerate loops at low temperatures (Confidence: MODERATE. I will not state this definitively as it is a comparison between single runs).

The before-and-after comparison for each quantization is as follows.

Only Q6_K has dropped, but this is for one task (6.25pt), which is within the range of re-run variance. **The average of +5.7pt is larger than any difference between quantizations I have seen in this series.** This happened just by fixing one line in the configuration file, without changing the model or the quantization.

The most important discovery: Only 4 out of 16 questions determined the ranking

This is the pass table by task for 16 L7 questions × 6 quantizations.

Here is how to read it.

• 11 questions (69%) passed all 6 quantizations. There is absolutely no difference in quantization here

• Question 1 (t105) failed in all 6 quantizations. There is no difference here either.

• Only the remaining 4 questions (t069, t095, t102, and t103) are creating the rankings.

In other words, the difference between 87.5% and 75.0% is essentially just a matter of 'how many of the 4 questions were answered correctly'. Moreover, there is absolutely no correlation between the passing patterns of t102 and t103 and the quantization (t103 passed with UD-Q4 and Q6_K, but failed with UD-Q6).

**At this scale, the superiority or inferiority of quantization cannot be measured.** What is being measured is the distribution of the model's actual capabilities: '11 out of 16 questions can be solved stably,' '1 question cannot be solved no matter what,' and '4 questions are close to a coin toss.'This is not because the benchmark design is poor, but rather the limitation of the act of comparing quantization in a single run.

Comparing with the cross-sectional verification from July: t101 failed

On July 25, I measured Claude Opus 5 and 4 other models (9 runs in total) using the same 16 L7 questions. I will compare those results here.

Source: Author's actual measurements from 2026-07-25. **The playing field is different.** Since Claude Opus 5 is via the Claude Code CLI, I am measuring an 'agent product' where the CLI can perform tool round-trips and self-correction internally. The temperature is also not controlled.Since this is not a comparison of raw models, I will not write that '27B is on par with Opus 5'.

Even so, a meaningful comparison can be made.Qwen3.8-27B passed t101 (log rotator 3-multiplex oracle), which had been completely wiped out in the 9 runs in July, in all 6 quantizations.

**However, it is dangerous to straightforwardly interpret this difference as 'the model has become stronger'.** The 9 runs in July were fixed at a temperature of 0.2, and they also encountered the extractor bug that I fixed in this article. It is currently impossible to separate how many of the 0/9 cases for t101 were 'could not be solved' versus 'discarded by extraction/thinking did not finish' (Confidence: MODERATE).

This is all I can say.'There are currently 9 runs worth of subjects that should be re-measured under the same conditions'—this is my homework.

The aftermath of the three walls: Only t105 was the real one

A follow-up on t095 / t103 / t105, which were completely wiped out in the 9 runs at temp 0.2.

**I will leave a note about the judgment error for t103.** I wrote that it was a 'type that won't pass even if the limit is raised,' even assigning it a 'Confidence: HIGH.' In reality, it had nothing to do with the limit, and it passed when I reverted the temperature.When assigning high confidence to an inference of causality, I should have run a control experiment first.

t105 is a real wall. Although the hidden test requires that 'a KeyError be thrown when an empty record is passed,' that requirement is not written in the issue statement. In fact, the string 'KeyError' appears 0 times when searching the output of all 15 L7 runs. It is a defect on the task side, requiring information that the model cannot read (Confidence: HIGH). Combined with the 9 runs in July, 24 runs have been completely wiped out, so I will replace it in the next task ledger update.

Usability: How much can you actually rely on it?

Even though Resolved is the same 87.5%, Q4_K_M is 8/16, and UD-Q6_K_XL is 11/16 for 🟢Autonomous. The difference comes from Quality (78.8 vs 85.3).There is still room for bit count to have an effect between 'writing code that passes' and 'writing code that can be passed without review' (Confidence: MODERATE).

On the 60-question side (including architect), 🟢 is 37–42/60, so it clearly drops in L7. The straightforward reading is that in the grandmaster band, the proportion of code that 'works but requires review' increases.

Speed: RTX5090 vs RTX3090, and the price of 64k ctx

**2.71 times faster with the same GGUF and same startup options.** Although the 3090 is running with 18.9GB loaded, the practical feel is quite different.

The price of context length has also become visible.

**When you double the ctx, the speed drops by about 10%, and VRAM usage increases by about 2.2GB.** If you try to secure a budget for thinking in a thinking model, you will have to pay both of these costs.

How to read tok/s (a reminder every time)

1. **Models with MTP (Multi-Token Prediction/Speculative Decoding) will structurally show faster numbers.** This time, all runs were performed with `--spec-type draft-mtp`. This is not proof of intelligence.

2. **MoE models are fast because their activated parameters are small.** Qwen3.8-27B is dense, so it does not have this advantage.

3. **The standard dense version is honestly slow.** Slow does not mean inferior.

All the numbers above are for single-run execution (`--parallel 1`). The tok/s per stream during parallel execution will change by an order of magnitude, so please do not mix them for comparison.

Which one should you choose in the end?

• **If you have 24GB of VRAM, UD-Q4_K_XL is the only choice.** It fits in 22.3GB at 64k ctx, with an L7 of 87.5% (tied for top), and is the fastest at ~108 tok/s. Q4_K_M has the same score, but the Quality is 78.8 vs 82.9, meaning 🟢Autonomous is 1 question lower.

• **If you have 32GB of VRAM and prioritize quality, choose UD-Q6_K_XL.** With a Quality of 85.3 and 🟢Autonomous 11/16, it is clearly a cut above the rest. However, it uses 29.4GB at 64k ctx, leaving only 2.0GB.If you drop the ctx to 32k, you will have 27.3GB, giving you more breathing room.

• **I could not find a reason to choose 5bit (Q5_K_M / UD-Q5_K_XL) in this data.** Both standard and UD are at 75.0%, making them slower and heavier than 4bit.

• **Q6_K (standard) is in an awkward position.** Even though the Quality of 84.5 is good, the Resolved rate is 81.2%, and it uses 26.7GB. If you are in the same VRAM bracket, it is more straightforward to run UD-Q6_K_XL at 32k ctx.

**And a mandatory setting common to all quantizations: `temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0.0`.** If you set this to 0.2, you will lose 5 to 19 points regardless of which quantization you choose.

Four lessons to take away from this verification

1. Do not use low temperatures for thinking models.The idea that 'temperature should be low for code generation' is common sense for non-thinking models, but it is counterproductive for thinking models. The thinking process will not converge, and it will end up writing the same line 12 times for 15,000 characters until it hits the limit.The officially recommended sampling settings are initial values you should follow without doubt.

2. Increasing the budget will not fix broken settings. When I doubled the output limit, it just looped longer. (t101, which had passed at 11,026 tokens, failed after using up 49,152 tokens). Before thinking 'it will solve it if I let it think more,' suspect 'is it thinking unnecessarily?'

3. **Benchmark scores measure your own harness before they measure the model's true ability.** A single bug in the extractor caused L7 to fluctuate by 6.2 to 12.5 points.If you have your own custom benchmark, auditing your own code will yield higher returns than adding more models.

4. You cannot compare quantizations with 1 run of 16 questions. 11 questions were passed by all, 1 was failed by all, and only 4 questions determined the ranking.The reality is not '87.5% beat 75.0%,' but rather 'whether it got 3 out of 4 questions or 2 out of 4'.

Next homework

When doing ①, please be careful of the `sample_temp: 0.35` trap. If you set `runs > 1`, the temperature will be replaced by 0.35 during execution, and the official recommendation of 1.0 will be lost. Pass `--sample-temp 1.0` or set `sample_temp` in the config to 1.0. Also, comment out `seed` (otherwise all 5 runs will use the same seed).

To be honest, this article became more of a reflection on my own benchmark than a review of the model. However, I consider it a gain that I was able to descend from 'ending with 100% Resolved on 60 questions' to 'only being able to measure differences on 4 out of 16 questions'. It is worth questioning how you measure things, especially when the numbers look good.

The code and test results generated during verification have been archived. Please leave a comment if you would like to see details on a specific task or try it in your own environment. The fixes for the benchmark side are in the `fix/extraction-truncation-seed` branch of `zephel01/swe-bench`.

Model Information

• HuggingFace: https://huggingface.co/Qwen/Qwen3.8-27B

• GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

• Base: dense 27.3B (Hybrid attention / Vision encoder included / MTP trained)

• License: Apache 2.0

• Recommended sampling: thinking: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0

Related Articles

• Cross-benchmark verification of 16 L7 grandmaster questions across 5 models including Claude Opus 5 (2026-07-25)

• llmbench explanation: How to read features and scores

• Benchmark used: https://github.com/zephel01/swe-bench (MIT)

#LocalLLM #llamacpp #benchmark #quantization #GGUF #Qwen3 #27B #Unsloth #thinking #sampling #RTX5090