Validating Qwopus3.6-35B-A3B-Coder (MoE version) with 60 tasks — 88% pass@1 for Q6, 98% pass@k for Q5 with 5 attempts

Hello, this is CoolZero.

I have been following the Qwopus series, focusing mainly on the 27B dense versions (standard and MTP), but this time the 35B-A3B MoE version has arrived. It is Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF. It features a sparse MoE configuration with 35B total parameters and approximately 3B active, which is convenient for local execution.

This time, I verified three quantizations—Q4_K_M, Q5_K_M, and Q6_K—using a 60-task coding benchmark. Furthermore, I ran them in two modes: **(1) Single-shot (one attempt = pass@1)** and **(2) 5 attempts per task (pass@k / k=5)**.

Here is the conclusion first. For a single-shot attempt, Q6_K pulls ahead with 88.3% (53/60). On the other hand, if you allow up to 5 attempts, Q5_K_M reaches a pass@k of 98.3% (equivalent to 59/60), which means 'Q5 is sufficient for almost everything if you are using an agent that assumes retries.' Also, the phenomenon familiar from the 27B version, where 'only Q6 can solve the calculator (t020) task,' was reproduced in this MoE version as well.

Characteristics of this model

The concept of this model is clear: it is optimized to 'stop inserting long thoughts at every step, and instead focus on reading -> choosing a tool -> fixing -> testing -> moving to the next, all with low tokens and low latency.' It is designed to fit into agent harnesses like Codex, OpenHands, Claude Code, and OpenCode.

According to the official benchmark, the thinking-off Qwopus 35B scored 62.4% on 300 SWE-bench cases, and the HF card states that it leads Ornith-1.0 35B (thinking-on) with an average of 82.1 vs 78.9 in behavioral comparisons. This time, I will verify its raw coding ability across different quantizations using my own 60 tasks.

GGUF size (HF official/MTP version):

Verification environment

• Execution environment: llama.cpp (CUDA build, MTP enabled)

• Hardware: EVO-X2 + RTX5090 32GB

• Number of tasks: 60 questions (easy 5 / medium 5 / hard 10 / expert 12 / frontier 8 / architect 20)

• Content: Python bug fixing and implementation tasks (from functions to multiple files, with deterministic tests)

• Evaluation metrics: Resolved rate (all pytest passed) / Quality (ruff/cyclomatic complexity) / Combined (overall) / usability (🟢autonomous / 🟡assisted / 🔴failed)

• 2 modes: (1) Single-shot (pass@1) / (2) 5 attempts (calculating pass@1 and pass@k=5, 5 parallel executions)

This is a 60-task configuration, adding 20 architect-level (multiple-file, system design level) questions to the 40 tasks from last time. Think of the 40 questions from easy to frontier as 'straightforward function-level bug fixes,' and the latter 20 architect questions as 'difficult points involving design decisions.' The model's true capability is revealed in how it misses these.

(1) Summary by quantization for single-shot (pass@1)

The champion for a single-shot attempt is Q6_K. With 53/60, it pulls ahead in Resolved, and its Combined score is 80.7, the only one in the 80s. Since the Quality (code quality score) is almost the same for all three quantizations at 82-84, the difference is directly reflected in the Resolved rate, i.e., whether it passed or failed.

Breaking it down by difficulty makes the composition clear.

For the 40 questions from easy to frontier, every quantization gets almost a perfect score (Q6 passed 39 out of 40). The difference only appears in the final 20 architect questions, where Q6 is 14/20, which is 3 questions stronger than the 11/20 of Q4/Q5. In other words, the difference in the raw capability of Qwopus 35B-A3B only surfaces in design-level tasks.

② 5-trial (pass@k): Q5 shines if retries are assumed

In agent operations, it is standard to automatically retry even if a task fails once. Therefore, I measured pass@k (k=5) by running each task 5 times and counting it as a success if it passes at least once.

Here, the lead role changes. When allowing up to 5 attempts, Q5_K_M reaches a pass@k of 98.3%, or 59 out of 60 questions. It outperformed Q6 (95.0%) and Q4 (93.3%), making it the best among 3-bit quantizations.

Why does Q5 win in pass@k instead of Q6, which is the top performer in single-shot? The key is the "quality" of the failures. Looking at the failure breakdown for 5-runs, many of Q5's misses were 'flaky 1/5 passed' (passed once out of 5). In other words, Q5 has many tasks it can 'hit occasionally,' which can be picked up by increasing the number of trials. On the other hand, Q6 passes consistently, but the difficult spots where it gets stuck and results in 0/5 remain, meaning there is less room for improvement in pass@k—that is the structure.

The translation for practical operation is this: If you want a high-quality answer in one shot, use Q6; if you want to increase the completion rate in an agent loop with automatic retries, use Q5. It is rational to decide which one to choose based on whether the harness retries.

t020 (Calculator): Once again, only Q6 solved it

A difficult challenge customary to this series, t020 (order of arithmetic operations and parenthesis handling). In the verification of the 27B version, it was legendary that 'only the MTP version of Q6 solved it,' but even in this MoE version—

In single-shot, only Q6 passed on the first try. Q4/Q5 failed. In 5-trial runs, every quantization was in a 'flaky' state, meaning they could not solve it consistently. The evaluation that recursive descent parsers and operator precedence are delicate boundaries where the quantization bit count matters for this model family remained consistent even in the MoE version.

The 'wall of raw ability' that failed across all quantizations

Conversely, tasks that did not pass with any quantization or in either mode are the raw limits of this model. The ones that were nearly wiped out across both single-shot and 5-run were around the following:

• t021 (expert / banker's rounding) ... Failed in more than half of the trials

• t043 (architect / double counting of replay after snapshot)

• t046 (architect / DI lifetime management)

• t047 (architect / rollback of failed migration)

• t059 (architect / error correction for floating-point sum)

All of these are expert-to-architect level tasks involving state management, numerical precision, and lifecycles. While it handles easy-to-frontier tasks almost perfectly, areas requiring 'design decisions and maintenance of long invariants' still collapse—this can be said to be the direct flip side of a design that earns its performance by turning off thinking.

Speed: Single-shot is about 300 tok/s, 5-parallel is about 80 tok/s per stream

You might think the digits are different when looking at the numbers, but this is reasonable. Single-shot is the effective speed for 1 stream, while 5-run is 5-parallel execution, so the tok/s per stream just drops, but the total throughput is actually higher.

The focus is on the single-shot speed. Thanks to the active ~3B MoE + MTP (speculative decoding), it achieves a median of about 300 tok/s while being in the 35B class. Even from Q4 to Q6, the drop is limited to about 10%, from 325 to 296, showing the effectiveness of MTP, which is a 'speed-up layer independent of bit count.' Considering that the 27B dense MTP version was around ~130 tok/s, the speed advantage of the 3B-active MoE is clear. For local agent loop use cases, this single-shot speed directly affects the user experience.

usability: Minimum failures 7/60

Here is the usability assessment (single-shot) of "how much can it actually be trusted without review?"

Q6_K performed best with 40/60 autonomous and 7/60 failures. A 12% failure rate is a strong showing for a 60-question set containing this many architect tasks. Q4/Q5 failures were 11–12, mostly occupied by the usual suspects: calculators, banker's rounding, and design-level tasks.

Summary: Choose quantization based on use case

• Q6_K: 88.3% single-shot Resolved, 80.7 Combined, 67% autonomous. The top choice for single-shot quality. The primary choice if 29GB of VRAM is available.

• Q5_K_M: pass@k 98.3% (59/60). No. 1 completion rate for agent operations assuming retries. A realistic daily-use option at 25GB size.

• Q4_K_M: 80% single-shot, 93.3% pass@k. The 21.7GB lightness is attractive, but it misses a bit more on calculator and design-level tasks.

In a nutshell, "Q6 for quality in one shot, Q5 for completion rate with automatic retries included". And the combination of 35B-A3B + MTP has a major weapon in its speed of approximately 300 tok/s in single-shot. The 40 easy-to-frontier questions are already in the practical range, and the map clearly showed with these 60 tasks that the real battleground is the architect-level design tasks.

It was a personal gain to see the phenomenon where the calculator (t020) only passes with Q6 reproduced in the MoE 35B-A3B, following the dense 27B. The trend of the Qwopus series, where "the highest bit is not always the correct answer, but the peak of accuracy comes around Q6," seems consistent across architectures.

The code and test results generated during validation (raw LLM output, generated code, pytest output) have been archived. Please leave a comment if you want to see details of a specific task or try it in your own environment.

Related articles (in the same magazine)

• 40-task validation of Qwopus3.6-27B-Coder MTP version/standard version

• Qwable-3.6 series / Qwable-v2

• Ornith-1.0 9B/35B validation

Model Information

• Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF (HuggingFace)

• Base: Qwen3.6-35B-A3B (Total 35B / Active approx. 3B, hybrid sparse MoE)

• Design: thinking-off agent coder / MTP enabled

• SWE-bench: 62.4% (300 cases, thinking-off, Q5_K_M, nominal)

• https://github.com/zephel01/swe-bench

#LocalLLM #Qwopus #Qwen #MoE #CodingModel #Quantization #llamacpp #Benchmark #GGUF #35B #MTP