Speculative decoding for Qwen3.6-27B

[jmig1109]

Has anyone managed to get positive results using a draft model for Qwen3.6-27B ? and if so, How?

I'm on the latest commit, in a system with 2x RTX 3090 and a RTX Pro 4000.

If I load the model at full context on the two 3090's i get ~38 tps as a default. If i try out '-sm tensor' i get a nice bump to 53 tps but it tends to be quite buggy.

Now, if I add Qwen 3.5 0.8B as a draft model, i'm only seeing regressions. i've tried lowering the context and kv cache size to make it fit on the 3090's but the best i get is 33 tps, and even lower when combined with the ngram-mod (18 tps). What am i doing wrong?

Here's my setup:

llama-server --version
ggml_cuda_init: found 3 CUDA devices (Total VRAM: 72206 MiB):
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24126 MiB
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24092 MiB
  Device 2: NVIDIA RTX PRO 4000 Blackwell, compute capability 12.0, VMM: yes, VRAM: 23987 MiB
version: 8960 (19821178b)
built with GNU 13.3.0 for Linux x86_64

Here are my test arguments

CUDA_VISIBLE_DEVICES=0,1 llama-server \
--model /mnt/fast1/gguf-models/Qwen3.6-27B/Qwen3.6-27B-Q5_K_M.gguf \
--model-draft /mnt/fast1/gguf-models/Qwen3.5-0.8B/Qwen3.5-
0.8B-Q4_K_M.gguf \
--host 0.0.0.0 --port 8080 \
-fa on \
--cache-prompt \
-c 65536 \
-np 4 \
-kvo \
-kvu \
-ngl 99 \
-ngld 99 \
--no-mmproj
# 33 tps average

and without draft-model:

CUDA_VISIBLE_DEVICES=0,1 llama-server \
--model /mnt/fast1/gguf-models/Qwen3.6-27B/Qwen3.6-27B-Q5_K_M.gguf \
--host 0.0.0.0 --port 8080 \
-fa on \
--cache-prompt \
-c 65536 \
-np 4 \
-kvo \
-kvu \
-ngl 99 \
--no-mmproj
# 38 tps average

and with the ngram mod and -sm tensor:

CUDA_VISIBLE_DEVICES=0,1 llama-server \
--model /mnt/fast1/gguf-models/Qwen3.6-27B/Qwen3.6-27B-Q5_K_M.gguf \
--host 0.0.0.0 --port 8080 \
-fa on \
--cache-prompt \
-c 65536 \
-np 4 \
-kvo \
-kvu \
-ngl 99 \
--no-mmproj \
--spec-type ngram-mod \
--spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 48 \
--spec-ngram-mod-n-max 64 \
--split-mode tensor
# 53 tps average narrative - up to 65-70 tps for repetitive coding.

If others have more luck than me, i'd be glad to see what is the magic recipe!

Thanks

[am17an]

Possibly you can try #22400

[jmig1109]

I tried it out -
It definitely created an improvement when using the draft model alone - closer to 50-60 tps when coding. However, if i tried to use -sm tensor or -spec-type ngram-mod in any capacity (alone or with draft model) it would silently stop generating an output while locking the gpus at full power.

[am17an]

Yes it's a WIP, will fix issues

[jmig1109]

thanks for the hard work!