Highlights

582 PRs from 194 contributors.

Kimi K3 day-0 support: A 2.8T-parameter multimodal LatentMoE (896 experts, top-16, routed in a 3584-dim latent space) with a 1M-token context, 69 KDA linear-attention layers interleaved with 24 MLA layers, and a MoonViT3d vision tower, shipping as a native MXFP4 checkpoint. SGLang serves it from day 0 with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on the quantized weights, and reasoning, tool-call and OpenAI-compatible serving, verified on NVIDIA GB300 and AMD MI35x (#32541, #32828, #32890, #33025, #33112, blog, cookbook, roadmap).

MiniMax-H3 day-0 support: MiniMax's video generation model that produces a video and a synchronized stereo audio track in one request, served natively on SGLang-Diffusion across all three public task profiles: text-to-video-and-audio (t2va), first/last-frame conditioning (fl2va), and image/video/audio reference conditioning (ref2va, which also covers video-to-video). Verified on B200 (TP2 + Ulysses4), H100 (TP2 + Ulysses2), AMD MI300X and MI355X (Ulysses1/2/4/8), and 2x RTX 5090 with layerwise offload (#33275, cookbook).

Other new models added: EmbeddingGemma and LFM2.5 embedding models, nvidia/MiniMax-M3-NVFP4, plus cookbook recipes for Poolside's Laguna-S-2.1 family and Inkling-Small.

Initial support for the Rust frontend: Migrates the front half of the server, everything from network ingress up to the point a tokenized request is handed to the GPU scheduler, from Python to a multi-threaded Rust implementation (#29799).

DCP communication backends and q-replicate (Helix): The DeepSeek-MLA decode context-parallel path gains pluggable comm backends. a2a exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer, with fp8 KV carried as uint8 byte transport; fi_a2a delegates the cross-rank exchange to the FlashInfer MNNVL kernel on GB200. --dcp-replicate-q-proj projects full-head Q locally and skips the per-layer Q head-dim all-gather. Select with --dcp-comm-backend {ag_rs, a2a, fi_a2a} (#21637).

DWDP for MoE prefill: A new prefill parallelism strategy that prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch. On 4x B200 with gpt-oss-120b, prefill-only, DWDP4 reaches 1.92x over DEP4 at MNT 32K / ISL 32K, and 506K vs 329K tok/s (1.54x) at saturation (CONC=128, ISL=8K). Enable with --dwdp-size; the authors mark it early-development (#29778).

Session-reference-aware Unified Radix Cache: For agentic and RL-rollout workloads, requests can carry a stable session_id so eviction knows which prefixes an active session still references, instead of evicting purely by cache policy. Release the references with /close_session. Opt in with --enable-session-radix-cache (#29173).

SM90 FP8 MegaMoE for DeepSeek-V4: Adds the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation. Guarded behind SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 (#29016).

Faster large-MoE model loading: Oversized or non-contiguous CPU weight views were driving pathological H2D transfers, with DeepSeek-V4-Pro TP8 spending 27 to 32 minutes in H2D on some ranks. Copying those views into contiguous storage before H2D cuts full model loading from about 35 minutes to 6m20s (5.6x), and GPT-OSS-20B BF16 from 545s to 70s (7.8x), with Qwen3.5-397B measured at 1.93x to 2.3x. Opt in with SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D, off by default (#32315).

Lower DeepSeek-V4 memory on AMD: Removing unnecessary expert padding drops MI355X FP4 MoE model weights from 159.07 GB to 112.36 GB, and bringing the HIP compress-state pool into the memory_saver KV_CACHE region lets colocated RL reclaim it, cutting the measured training-phase footprint from about 143 GiB to 87 GiB per GPU (#31450, #31747).

Faster engine recovery: Large-model restarts cost 3 to 6+ minutes today, about 6.5 minutes for Qwen3-235B FP8 on 4 GPUs, because weights reload from storage and CUDA graphs recapture. A weight-cache daemon holds weights per GPU so a restarting engine can recover from cache instead (#27139).

Lower host overhead in hybrid-linear MTP decode: Under spec-v2 overlap scheduling each decode step runs draft, verify and extend CUDA graphs, and the eager seams between them become GPU idle time at low concurrency. This trims that host work so the host stays off the critical path (#32219).

Dependencies: flashinfer 0.6.15.post1 (#31927), sgl-deep-gemm 0.1.5.post1 (#32345, #33143), helion 1.4 (#32562), mooncake 0.3.12.post1 (#32302), dynamo-tokenizers 1.7.0 (#32981). PyTorch stays at 2.11.0 and the CUDA base image at 13.0.1.

Full release notes by category below; breaking changes and known issues are at the end.

New Model Support

Kimi K3

• [Kimi] Support kimi-k3: #32541 ⭐
• [Kimi] Support DCP + DSpark (ported from kimi-k3 branch): #32828
• [Kimi K3] Add reasoning, tool-call, and OpenAI serving support: #33025
• feat(kernels): port standalone Kimi K3 kernels: #32890
• [Feat] DCP + HiCache L2 Support (ported from kimi-k3): #33112
• Replace Kimi K3 DeepGEMM patch with 0.1.5.post1: #33143
• docker: add Kimi K3 images: #32760

Rust Server

A native Rust serving layer: tokenizer manager, ingress validation and egress, an OpenAI-compatible API server, and PD disaggregation support, shipped as prebuilt release artifacts.

• support rust sglang server: #29799
• create rust workspace: #32014
• init sglang rust server project: #32256
• add the rust server tokenizer, detokenizer, and egress modules: #32872
• add the rust server ingress request validation and api server common types: #32873
• add the rust server ingress tests, guard, and submit modules: #32874
• add the rust server api frame codec and http server entry: #32875
• add the rust server native api handlers and runtime threads: #32876
• wire the rust server modules into lib, runtime, and tokenizer manager: #32877
• sglang rust server tokenizer manager, ring and runtime: #32358
• feat: rust sglang server openai apis: [#33103](https://github.com/sgl-proj...

Highlights

574 PRs from 169 contributors.

DSpark: confidence-driven speculative decoding: A new speculative algorithm. It drafts semi-autoregressively in blocks, then sizes each verify window from the draft's own confidence instead of a fixed draft length. Reaches 383.7 tok/s at accept length ~5 on DeepSeek-V4-Pro, TP8 on B300 (bs=1). Enable with --speculative-algorithm DSPARK and SGLANG_RAGGED_VERIFY_MODE=compact; tune the block with --speculative-dspark-block-size (#30261, #31434, blog).

Inkling support: A 975B-parameter multimodal MoE with a 1M-token context. It mixes sliding-window, full and Mamba2 linear attention, and adds an NVFP4 MoE, optional vision/audio towers and native MTP. On Blackwell it reaches up to 71.7k tok/s input and 171.0 tok/s per-user decode. Verified on Blackwell TP4/TP8, H200 and AMD MI350X / MI355X (#31681, blog, cookbook).

Other new models added: LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5, plus diffusion support for LongLive 2.0.

UnifiedRadixTree is now the default for SWA, Mamba and DSA models. Replay SSM and Mamba int8 checkpoints are synced onto it, and a cache hit now resets only the state it used (#30468, #30636, #30626, #31643).

GLM-5.2 DSA cache layer split under prefill CP: KV and indexer cache layers are sharded across CP ranks. Each rank owns a disjoint layer range instead of all layers. That cuts per-rank KV memory by ~74% (0.77 to 0.20 GB/rank) at 8192 tokens on GLM-5.2-FP8, 78 layers, cp_size=4. Enable with --enable-dsa-cache-layer-split, which needs --enable-prefill-cp --cp-strategy interleave (#29421).

ReplaySSM Ring Spec-Verify (GDN): Drops the per-draft SSM snapshot. Speculative scratch goes from 11.5 GB to 1.8 GB per GPU (6.4x smaller) on Qwen3.5-35B-A3B at TP1, at accuracy and throughput parity. Opt in with --enable-gdn-replayssm-spec (default off; GDN with a linear draft chain only, --speculative-eagle-topk in {None, 1}), and tune the ring via --linear-replayssm-cache-len (#28695).

Linear attention on Blackwell (SM100): The first correct KDA MTP path. Its recurrent_kda decode kernel runs at 29.6 us vs 36.8 us for Triton (ncu, B=64). The full decode path reaches parity by B=128 and 1.35x at B=256, and is slower below that (#30113). Separately, GDN/KDA CuteDSL prefill fuses state I/O into the chunk-h kernel (#30169).

QServe and FBGEMM FP8 quantization are removed: the experimental QServe (QoQ) W4A8 and FBGEMM FP8 paths are gone. --fp4-gemm-backend cutlass goes too, along with the in-tree NVFP4 JIT kernels, so NVFP4 GEMM now requires FlashInfer (#31109, #30448).

Dependencies: flashinfer 0.6.14 (#29910), CuTe DSL 4.6.0 (#31714), sgl-kernel 0.4.5 (#31496), llguidance 1.7.6 (#31484).

Breaking Changes & Upgrade Notes

• The experimental QServe (QoQ) W4A8 and FBGEMM FP8 quantization paths are removed (per #28543): #31109
• CUTLASS FP8 blockwise deleted for SM90 / SM100, SM120 moved to JIT: #30438
• --fp4-gemm-backend cutlass is removed along with the in-tree NVFP4 JIT kernels, so NVFP4 GEMM now requires FlashInfer. Use auto, which picks flashinfer_cutedsl on SM100 and flashinfer_cutlass on SM120: #30448
• UnifiedRadixTree is now the default for SWA, Mamba and DSA models. A behavior change on those architectures: #30468
• Chunked input-logprob processing is now on by default to cap peak memory: #31498
• FA3 sparse mask kernels are off by default: #30356
• Legacy Sphinx docs/ removed; the Mintlify cutover is complete: #28964
• sglang.kernels namespace: kernels are relocated verbatim and only import paths change; public wrappers keep defaulting to the AOT sgl_kernel backend, so code reaching past them to internal paths must update (RFC #29630): #30044, #31582
• num_tokens_per_bs renamed to num_tokens_per_req across spec-decoding runners: #30977
• --enable-deepep-waterfill is renamed to --enable-waterfill with no deprecated alias, so existing launch commands fail with unrecognized arguments: #27350
• --optimistic-prefill-retries is renamed to --optimistic-prefill-attempts with no deprecated alias: #30951
• The SGLang-Diffusion post-training rollout endpoint now returns application/msgpack instead of JSON, with tensors as raw msgpack bytes rather than base64 (tensor_to_base64 / base64_to_tensor become tensor_to_bytes / bytes_to_tensor), so RL rollout consumers must be upgraded in lockstep with the server: #31565

Known Issues

• Temperature-0 nondeterminism under DP attention with breakable prefill CUDA graph. On the DSV4-Flash FP4 recipe, the idle-rank dummy extend introduced by #30898 perturbs real requests' logits, so identical temperature-0 requests can diverge. The guarding determinism test is disabled as a stopgap rather than fixed (#31125); not enabling breakable prefill CUDA graph avoids the path.
• A bump to flashinfer 0.6.15 was landed and reverted this cycle; this release pins 0.6.14 (#31502, #31625).
• Mamba track-boundary seqlen under the overlap scheduler was fixed and then reverted (#31369, #31622). The underlying issue is still open.
• CPU AMX optimizations for diffusion were reverted (#28527, #30716).
• GB300 CI jobs were temporarily disabled for runner availability during this cycle (#31764), so GB300 coverage rests on the cookbook's manual end-to-end validation.

Full release notes by category below.

New Model Support

Landed this cycle but not yet usable end-to-end: MiniMax-M3 completes its four-part landing (#28715, begun in v0.5.14) but its cookbook still points at a dev image (#31819).

Inkling

• Add Inkling model support: #31681 ⭐
• Add Inkling cookbook: #31360
• [Docs] Inkling cookbook: mark B300/GB300 recipes verified, tune B300 MTP mem fractions: #31550
• [Cookbook] Inkling: add measured accuracy numbers to benchmark cards: #31823
• [Docs] Inkling cookbook: LoRA cells require --disable-prefill-cuda-graph: #31418
• Fix dropped Inkling reasoning at stream end: #31787
• [Spec] fix inkling multi lay...

v0.5.15.post1 includes a few patches, mostly for GLM 5.2

• #30454 #30627: Fix DSA model launching on non Cuda/HIP devices
• #30858: Fix flashinfer dependency on Cuda 12 images
• #31001: Fix NaN outputs caused by flashinfer trtllm FP4 MoE kernels on long input
• #30839: Fix GLM 5.2 IndexShare on PD disaggregation setting
• #30992: Fix GLM 5.2 IndexShare on Context Parallel setting

Highlights

GLM-5.2 NVFP4, tuned for production: We took time this cycle to tune GLM-5.2 NVFP4 on Blackwell for optimized production serving. It now runs at 500+ tok/s/user on 8x B300, 450 on 4x GB300 (bs=1). Run GLM-5.2 with our cookbook.

• Spec V2 by default: zero-overhead scheduling via CUDA-graphable DSA draft-extend, dropped D2H/H2D syncs, fused metadata ops. +11% end-to-end TPS (#29413, #29343, #29166, #29077).
• IndexShare MTP: reuses the indexer top-k across draft steps, up to 1.9x lower draft-step cost at long context (#29959, #29787, #29654).
• TopK V2: fuses top-k selection with the page-table transform, runtime k up to 2048 (#30274, #26788).
• Indexer prologue fusion: 12 kernels to 4, ~8% faster decode at bs=1 (#27705).
• GEMM: shape-specialized JIT router GEMM + CuteDSL BF16 GEMM for Blackwell (#21531, #30117).
• FlashInfer autotune: now covers draft-model graphs (#29595).

New Model Support: Hunyuan 3 (Hy3), Hierarchical Reasoning Model (HRM-Text), NVIDIA LocateAnything-3B, Baidu Unlimited-OCR, JoyEcho multi-shot A/V, plus Qwen3.6 NVFP4 support.

Native web search (Exa): Built-in web_search support backed by Exa (#29342).

Breakable CUDA Graph on by default: Breakable CUDA Graph is now the default capture path, reducing per-step kernel-launch overhead (#29458); full CUDA Graph support for the prefill phase lands as experimental (#27988).

Linear-attention kernels (KDA / GDN): New FlashKDA prefill backend for safe-gate KDA linear attention (#29472), plus ReplaySSM buffered output-only decode for linear attention (#28451).

FlashInfer A2A for routed MoE: Adds FlashInfer all-to-all with the flashinfer_trtllm_routed MoE runner (#22394).

DeepSeek-V4 Optimization:

• Optimizes C128 state-pool allocation using the request state pool (#28612).
• FlashMLA sparse prefill is now enabled by default for DeepSeek-V4, reaching >10% throghput gain on long context. (#29775).
• Non paged indexer support for long context prefill, with >5% e2e throughput gain.(#29619).

Decode Context Parallelism: decode context parallelism lands for MLA models, including DeepSeek V3 and Kimi K2 series (#14194).

Dependency upgrades: transformers bumped to 5.12.1 (#29393); tvm-ffi / sgl-deep-gemm / tilelang upgraded (#29554).
and . See the DeepSeek-V4 cookbook.

Full release notes by category below.

New Model Support

• Hunyuan 3 (Hy3): #30201 (cookbook)
• Hierarchical Reasoning Model (HRM-Text): #27887
• NVIDIA LocateAnything-3B: #28958
• Baidu Unlimited-OCR: #29186 (cookbook)
• JoyEcho multi-shot A/V: #27420 (cookbook)
• Qwen3.6 ModelOpt mixed NVFP4: #27906, #29905 (cookbook) ⭐

GLM-5.2

• [BCG][GLM5] perf: BCG support and prefill enhancements: #27053 ⭐
• [CI] Add GLM52 NVFP4 MTP B200 tests: #30021
• [cookbook] GLM-5.2 NVFP4 B300: TP8 recipe + 3 strategies: #29557
• [Cookbook] GLM-5.2: tune GB300 NVFP4 recipes + fill benchmarks: #29486
• [cookbook] drop redundant serve flags (GLM-5.2) + fix M3 page-size note: #28731
• [Docs] Add NVFP4 quantization to GLM-5.2 cookbook: #29380
• [DSA][GLM5.2] Index Share for MHA: #29959
• [GLM-5] Tune the threshold of router GEMM: #29470
• [Spec] Anchor GLM-5.2 MTP IndexShare topk on the draft-extend step: #29787
• Bypass legacy GLM DSA layer types validation: #29454
• docs: add B200 NVFP4 recipes + benchmarks to GLM-5.2 cookbook: #29674
• docs: add PD disaggregation to GLM-5.2 cookbook playground: #29544
• docs(cookbook): add AMD MI300X/MI325X/MI355X support for GLM-5.2: #28471
• Fuse the DSA (V3.2, GLM-5.x) indexer Q/K paths into single kernels: #27705 ⭐
• glm5.2 on ascend doc (new version): #29828
• Support JIT fused A GEMM (MLA down projection) and support GLM-5 hidden size, SM120: #27397
• Update GLM-5.2 B300 and GB300 NVFP4 cookbook settings: #29466
• Update GLM tests to 5.2 and delete redundant tests: #29686
• [AMD] [GLM5] Guard cuda_runtime.h for ROCm in fused_metadata_copy: #29373
• [AMD] [GLM5] Mark EAGLE verified on MI300X/MI325X (gfx942) in GLM-5.1 cookbook: #29313
• [AMD] [GLM5] GLM-5.1 MXFP4 (MI355X) + enable EAGLE for gfx950 in cookbook: #29194
• [AMD] [GLM5] Add opt-in Triton fp8 sparse-MLA prefill kernel for gfx950: #28975
• [AMD] [GLM5] skip redundant -inf pre-fill of HIP indexer MQA-logits: #28757

DeepSeek V4

• [DeepSeek V4] Enable FlashMLA sparse prefill by default: #29775 ⭐
• [DeepSeek-V4] Add an opt-in non-paged indexer for long-context prefill: #29619 ⭐
• [DSA] Fold page-table into fused top-k v2 (decode): drop page_size=1 expansion: #30274
• [JIT Kernel] DeepSeek-V4 DSA indexer: faster top-k + page-table transform (runtime k <= 2048): #26788 ⭐
• [Cherry-pick to release/v0.5.15] [DSA] Fix IMA in fused top-k v2: write all output slots on tie overflow (#30512): #30559
• [Cherry-pick to release/v0.5.15] [DSV4] perf: Make FP8 quant output tensor contiguous (#27926): #30449
• [Cherry-pick to release/v0.5.15] [DeepSeek-V4] Enable non-paged indexer by default for large prefill chunks (#30140): #30436
• [Cherry-pick to release/v0.5.15] [DSA] Re-enable fused top-k v2 for MTP: clamp padded-row seq_lens to >= 0 (#30378): #30427
• [AMD] Improve performance of DSV4 in high concurrency: #28938
• [AMD] DSV4 aiter reduce-scatter decode: #29103
• [AMD][DSV4] Remove per-batch D2H syncs in MTP to avoid bubbles between 2 batches: #29420
• [AMD][DeepSeek V4] Fix default FlashMLA sparse prefill off on ROCm/HIP: #29982
• [AMD] Fix DeepSeek V4 MTP accuracy issue: #30333
• [AMD] Fix dsv4 indexer dtype dispatch on gfx950: #29479
• [AMD] Cap DSV4 Flash max_total_num_tokens: #30313
• [AMD] Fix DeepSeekV4 serve...

Highlights

New Model Support: GLM-5.2, LiquidAI LFM2.5, Kimi-K2.7-Code, Poolside Laguna-M.1, DiffusionGemma, Zyphra ZAYA1, MiMo-V2-ASR

DeepSeek-V4 on GB300 since Day 0: 5x higher throughput at the same interactivity, serving DeepSeek-V4 on NVIDIA GB300 with SGLang (blog).

Waterfill & LPLB MoE load balancing: Two dispatch-time load-balancing methods for DeepEP expert parallelism: Waterfill for shared-expert dispatch and LPLB for redundant expert replicas, improving throughput for DeepSeek-V3/R1 and DeepSeek-V4 (blog).

KDA CuteDSL prefill kernel on Blackwell (SM100): New CuteDSL prefill kernel for Kimi-Linear (KDA), 1.08-1.52x faster than the Triton path via a reusable scratch workspace, plus a cuda-graph padding fix (#27488); see the Kimi-Linear cookbook.

Linear-attention prefix-cache memory savings: An int8 checkpoint pool stores recurrent states compactly in the Mamba radix cache, substantially increasing prefix-cache capacity for KDA / GDN models (#28185); the speculative conv-window intermediate cache is deduplicated with a sliding-window layout, halving its footprint with no numerical change (#28302).

LPLB: linear-programming load balancer for MoE expert parallelism: Balances token routing across redundant expert replicas by solving a per-layer LP; opt-in via --ep-dispatch-algorithm=lp, default behavior unchanged (#24515).

MSCCL++ integration & MNNVL allreduce fusion: MSCCL++ migrates to the upstream mscclpp Python package (Executor + DSL compiler) with auto-tuned collectives for TP=8 single-node and TP=16 two-node (#22734); FlashInfer fused allreduce + residual + RMSNorm re-enables an MNNVL backend behind --flashinfer-allreduce-fusion-backend (auto / trtllm / mnnvl), fixing the piecewise-CUDA-graph interaction (#23402).

Nemotron DP attention + MTP: Data-parallel attention for the hybrid Nemotron-H (Mamba2 + full attention + MoE), plus MTP support (#24955); see the Nemotron 3 Ultra cookbook.

AMD: breakable CUDA graph on ROCm/HIP: The breakable CUDA graph execution path now runs on AMD GPUs (#28173).

NVFP4 MoE for DeepSeek-V4: Adds an NVFP4 MoE quantization path for DeepSeek-V4 on Blackwell for higher MoE throughput; enable with --moe-runner-backend flashinfer_trtllm_routed (#25820); see the DeepSeek-V4 cookbook.

DeepSeek-V4 decode & quantization optimizations: FP8 group quantization now emits power-of-two (UE8M0) scales directly from the per-token group-quant kernel, dropping a separate rounding pass (#26766); MLA decode q-heads are padded to 64 under attention-TP so FlashMLA dispatches the ~2x cheaper head64 kernel instead of head128 (#27954); the MHC prenorm kernel is prewarmed at startup to remove the first-run JIT slowdown on a fresh server (#27986); and BF16 mixed-dtype compression states are supported on the C4 / C128 paths (#27277); see the DeepSeek-V4 cookbook.

Full release notes by category below.

New Model Support

• GLM-5.2: #28437 (cookbook)
• LiquidAI LFM2.5: #27409 (cookbook)
• Kimi-K2.7-Code: #28064 (cookbook)
• Poolside Laguna-M.1: #28661, #28400 (cookbook)
• DiffusionGemma: #27824 (cookbook)
• Zyphra ZAYA1: #26347 (cookbook wip)
• MiMo-V2-ASR: #26278 (cookbook wip)

DeepSeek V4

• [NVIDIA] Support NVFP4 MoE for DeepSeek-V4: #25820
• [DeepSeek-V4] Fuse UE8M0 scale rounding into FP8 group quantization: #26766
• [NPU] Add Ascend NPU support for DeepSeek-V4: #25144
• Deepseek v4: support mixed dtype compression states: #27277
• [AMD] Feat: Add prefill context parallel support for deepseek v4 unified kv attention: #27928
• DeepSeek-V4 Online Compress support MTP: #26471
• [dsv4] Pad MLA decode q-heads to 64 (not full n_heads) for FlashMLA head64 kernel: #27954
• [dsv4] Prewarm MHC prenorm kernel at startup: #27986
• [LoRA] Support DSA indexer LoRA targets for GLM-5.1 / DeepSeek-V3.2-family models: #28110
• Add DeepSeek V4 MTP acceptance length checks: #28098

Speculative Decoding

• [Spec] Add sync-free fast_prefill_plan for EAGLE draft-extend CUDA graph: #28854
• [Spec] Support FlashInfer CUDA graph for EAGLE draft-extend: #28782
• [mtp] add rejection sampling for speculative decoding: #26312
• [NPU] Add MTP support for GLM-4.7-Flash: #28516
• Dflash add sliding window attention draft layer support: #27469
• Support Nemotron DP attention and MTP: #24955
• [Feature] [Ngram spec] Support ngram spec v2: #17260

Piecewise & Breakable CUDA Graph

• [AMD] Make breakable CUDA graph run on ROCm/HIP: #28173
• Dflash piecewise cuda graphs support: #27468

Attention Backends

• [Cookbook] Nemotron3-Ultra: Add mamba-backend and SSM dtype flags: #28675
• [Mamba][GDN] Deduplicate spec conv-window intermediate cache via sliding window layout: #28302
• [GDN][KDA][mem_cache] int8 checkpoint pool for the linear-attn prefix cache: #28185
• [diffusion] feat: use LocalAttention for mistral3 encoder: #28176
• [NPU] Add Gemma4 Sliding Window Attention support on Ascend backend: #26147
• [AMD] Fuse sigmoid + mul attention output gate into single Triton kernel: #27630
• [AMD] Enable fused GDN QKV split Triton kernel on HIP: #27583
• [KDA] Add CuteDSL Prefill Kernel on SM100: #27488
• [AMD] Add unified kv attention support in dpsk-v4: #27380

MoE & Expert Parallelism

• Add GB10 FP8 fused MoE Triton config: #25665
• Support asymmetric compressed-tensors MoE: #27690
• LPLB: linear-programming load balancer for MoE expert parallelism: #24515
• [AMD] Fuse sigmoid + mul into single Triton kernel for shared expert gating: #27636
• [quantization] NVFP4 MoE: split fused w13 gate/up global scales: #27588
• [Apple Silicon] [MLX] Fuse SwiGLU activation into gate gather_qmv for SwitchGLU MoE blocks: #26188
• [DeepSeek V3] Defer moe finalize and fused it with main stream add: #27720

Quantization

• ✨ [llm][npu][quant] Add W8A8 MXFP8 quantization support for...

Highlights

New Model Support:

• Autoregressive: Nemotron 3 Ultra (Day-0, blog), Step-3.7-Flash, Command A+
• Diffusion: Cosmos3, LingBot-World, SANA-WM, Ernie-Image, FLUX.2-Klein 4B/9B, Ideogram 4

Spec V2 is now the default speculative-decoding path: Tree drafting with topk > 1 is production-ready across the triton / FA3 / MLA / aiter backends, including page_size > 1 and Mamba/hybrid-linear models (#26997, #26972, #27463). Spec V1 is deprecated, with EAGLE/MTP now running on the unified V2 worker (#25464), and topk = 1 drafting is faster (#26397, #26424).

Lower per-step scheduler overhead: Unified async value passing through FutureMap plus moving prefill input transfer onto the forward stream reduced per-step launch overhead and improved stability under high concurrency (#25945, #25879, #26380).

Piecewise & Breakable CUDA Graph coverage: Piecewise (PCG) and Breakable (BCG) CUDA Graph capture more of the model to cut per-step kernel-launch overhead, now extended to DSA models, Kimi-K2.5, and DeepSeek V4: #23351, #26382, #25195.

Faster Qwen 3.5 on Blackwell: New FlashInfer Gated DeltaNet (GDN) kernels and a CuTeDSL GDN prefill kernel speed up Qwen 3.5 on Blackwell GPUs: #22921, #23273, #26200.

HiCache for hybrid models by default: HybridModel (SWA/Mamba) launches HiCache through UnifiedTree by default, bringing hierarchical KV-cache offload to sliding-window and Mamba hybrids out of the box: #27759.

Heterogeneous CPU + GPU EPD disaggregation (with Intel): Offload VLM vision encoding onto Intel Xeon CPUs alongside GPUs, with up to ~1.3x P99 TTFT and request-throughput gains under load. (blog)

MoRI on AMD Instinct MI355X (with AMD): Cost-competitive DeepSeek-R1 disaggregated inference via AMD's MoRI communication library, $0.169 per million tokens at 129 tok/s/user. (blog)

DeepSeek V4 — context parallelism & sparse-attention kernels: Building on the v0.5.12 Day-0 path, v0.5.13 extends DeepSeek-V4 to context-parallel serving and adds its sparse-attention kernels:

• Context Parallel + MTP: #24934
• Context Parallel + fused MoE kernel (non-DeepEP): #24947
• Sparse FlashMLA via flash_mla_sparse_fwd: #25418
• FP4 indexer support: #26209
• SM120 support: #24692
• DeepEP waterfill load balancing: #25391
• MHC kernel warmup: #25810
• Breakable CUDA Graph for DeepSeek V4: #25195
• Backed by sgl-kernel 0.4.3 exposing sgl_kernel.flashmla: #26421, #26132

See the DeepSeek-V4 cookbook for tuned deployment commands.

SGLang-Diffusion — realtime & progressive resolution: OpenAI-style realtime video generation with msgpack frame streaming and a standalone browser WebUI (#26954, #26959), continuous camera controls + super-resolution controls (#27026, #27297), and progressive-resolution growing across FLUX / FLUX.2 / Qwen-Image / Wan / Z-Image (#27524).

Full release notes by category below.

New Model Support

• Nemotron 3 Ultra (Day-0, kernel optimizations): #26733 (see cookbook)
• Step-3.7-Flash: #26565 (see cookbook)
• Cosmos3-Nano / Cosmos3-Super — T2V / I2V / T2I (Diffusion): #24994, #26492, #26926, #26950 (see cookbook)
• Ernie-Image (Diffusion): #22439, #27195 (see cookbook)
• LingBot-World — realtime / causal-DMD generation (Diffusion): #26954 (see cookbook)
• SANA-WM — streaming + realtime (Diffusion): #27531 (see cookbook)
• FLUX.2-Klein 4B / 9B (Diffusion): #25661 (see cookbook)
• Ideogram 4 — FP8 / NVFP4 with tensor parallelism (Diffusion): #27279, #27379, #27393 (see cookbook)
• Command A+ (Cohere 2 family): #26106, #27401 (cookbook page pending)

DeepSeek V4

• Context Parallel + MTP: #24934
• Context Parallel + fused MoE kernel (non-DeepEP): #24947
• MHC kernel warmup: #25810
• SM120 support: #24692
• FP4 indexer support: #26209
• Integrate flash_mla_sparse_fwd kernel: #25418
• DeepEP waterfill support: #25391
• Breakable CUDA Graph for DeepSeek V4: #25195

Speculative Decoding

• Spec V2 is now the default speculative-decoding path
• Tree speculative drafting (topk > 1) on Spec V2 — page_size > 1 and Mamba/hybrid-linear, validated across triton / FA3 / MLA / aiter: #26997, #26972, #27463
• Spec V1 deprecated; EAGLE/MTP run on the unified V2 worker: #25464
• Spec V2 extended to the FlashMLA backend: #24640
• Adaptive speculative decoding: batch-size-aware num_steps + observability metrics: #24055, #25940
• Faster topk = 1 drafting (skip full-vocab softmax + redundant cat/topk/sort/gather ops): #26397, #26424
• Draft-extend CUDA Graph for the trtllm mha attention backend: #25002

Piecewise & Breakable CUDA Graph

• PCG support for DSA models: #23351
• PCG support for Kimi-K2.5: #26382
• BCG support for DeepSeek V4: #25195

Context Parallelism

• Support bs > 1 for prefill CP: #23269
• Prefill CP for MLA models (Kimi K2.5, DeepSeek V3): #23292

Attention Backends

• CuTeDSL MLA attention kernels from FlashInfer: #24737
• Qwen 3.5: Flash...

v0.5.12.post1 is a stability patch on top of v0.5.12. It cherry-picks 12 fixes — primarily for DeepSeek V4 — onto the release branch.

Bug Fixes

DeepSeek V4

• DSV4-Pro emits garbled text during single-token decode on B200/B300 (fix deep_gemm UE8M0 scale-packing path by ceiling activation scales before packing): #25733
• DSV4 + EAGLE/MTP in disaggregation decode crashes around 2000 requests with a SWA allocator assertion (recycled KV pages kept stale sliding-window mappings): #25805
• DSV4 NSA prefill context-parallel (--enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode round-robin-split) in --disaggregation-mode prefill: scheduler crash at startup: #25396
• DSV4 HiSparse + SGLANG_OPT_USE_COMPRESSOR_V2=1: GSM8K accuracy restored from 0.825 → 0.960: #25646
• DSV4 PD disaggregation now works with pipeline parallelism > 1 (removed stale pp_size=1 assertion): #25771
• DSV4-Flash with --load-format dummy + FlashInfer mxfp4 hits CUDA illegal memory access during CUDA-graph capture (the integer HashTopK.tid2eid lookup table was left uninitialized by dummy load): #25892
• DSV4 HiCache + SGLANG_OPT_CACHE_SWA_TRANSLATION=1 returns stale translation indices after a cache rebuild, causing OOB writes / wrong outputs: #25889

Disaggregation

• [PD][NIXL] Always send aux on is_last; only expect state when truthy: #25699

Other

• Fix missing group arg in get_dp_buffer: #25585

Performance

• DSV4: warm MHC token-count buckets at startup (gated to SGLANG_OPT_DEEPGEMM_HC_PRENORM=1 + SGLANG_OPT_USE_TILELANG_MHC_PRE=1 + hybrid SWA) to eliminate 20–40s cold-bucket forward stalls: #25810
• DSV4-Pro: precompile a DeepGEMM branch for _dispatch_bf16_fp32_backend to cut runtime JIT compile cost: #25860

Dependencies

• Use [cu13] extra for nvidia-cutlass-dsl (default to CUDA 13; required for sm_103 / B300): #25576

All PRs included in this release: v0.5.12...v0.5.12.post1

Full Changelog: v0.5.12...v0.5.12.post1

Highlights

• DeepSeek V4 support: Full inference path for DeepSeek-V4 (#23882), including:

  Day-0 Features: #23882

  • Parallelism: Tensor Parallelism/Expert Parallelism/Context Parallelism/Data Parallel Attention
  • Hardware: Nvidia B300/B200/H200/H100/GB200/GB300, AMD MI35X
  • Prefill-Decode Disaggregation
  • HiSparse for offloading inactive KV cache to CPU memory
  • Reasoning parser and Tool Call Parser
  • DeepGemm and FlashMLA kernels for DeepSeek V4, including MegaMoE

  Post-Day-0 additions:

  • HiCache for DeepSeek V4 under unified Radix Tree [UnifiedTree]: #24691
  • W4A4 MegaMoE kernels — faster speed with negligible accuracy drop: #25052
  • Marlin/FlashInfer W4A8 MoE kernels on Hopper: #24816 #24986
  • Faster V2 fused compression kernels: #24890
  • TP16 support on H100/H20: #24949
  • Fused SiLU+clamp+FP8 quant kernel: #24897
  • Optimized MHC + DeepGemm pipeline (fused norm, fused hc_head): #24775
  • Non-standard chat template support for DSv4: #23915
  • Multi-detokenizer support: #24944
  • Pipeline Parallelism + PD support for DeepSeek-V4: #24700
  • A unified docker tag lmsysorg/sglang:v0.5.12 for all Nvidia GPUs

  See the LMSYS blog and the DeepSeek-V4 cookbook for more details.

• TokenSpeed MLA attention backend (Blackwell, FP8 KV cache): New MLA prefill/decode kernels integrated as an attention backend on SM100, with FP8 KV cache support for low-latency MLA serving: #24925

• DSv3.2 / GLM-5 FP4 low-latency perf: PDL enabled across DSv3.2 / GLM-5 kernels, torch.mm for the DeepSeek V3.2 indexer GEMM, and a reland of the Cute-DSL FP4 dense GEMM — materially trimming low-latency overheads on FP4 paths: #23965, #23856, #23590, #25311

• New Model Support: DeepSeek V4 #23882, Intern-S2-Preview #24875, MiniCPM-V 4.6 #24855, Laguna-XS.2 #24204, Ring-2.6-1T #25360, and Gemma 4 MTP #24436 — with cookbook recipes for tuned deployment commands. See docs.sglang.io/cookbook

• HiCache + UnifiedRadixTree: HiCache framework support for UnifiedRadixTree (with SWA), HiCache for DeepSeek V4, SSD offload through Mooncake store, and stability fixes across cascade eviction, tombstone replay, and partial-match paths: #23316, #23391, #24691, #24277, #24943, #24972, #25068, #25277

• Speculative Decoding V2 maturation: Adaptive Spec V2, EAGLE-3 SWA + newer drafters, Kimi K2.5 EAGLE-3 MLA, Gemma 3/4 + EAGLE-3, and an extensive naming / shape-handling refactor across draft-extend paths: #23336, #24663, #24664, #24826, #23976, #24859

• CUDA 13 DeepEP migration: Gateway DeepEP source swapped from a community fork to deepseek-ai/DeepEP@hybrid-ep so DeepEP builds and runs cleanly on the CUDA 13 default; FlashInfer pinned at 0.6.11.post1 alongside a gpt-oss triton-kernel fix: #25113

New Model Support

Entries with a published cookbook recipe come first; entries whose cookbook page is still pending are grouped at the bottom.

• DeepSeek V4 (see cookbook; LMSYS blog)
• Intern-S2-Preview: #24875, #25115, #25134 (see cookbook)
• MiniCPM-V 4.6: #24855, #24876, #24991, #24998 (see cookbook)
• Laguna-XS.2 (Poolside): #24204, #24730 (see cookbook)
• Ring-2.6-1T (InclusionAI, trillion-param reasoning): #25360, #25370 (see cookbook)
• Gemma 4 MTP (MTP head for Gemma 4): #24436, #24433
• Trinity-mini (Ascend NPU, ~90% accuracy): #18172
• HunyuanVideo ModelOpt FP8 (Diffusion): #23199
• Qwen Image ModelOpt FP8 (Diffusion): #23155

Speculative Decoding

• TokenSpeed MLA prefill/decode kernels integrated as attention backend (FP8 KV cache, Blackwell): #24925
• Adaptive Spec V2 (2/N): #23336
• SWA support for EAGLE-3 drafter: #24664
• Support newer EAGLE-3 drafters: #24663
• Kimi K2.5 EAGLE-3 MLA spec decoding: #24826
• Gemma 3 / Gemma 4 + EAGLE-3 support: #23976
• Spec V1 — split draft-extend into EagleDraftExtendInput: #24859
• Custom speculative-algorithm registry: #23991
• Spec-V2 overlap stale-state fix: #23456
• trtllm decode kernel for draft extend: #24566
• AMD: EAGLE on Qwen3.5 FP8/MXFP4 via aiter unified attention: #23146
• Fix Kimi K2.5 MLA EAGLE + DP attention: #25033
• Fix ngram metric off-by-1 in num_accepted_drafts_per_req_cpu: #24965
• Fix frozen-KV MTP crash when bonus_tokens is None: #25204
• Fix stuck-MTP on DSA models: #24635
• Reduce specdec CPU overhead: #23321
• Spec-decoding naming-convention rule + refactors: #24094, #25014, #25038, #24081, #24724, #24735, #24881, #25010, #25012, #25030, #25029, #25037, #25109

PD Disaggregation

• DSv4 Flash disaggregation test: #24973
• Unify DSv4 dispatch with SWA: #24888
• DSv4 mooncake state_type branch: #24878
• Hybrid state transfer refactor: #24932
• Priority scheduling in PD mode fix: #25062
• NIXL: staging buffer for heterogeneous-TP KV transfer: #22536
• NIXL: async transfer: #23967
• NIXL XPU: uint64 pointer overflow + mismatched P/D TP fixes: #24188, #24648
• Mooncake: incremental transfer + SSD offload: #24257, #24277
• Multi-node prefill bootstrap-port broadcast: #24378
• Add retry-with-backoff for prefill bootstrap registration: #25125
• PrefillDelayer: NCCL all-gather for cross-DP info sync: #24768
• MORI-IO: state transfer + high-concurrency fixes: #22665
• Per-room cleanup centralization; prevent update_status from cleared entries; fix abort update_status across KV backends: #24601, #24539, #24522
• PD KV transfer metrics fix: #24416
• SWA memory preallocation for disaggregated decode: #24857
• IntraNode NVLink configuration docs: #23329

HiCache & Radix Cache

• HiCache framework for UnifiedRadixTree: #23316
• SWA HiCache for unified radix cache: #23391
• HiCache for DeepSeek V4 + nightly CI for DSA model: #24691, #25369, #25348
• SSD offload through Mooncake store: #24277
• HiSparse FP8 KV cache via flashmla_kv backend: #23013
• Default storage prefetch timeout: #23309
• UnifiedRadixCache device match semantics with HiCache: #25277
• UnifiedTree partial match on evicted+backuped nodes: #24943
• UnifiedTree tombstone lock release replay fix: #24972
• UnifiedTree _cascade_evict leaf determination fix: #25068
• UnifiedRadixTree align cache_empty_result with RadixTree: #24779
• Mamba radix cache KV events; SWA radix cache events: #23678, #24718
• SWA chunk req deferred fix; SWA component host hit fix: #24318, #25085

LoRA

• MLA attention LoRA (q_b_proj / kv_b_proj): #25001
• CSGMV backend with virtual experts for MoE LoRA: #24007
• MoE LoRA: remove CPU-GPU sync barriers and duplicate code (prefill optimize 2/n, 3/n): #24246, #24262
• LoRADrainer for high P99 TTFT: #17913
• qkv_proj buffer sizing when tp_size > num_key_value_heads: #24420
• Torch-Native LoRA: embedding + graph optimization: #21885
• Deterministic lora_id for multi-node --lora-paths: #24555
• Fix broken sgemm_lora_a_graph_fwd due to invalid torch.mm(): #24760
• Diffusion: fix RowParallel LoRA merged forwarding: #24410

Performance

• TMA bulk-store set_mla_kv_buffer (up to 12× over baseline): #25311
• Kimi tokenizer TTFT optimization: #25265
• Avoid hidden-states D2H copy when return_hidden_states=false: #25155
• DeepseekV2MoE: defer shared experts when routed kernel is non-mutating: #25279
• SGLANG_OPT_FP8_WO_A_GEMM on by default: #25181
• --prefill-only-disable-kv-cache to skip KV pool allocation: #23675
• Gemma 4 MoE: fused Q/K/V RMSNorm + per-expert FP8 ckpt loader: #24696
• Gemma 4 VLM: PCG + fused RMSNorm + residual: #24048
• MHC pipeline: DeepGemm + fused norm + fused hc_head: #24775
• JIT custom all-reduce default; non-NVL follow-up: #24363, #24742
• SGLANG_USE_JIT_ALL_REDUCE → SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2: #24297
• Eliminate logits H2D blocking copy: #24627
• Cache empty MatchResult in RadixCache: #24470
• Breakable CUDA graph for bs > 1: #24662
• FA3: skip scheduler_metadata precompute under DP attention: #24632
• aten::rms_norm / aten::mm.dtype registration in batch-invariant mode: #24459
• Optimize Helios fused norm modulation: #24059
• Z-Image packed QKV optimization: #24117
• KDA prefill kernels: diagonal + recompute fuse: #24271

Observability

• sglang:get_loads_duration_seconds Prometheus metric: #25163
• Per-iteration forward-pass metrics via ZMQ PUB: #22789
• SGLANG_TRACE_LEVEL env for startup trace level: #24716
• fwd_occupancy metric in SchedulerStats + Prometheus collector: #24458
• SWA / Mamba cache metrics: #24396
• Mamba radix cache + SWA radix cache KV events: #23678, #24718
• PD KV transfer metrics fix: #24416
• CP allgather buffer registered with symmetric memory: #24040
• Decode-side bootstrap/alloc metrics + non-int token-id filter: #24684

Frontend & API

• /v1/tokenize chat-completion-style support: #23981
• Multi-detokenizer support: #24944
• Structural tags for strict tool calling & reasoning across more models: #21722
• Auto-detect reasoning / tool-call parser from chat template: #23952
• Two-phase reasoning grammar + --enable-strict-thinking: #23953
• OpenAI reasoning.enabled mapping to thinking + enable_thinking: #23951
• Kimi-K2.5 bare-numeric tool-call IDs: #23950
• Crusoe managed-inference backend: #20475
• Azure Blob Storage connector (az:// and *.blob.core.windows.net): #23995
• Adaptive queue-based prefill-delayer trigg...

Highlights

• CUDA 13 + Torch 2.11: Default CUDA version moves to 13.0 across SGLang, sgl-kernel, and Docker images, and PyTorch is upgraded from 2.9 to 2.11 — modernizing the build matrix and unlocking newer kernels: #21247, #24162, #24183, #23593 (tracking issue #21498)

• Speculative Decoding V2 by default: Spec V2 (with overlap scheduling to hide CPU overhead) is now the default, materially reducing per-step CPU cost for EAGLE/MTP/DFLASH paths: #21062

• Decode Radix Cache for PD Disaggregation: Decode-side prefix caching now works under prefill/decode disaggregation, recovering radix-cache hit rates and TTFT savings for long shared prefixes in disaggregated deployments: #19746

• Day-0 / New Model Support: Gemma 4, GLM-5.1, Qwen3.6, MiMo-V2.5 / V2.5-Pro, Ling-2.6-Flash, Mistral Medium 3.5, and Kimi-K2.6 — with cookbook recipes for tuned deployment commands. See docs.sglang.io/cookbook: #21952, #23808, #23811, #23851, #23947, #23486, #23394

• DFLASH Speculative Decoding: New high-throughput spec-decode kernel from the kernel community, expanded across model backends and AMD ROCm: #22077, #22358, #22342, #23553

• FA3 Kernels from the Kernel Community: Drop-in FA3 kernels contributed by the community, integrated alongside FA4 to give users a high-performance option that's easy to maintain: #20796

• LoRA support for DeepSeek-V3 and Kimi-K2: LoRA now works on the largest MLA-based MoE models, including DeepSeek-V3 MLA LoRA and Kimi K2 — enabling adapter-based fine-tuning of frontier-scale models: #22323, #22381

• Context Parallel (CP) Enhancements: All-reduce + RMSNorm fusion under CP for end-to-end speedups, plus support for moe_dp_size = 1 paired with arbitrary attention_cp_size so MoE and attention parallelism can be tuned independently: #21249, #22003

• FlashInfer CuteDSL MoE Runner Backend: New dedicated FlashInferCuteDslMoE layer for the standard FP4 MoE path, giving an additional high-performance fused-MoE option: #21339

New Model Support

Entries with a published cookbook recipe come first; entries whose cookbook page is still pending are grouped at the bottom.

• Gemma 4: #21952 (and follow-ups #22079, #24048, #22842; see cookbook)
• GLM-5.1: #22543, #23037 (see cookbook)
• Qwen3.6: #23486 (see cookbook)
• MiMo-V2.5 / MiMo-V2.5-Pro: #23808, #23811, #23851, #23945, #24118 (see cookbook)
• Ling-2.6-Flash: #23947 (see cookbook)
• Mistral Medium 3.5: see cookbook
• Kimi-K2.6: #23394, #23408 (see cookbook)
• Hunyuan v3 (Tencent, preview): #23533 (see cookbook)
• FLUX.1-dev ModelOpt NVFP4 (Diffusion): #22672 (see FLUX cookbook)
• FLUX.2-small-decoder (Diffusion): #22414 (see FLUX cookbook)
• Qwen Image ModelOpt FP8 (Diffusion): #23155 (see Qwen-Image cookbook)
• LTX-2.3 / LTX-2.3 two-stage / TI2V (Diffusion): #22182, #22667, #22869 (see LTX cookbook)
• Qwen3-ASR (chunk-based streaming): #22073, #22089
• Voxtral (Mistral speech-to-text): #21635
• Parakeet (NVIDIA Nemotron encoder): #23568
• Moss-VL: #23454
• SequenceClassification model architecture (powers the Score API): #22118
• Stable Diffusion 3 medium (Diffusion): #19225
• ERNIE-Image (Diffusion): #22439
• JoyAI-Image-Edit (Diffusion): #22625

Speculative Decoding

• DFLASH speculative decoding initial support: #22077
• DFLASH enabled across additional model backends: #22358
• DFLASH speculative decoding on AMD ROCm: #22342
• Spec V2 enabled by default with overlap scheduling: #21062
• Penalty support for Spec V2 overlap scheduling: #22049
• Adaptive speculative_num_steps for EAGLE topk=1: #21599
• Allow piecewise CUDA graph with speculative decoding: #22128
• Eagle3 / DFLASH aux hidden state capture during CUDA graph init fixed: #22836
• Split accept_length into num_accepted_drafts / num_accepted_tokens: #23962
• DFLASH speculative decoding documentation: #23553

PD Disaggregation

• Decode-side radix cache support: #19746
• Incremental transfer for Mooncake transfer engine: #24257
• Allow PrefillDelayer in disaggregated-prefill mode: #23588
• NIXL: heterogeneous TP KV transfer for non-MLA models (Step 1/2 for Qwen3.5): #22145
• NIXL: Mamba state slice transfer for heterogeneous TP (Step 2/2 for Qwen3.5): #22240
• Bug fixes for IntraNode NVLink, MTP-layer KV transfer, and disagg-prefill DP rank resolution: #23252, #23539, #22901, #22990

Context Parallel & Parallelism

• All-reduce fusion support under CP: #21249
• moe_dp_size = 1 paired with arbitrary attention_cp_size: #22003
• All-reduce fusion enabled for DSA models: #22390
• Replace all-reduce + dp_scatter with reduce_scatterv for DP attention: #22642
• Step3p5: optimize all-reduce in MoE layers: #22773
• Pipeline parallelism on Intel XPU: #23472
• OpenTelemetry tracing for pipeline parallelism: #23169

LoRA

• DeepSeek-V3 MLA LoRA support and quantization-info refactor: #22323
• Kimi K2 LoRA support: #22381
• LoRADrainer to address high P99 TTFT: #17913
• Decoupled LoRA MoE backend with Marlin support: #21858
• Virtual experts for LoRA MoE (1/n): #22122, #24007
• CSGMV kernel offline auto-tuning: #20391
• Triton sgemm speedup with better grid selection: #22386
• Dual MoE CUDA graph capture for lora/nolora batches: #22809

Performance

• FA3 kernels from the kernel community: #20796
• Precompute FA3 scheduler_metadata to eliminate per-layer prepare cost: #21104
• Precompute gemma_weight to avoid redundant add on every forward: #22673
• Eliminate attention DtoD copy by passing pre-allocated output to FA: #21985
• Skip KV cache in FA backend for embedding mode: #21971
• O(1) RadixKey view for EAGLE bigram key: #23106
• PCG inductor path optimization for FP8 models: #23227
• Combo-kernels for horizontal fusion: #21977
• Optimize Gemma4 VLM with PCG and fused RMSNorm + residual add + scalar: #24048
• Restore torch.compile fusion for topk postprocessing: #21771
• Reduce unnecessary kernels and copies in the NSA indexer: #22232

Observability

• Pending token count surfaced in prefill log and get_load: #22480
• OpenTelemetry tracing for speculative decoding: #19545
• OpenTelemetry tracing for pipeline parallelism: #23169
• OpenTelemetry tracing in DiffGenerator: #21254
• Prometheus metrics endpoint for gRPC mode: #20801
• HTTP sidecar endpoints and FlushCache gRPC RPC for gRPC mode: #22500
• Raw KV cache pool token counts as Prometheus gauges: #22726

SGLang-Diffusion

• New model support: LTX-2.3 (#22182, #22667, #22869), ERNIE-Image (#22439), FLUX.2-small-decoder (#22414), JoyAI-Image-Edit (#22625), FLUX.1-dev ModelOpt NVFP4 (#22672), Qwen Image ModelOpt FP8 (#23155), Stable Diffusion 3 medium (#19225)
• ModelOpt diffusion FP8 support for Flux1/Flux2 and Wan2.2: #22365
• Standalone Rollout API + Denoising Environment Backpass + SP-Aligned Log-Prob for T2I post-training: #22604
• Disaggregated diffusion: #21701
• Dynamic batching v0: #18764
• CPU platform support for SGLang Diffusion: #20816
• AITER backends in Flux 2 pipeline (AMD): #22802
• LTX-2 feed-forward tensor parallelism optimization: #23221
• In-memory loading for URL/base64 image inputs (default): #23118
• Mixed-resolution benchmark support: #20863
• Auto-enable best parallel setting if unspecified: #22763

AMD

• MiniMax-M2.5 optimizations (aiter biased grouped topk; fused FP8 KV cache write): #23611, #23620
• Fused QK Gemma norm kernels (4 → fewer kernels): #23575
• Fused all-reduce + RMSNorm simplification: #21986
• GLM-5 / GLM-5.1 MXFP4 nightly accuracy + perf benchmarks (MI30x / MI35x): #21773, #22336
• MTP for GLM-5-mxfp4: #23219
• Aiter v0.1.12.post1 upgrade: #22264
• DFLASH speculative decoding enabled on ROCm: #22342
• Fix --page-size > 1 memory access fault with speculative decoding: #23596

NPU / Ascend

• Ascend backend supports Qwen3 MoE attention CP: #21685
• GLM-4.5V and GLM-4.7-Flash NPU support / fixes: #22961, #22509
• MTP for Qwen3.5: #20918
• TP communications compression for Qwen3 on NPU: #20520
• Add support-new-models documentation for NPU: #23824
• GGUF quantization for Ascend NPU (dense + MoE): #17883

CPU

• GPTQ / AWQ 4-bit quantization on CPU: #22685
• gemma4_rmsnorm_cpu kernel: #22842
• Qwen3.5 model optimization for CPU: #19484
• Apply routed scaling factor on output for biased grouped topk fusion: #22413
• Fix extend_attention_cpu / flash_attn_varlen_func NaN for large seq: #22434

Quantization

• MXFP4 quantized dense models on AMD CDNA2/CDNA3 GPUs: #19143 (later reverted in #23031, follow-up forthcoming)
• NVFP4 KV cache: quantization strategy abstraction and kernel: #21954
• DeepSeek-R1-0528-w4a8 + DeepEP Low-Latency FP8 dispatch: #22316
• MXFP8 sm100 path cleanup: #21881
• GLM-5/5.1 MXFP4 checkpoint inference compatibility fix: #22543

Dependencies

• Torch upgraded 2.9 → 2.11: #21247
• Default CUDA bumped to 13.0 across sglang, sgl-kernel, and Docker images: #21498 (tracking), #24162, #24183, #23593, #23119
• Flashinfer 0.6.7.post2 → 0.6.8.post1: #23281
• sgl-kernel bumped to 0.4.1.post1: #2...

Full Changelog: v0.5.10...v0.5.10.post1

Bumps flashinfer from v0.6.7.post2 to v0.6.7.post3 to resolve an issue in its jit cubin downloader.