要点速览
- Qwen3.5 是 full-attention + GDN 混合架构，在 GB200 NVL72 上用 disaggregated serving（N×DEP2 prefill + 1×DEP8 decode）服务 Qwen3.5-397B-A17B-NVFP4。
- 三大关键优化：Blackwell 优化的 GDN prefill kernel（FlashInfer）、HMA 混合 cache 与 GDN state 的 NIXL 传输、async scheduling 的两个竞态修复。
- 在 ISL/OSL=8192/1024、并发 64–5120 下 Total TPS/GPU 达到 25,000；GSM8K 精度 88%，与聚合部署一致。
- prefill 侧 --max-num-batched-tokens 16384（2× ISL）让每步批两个完整 prompt，高并发下值约 +8% 总吞吐；async scheduling 是跨过 25K 的关键特性。
- 全部配方开源在 srt-slurm-recipes 仓库，srtctl run 一条命令即可复现，含精度与各并发点的变体。
解决什么问题
Qwen3.5 于 2026 年初发布且仍是客户使用最广的模型之一，其混合注意力架构（full-attention 层 + GDN 层）在 disaggregated 模式下带来两类额外难题：如何在 Blackwell GPU 上加速 GDN 计算，以及如何在 prefill 与 decode worker 之间正确且高效地传输异构的 attention/GDN 状态。文章总结社区把该 disaggregated 路径做熟的完整过程：关键 PR、GB200 NVL72 上的最新性能（25K Total TPS/GPU）以及可复现的配方与最佳实践。
核心思路
社区沿三条线推进：FlashInfer 新增 Blackwell 优化的 GDN prefill kernel 并由 vLLM 在支持的配置上自动启用（PR #40717 集成）；通过 HMA+NixlConnector（PR #35758）与混合 SSM-FA disaggregation 主 PR #36687 及一串后续，支持 full-attention KV cache 与 Mamba/GDN 状态经 NIXL 跨节点传输，PR #41869 把该路径扩展到 Qwen3.5 的 GDN 层；修复 async scheduling 在 PD + hybrid attention 下的两个 KV 块传输竞态（此前会令精度归零）。性能上固定 decode 端点为单 DEP8（8×GB200），扫描 4–8 个 DEP2 prefill 端点与 64–5120 并发，组合出 Pareto 前沿，先用 GSM8K 验证精度，再把全部配置沉淀为 srt-slurm recipes。
关键要点
- Blackwell 优化的 GDN prefill kernel — FlashInfer PR #3001 相比旧的 FLA/Triton 实现提升约 1.02×–5.78×（覆盖各 Qwen3.5 模型规模、TP 配置、序列长度与批次形状）。vLLM PR #40717 在 prefill 侧启用该 kernel：8×B200 跑 Qwen3.5-397B-A17B-NVFP4 的微基准中 GDN kernel 最高快 5.92×，prefill-only（ISL/OSL=8192/1）端到端吞吐 1.13×，mean TTFT 降低 12%。GDN backend 设为 auto 时 vLLM 在支持的 Blackwell 配置上自动选择 FlashInfer 路径，也可用 --gdn-prefill-backend flashinfer 显式指定。
- 混合 cache 与 GDN 状态的跨节点传输 — PR #35758（HMA+NixlConnector）把 HMA 的逻辑块映射到正确的物理内存区域，使 NIXL 只传输各层类型对应的 cache 区域，传输描述符从 4,284 降到 1,650，小规模节点内 H100 上吞吐最高提升约 7%。但 Mamba 类状态在布局、大小与传输语义上差异足够大，仅有 HMA 支持不足以正确高效地做 P/D serving；主 PR #36687 增加 dual descriptor views 与 homogeneous-TP 支持，使 prefill/decode worker 能同时传输 full-attention KV cache 与 Mamba 风格 SSM 状态。后续栈包括 #37416（conv state 不同布局）、#37635（异构 TP 的 3-read conv state 传输）、#37310（N-1 prefill），PR #41869 最终把该路径扩展到 Qwen3.5 的 GDN 层。
- 无竞态的 async scheduling — 两个补丁修复了 KV 块传输中的竞态条件，此前这些竞态使 async scheduling 完全不可用——开启后精度直接归零。PR #48481 修复混合注意力模型 PD async 调度竞态；PR #45357 把块释放推迟到 in-flight step 完成后再执行（针对 async scheduling + PD KV consumer 组合）。Async scheduling 是跨过 25K tok/s/GPU 的关键特性之一，两个竞态都必须解决才能安全启用。
- 性能结果与环境 — 测量在经 NVLink72 互联的 GB200 集群上进行，模型为 Qwen3.5-397B-A17B-NVFP4，ISL/OSL=8192/1024，随机数据集 random_range_ratio=0.8。decode 侧固定单端点 DEP8（8 张 GB200），prefill 侧评估 4–8 个 DEP2 端点；并发从 64 扫到 5120——1–32 的低并发未测（本文聚焦 Pareto 曲线左半的 Total TPS/GPU），超过 5120 则因 decode 侧 KV cache 耗尽而止（decode 端点刻意固定，继续推高需加 decode GPU）。最终 Total TPS/GPU 达 25,000，GSM8K 五种配置全部 88%。
- 关键配方设置（一）：SSM 状态与调度 — VLLM_SSM_CONV_STATE_LAYOUT=DS 对 SSM 模型的 disaggregated serving 是必需的，没有它 conv state 传输无法工作。--async-scheduling 是达到 25K tok/s/GPU 的关键特性之一，要求 vLLM 构建已包含上述竞态修复。--mamba-ssm-cache-dtype bfloat16 显著增加 decode 端的有效 KV cache 容量。--language-model-only 针对纯文本负载：Qwen3.5 是多模态模型，该 flag 不仅关闭多模态输入，还解锁注意力层中 fused QK-norm + RoPE + gate 路径。
- 关键配方设置（二）：批处理与 CUDA graph — prefill 侧 --max-num-batched-tokens 16384（即 2× ISL）：prefill 端点较少（4–6×DEP2）时 prefill 成为瓶颈、decode 侧在峰值吞吐以下空转，让每个 prefill step 批两个完整 prompt 在高并发下值约 +8% 总 TPS/GPU。decode 侧在两个最高并发点把 --max-cudagraph-capture-size 提到 cc/8 + 128（cc=4096 时 640、cc=5120 时 768，8 为 decode 端 DP rank 数）；vLLM 默认上限 512 够用到 cc=3072，作者不确定 Pareto 数字是否真需要提高上限，仅作保险。随机数据集上 prefix caching 无收益故关闭。
- 实用排障与日志技巧 — --stream-interval 100 降低高并发下的前端开销，但会按 100 token 块缓冲流式输出、影响测得的 per-token latency——优化 ITL/TPOT 时要记住这一点。--api-server-count 1 在排查特定配置时非常有用：DP 端点上 vLLM 默认把 API server 数设为 DP size，多于一个时会关闭默认统计日志；强制为 1 可恢复每 10 秒（VLLM_LOG_STATS_INTERVAL 可配）打印 prompt/generation 吞吐与 KV cache 利用率，没有这些指标很难定位各配置的瓶颈。环境变量 DYN_LOG=error、DYN_SDK_DISABLE_ANSI_LOGGING=1、VLLM_LOGGING_COLOR=0 可大幅削减 Dynamo 的海量日志并抑制 ANSI 转义序列，否则日志几乎不可读。
- 后续方向 — 已有测量集中在 Pareto 曲线左半（最大化 Total TPS/GPU）。下一步计划扫出最大化 Gen TPS per user 的 PD 配置，这需要从 DEP 拓扑转向 TEP（Tensor Parallel + Expert Parallel）或纯 TP——它们通常带来更好的每用户性能；增加使用中的 GPU 数量是另一个预期有效的杠杆。
关键数据
| 指标 | 数值 | 条件 |
|---|---|---|
| Total TPS per GPU | 25,000 tokens/s | Qwen3.5-397B-A17B-NVFP4 / GB200 NVL72 / ISL/OSL=8192/1024 / 并发 64–5120 |
| GDN prefill kernel 相对旧实现的提升 | 约 1.02×–5.78× | FlashInfer Blackwell GDN kernel vs FLA/Triton，跨模型规模/TP 配置/序列长度/批次形状 |
| GDN kernel 微基准性能 | 最高 5.92× | 8×B200 / Qwen3.5-397B-A17B-NVFP4 / vLLM 集成 FlashInfer 路径后 |
| 端到端 prefill 吞吐提升 | 1.13× | prefill-only 负载 ISL/OSL=8192/1 |
| mean TTFT 降低 | 12% | prefill-only 负载 8K/1 |
| NIXL 传输描述符数量 | 4,284 → 1,650（吞吐最高 +7%） | HMA+NixlConnector / 小规模节点内 H100 环境 |
| GSM8K 精度 | 88% | 全部五种 PD 配置，与聚合（非分离）Qwen3.5 部署一致 |
| prefill 批两个完整 prompt 的吞吐收益 | 约 +8% Total TPS/GPU | 4–6×DEP2 prefill 端点成瓶颈时，--max-num-batched-tokens 16384（2× ISL） |
| 并发扫描范围 | 64–5120 | decode 固定单 DEP8 端点；超过 5120 时 decode 侧 KV cache 耗尽 |
| max-cudagraph-capture-size | 默认 512（够用到 cc=3072）；640（cc=4096）/ 768（cc=5120） | decode 端 DEP8，最高两个并发点按 cc/8 + 128 上调 |
“Total TPS per GPU reaches **25,000** tokens per second.”
“Compared with the previous FLA/Triton implementation, the new GDN kernel improves performance by approximately 1.02× to 5.78× across Qwen3.5 model sizes, tensor-parallel configurations, sequence lengths, and batch shapes.”
“Up to 5.92× higher GDN kernel performance in the tested microbenchmarks.”
前置知识
P/D disaggregation 基本原理与 KVConnector/NIXL 传输栈 · 混合架构模型：full-attention 层与 GDN/Mamba 类 SSM 层的状态差异 · MoE 并行拓扑：DEP（Data Parallel + Expert Parallel）与 TEP 的区别 · CUDA graph 捕获大小（max-cudagraph-capture-size）与并发的关系 · srt-slurm/Dynamo 部署工具链与 GSM8K、sa-bench 等基准 · TTFT、Total TPS/GPU 等推理性能指标与 Pareto 前沿概念
术语表
GDN: Gated Delta Network，线性注意力/SSM 类层，状态按 request 存储、不随 token 数增长。DEP: Data Parallel + Expert Parallel 拓扑组合（如 prefill 侧 DEP2、decode 侧 DEP8），利于聚合吞吐。TEP: Tensor Parallel + Expert Parallel 拓扑组合，通常比 DEP 带来更好的每用户性能。HMA: Hybrid (Mamba) Attention，vLLM 中管理混合 SSM+attention 模型缓存、映射逻辑块到物理内存区域的机制。NIXL: vLLM 用于跨节点传输 KV cache 与 SSM/GDN 状态的连接器与传输库。NVFP4: NVIDIA 4-bit 浮点量化格式，本文用于 Qwen3.5-397B-A17B 的权重量化。GB200 NVL72: 72 张 Blackwell GPU 经 NVLink 互联的超节点系统，本文性能测量环境。ISL/OSL: Input/Output Sequence Length，输入/输出序列长度，本文为 8192/1024。TTFT: Time To First Token，首 token 时延，prefill 性能的直接体现。Async scheduling: vLLM 的异步调度特性，重叠 CPU 调度与 GPU 计算，是达到 25K tok/s/GPU 的关键之一。Pareto frontier: 吞吐与延迟权衡下的有效配置集合；本文聚焦其左半（最大化 Total TPS/GPU）。srt-slurm: NVIDIA 推理部署编排工具，配方可经 srtctl run 一条命令复现，本文全部配置开源于此。Dual descriptor views: PR #36687 引入的机制，让 NIXL 能以两种视图同时描述 full-attention KV cache 与 Mamba 风格 SSM 状态。
vLLM Reaches 25K Total TPS/GPU on Qwen3.5
Introduction
Qwen3.5 was released in early 2026 and remains one of the most widely used models among customers. Because of its novel hybrid attention architecture, serving it in disaggregated mode brings in additional challenges and performance optimization opportunities. Thanks to the continuous contributions from the vLLM community, the disaggregated serving path for Qwen3.5 is now mature, and we are excited to report on the major contributions, latest performance on GB200 NVL72 systems, and recipes and best practices for you to reproduce. In this blog post we show how you can get over 25K total TPS/GPU performance.
Challenges and Key Optimizations
Qwen3.5's hybrid architecture combines full-attention layers with Gated Delta Network (GDN) layers. This creates two distinct optimization challenges: accelerating GDN computation on Blackwell GPUs and transferring heterogeneous attention/GDN state correctly between prefill and decode workers.
SSM support for P/D serving was driven by the vLLM community through the NIXL disaggregation roadmap. For a deeper discussion of heterogeneous cache layouts, logical and physical block mapping, and tensor-parallel state transfer, see the detailed hybrid SSM disaggregation blog post. We highlight the following contributions as particularly important to Qwen3.5 performance.
1. Blackwell-Optimized GDN Prefill
FlashInfer: Add Blackwell GDN prefill kernel #3001
Compared with the previous FLA/Triton implementation, the new GDN kernel improves performance by approximately 1.02× to 5.78× across Qwen3.5 model sizes, tensor-parallel configurations, sequence lengths, and batch shapes.
The kernel was subsequently enabled on the prefill side in vLLM by vLLM PR #40717. On an 8×B200 system running Qwen3.5-397B-A17B-NVFP4, the vLLM integration delivered:
- Up to 5.92× higher GDN kernel performance in the tested microbenchmarks.
- 1.13× higher end-to-end prefill throughput on a prefill-only workload (ISL/OSL = 8192/1).
- A 12% reduction in mean TTFT (prefill-only workload 8K/1).
On supported Blackwell configurations, vLLM automatically selects the FlashInfer path when the GDN backend is set to auto. It can also be requested explicitly with:
--gdn-prefill-backend flashinfer
2. Hybrid Cache and GDN-State Transfer
Disaggregated serving for hybrid SSM-attention models builds on [Core][KVConnector] Support HMA+NixlConnector #35758 and a stack of connector changes described in the hybrid SSM disaggregation blog post. This PR is a necessary prerequisite: it maps HMA's logical blocks onto the correct physical memory regions so NIXL can transfer only the cache regions belonging to each layer type, reducing transferred descriptors from 4,284 to 1,650 and improving throughput by up to approximately 7% in a small-scale intra-node H100 setup. But Mamba-style state differs enough in layout, size, and transfer semantics that HMA support alone would not have been sufficient for correct or efficient P/D serving.
The main PR for hybrid SSM-FA disaggregation is [PD][Nixl] Add support for hybrid SSM-FA models #36687, which adds dual descriptor views and homogeneous-TP support so prefill and decode workers can transfer both full-attention KV cache and Mamba-style SSM state over NIXL. Related follow-ups in the same stack include:
- [Kernel] Mamba support different layout for Conv state #37416
- [NIXL][Mamba][3/N] Heterogeneous TP: 3-read conv state transfer #37635
- [SSM/Mamba] Follow-up: N-1 prefill for P/D disaggregation #37310
See the hybrid SSM disaggregation blog post for how dual descriptor views, physical/logical block bridging, and conv-state transfer fit together.
For Qwen3.5 specifically, PD disagg with NIXL Connector: GDN support (Qwen3.5) #41869 extends this path to GDN layers.
3. Race-Free Async Scheduling
These two patches fix race conditions in KV block transfer that made async scheduling unusable — accuracy collapsed to zero with it enabled. Async scheduling turned out to be one of the key features behind crossing 25K tok/s/GPU, so both races had to be resolved.
- [KV Connector] Fix PD async scheduling race condition for hybrid attn models #48481
- [Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer #45357
Performance
1. Environment Setup
Measurements were conducted on a GB200 cluster connected via NVLink72. We used ISL/OSL = 8192/1024. The evaluated model was Qwen3.5-397B-A17B-NVFP4. Performance was measured on a fixed decode topology and a constant number of decode endpoints. In this setup, the decode side used one endpoint with DEP8 (Data Parallel + Expert Parallel across 8 GPUs). On the prefill side, we evaluated configurations ranging from 4 to 8 endpoints, each using a fixed DEP2 topology.
To reproduce the results, use the latest vLLM vllm/vllm-openai:nightly-d223c90 Docker image, Dynamo 1.2.0.dev20260526, and srt-slurm v1.0.32. All recipes used in this article are available in the srt-slurm-recipes repository.
Accuracy Results
First, we measured accuracy for all serving configurations to ensure that the performance results are valid. For this purpose, we used the standard GSM8K (Grade School Math 8K) benchmark. Running GSM8K with srt-slurm is straightforward. To enable it, add the following benchmarking block to your recipe file:
benchmark:
type: "gsm8k"
Accuracy results for all five configurations are 88%, which matches the accuracy we observe for the aggregated Qwen3.5 run.
2. Comments on Recipe Settings Choice
For performance measurements we used a fixed input sequence length / output sequence length benchmark, on a random dataset with random_range_ratio=0.8. The recipes themselves, and the settings that matter most, are covered in Recipes & best practices below.
3. Performance Results
Pareto curves for the individual configurations are shown in Figure 1, and the final Pareto frontier obtained after combining all configurations is shown in Figure 2. Total TPS per GPU reaches 25,000 tokens per second. Concurrency was swept from 64 up to 5120. We did not measure low concurrencies in the range of 1 to 32, since our focus here was the left part of the Pareto curve — maximizing the total TPS per GPU metric. At the other end, we did not go beyond 5120 because that is where we started running out of KV cache capacity on the decode side, which we deliberately fixed at a single 8×GB200 endpoint throughout these measurements. Pushing concurrency higher is entirely possible, but it requires adding GPUs on the decode side.
Recipes & best practices
All recipes used in this article live in the srt-slurm-recipes repository, and each one is launched with a single command:
srtctl run --file <recipe>.yaml
The naming scheme is NxDEP2-1xDEP8, where N is the number of prefill endpoints running DEP2 against a single DEP8 decode endpoint; there are five base configurations, from 4×DEP2 to 8×DEP2. Each comes with three derived variants: the base file sweeps sa-bench over concurrencies 64…3072, the -acc variant runs GSM8K five times on the same topology, and -cc4096 / -cc5120 each capture a single high-concurrency point with the decode-side max-cudagraph-capture-size raised to 640 and 768 respectively.
Most settings in the recipes are standard and shared across all configurations, but several are worth calling out:
VLLM_SSM_CONV_STATE_LAYOUT=DS— mandatory for SSM models in disaggregated serving; conv-state transfer does not work without it. Our recipes also passed--no-disable-hybrid-kv-cache-manager; HMA has since been enabled by default in vLLM for several versions, so that flag is no longer required.--async-scheduling— one of the key features behind reaching 25K tok/s per GPU. It requires a vLLM build that already contains the race-condition fixes discussed above.--mamba-ssm-cache-dtype bfloat16— significantly increases the effective KV cache capacity on the decode endpoint.--language-model-only— Qwen3.5 is a multimodal model, and for a purely textual workload this flag not only disables multimodal inputs but also unlocks the fused QK-norm + RoPE + gate path in the attention layers.--max-num-batched-tokens 16384on the prefill side, i.e. 2× ISL. With fewer prefill endpoints ({4, 5, 6}×DEP2) prefill became the bottleneck and left the decode side idling below its peak throughput, so we let each prefill step batch two full prompts instead of one — worth about +8% of total TPS per GPU at high concurrencies.--max-cudagraph-capture-sizeon decode — raised tocc/8 + 128for the two highest concurrency points (640 at cc=4096, 768 at cc=5120), where 8 is the number of DP ranks on the decode endpoint. The vLLM default caps captured graphs at 512, which is enough up to cc=3072. We are not certain this is actually required for the Pareto numbers reported here, but we set it as a precaution.- Prefix caching is disabled: it buys nothing on a random dataset.
--stream-interval 100— reduces frontend overhead at high concurrency. Note that it buffers streamed output in 100-token chunks, so it does affect measured per-token latency; keep that in mind if you are optimizing for ITL/TPOT rather than aggregate throughput.
Finally, a couple of practical things that saved us a lot of time.
--api-server-count 1 is very useful while you are investigating a particular configuration. On a data-parallel endpoint vLLM defaults the API server count to the data-parallel size, and with more than one API server it disables its default stats logging altogether in order not to report incomplete numbers. Forcing the count to 1 brings that logging back: every 10 seconds — the interval is configurable through VLLM_LOG_STATS_INTERVAL — the server prints prompt and generation throughput along with KV cache utilization. Without these metrics we would hardly have identified the bottlenecks of the individual configurations, or understood why a particular option helps on our workload.
It is also worth setting three environment variables: DYN_LOG=error, DYN_SDK_DISABLE_ANSI_LOGGING=1, and VLLM_LOGGING_COLOR=0. The first one drastically cuts down the amount of Dynamo logs, while the other two suppress some (not all!) ANSI escape sequences in the log output. Without them your log files are very likely to be unreadable for a human, mostly because Dynamo produces an enormous amount of logging by default.
What's next
Our measurements so far have concentrated mostly on the left part of the Pareto curve, squeezing out as much total TPS per GPU as possible. Next, we plan to sweep for the PD configurations that maximize Gen TPS per user instead. Reaching that regime will require shifting away from DEP topologies towards TEP (Tensor Parallel + Expert Parallel) or just TP, which as a rule deliver better per-user performance. Increasing the number of GPUs in use is another lever we expect to pay off here.
Acknowledgements
Artem Perevedentsev (NVIDIA), Vadim Gimpelson (NVIDIA), Jiangyun Zhu (Inferact), Nicolò Lucchesi (Mistral), Zhanqiu Hu (Red Hat), Nick Hill (Inferact), Linxuan Li (Alibaba), JingZe Cui (NVIDIA), Cyrus Chang (NVIDIA), Xin Li (NVIDIA)