要点速览
- MoE 中 Attention 有状态、与调度和 KV cache 紧耦合;FFN 由专家计算与 all-to-all 通信主导,两者扩展需求与职责迥异。
- Attention 侧保留 vLLM 的调度、KV cache 与采样;FFN 侧是无请求流量、无调度器的轻量 connector 驱动 daemon。
- 插件以外部插件形式接入,不改 vLLM 源码树;当前支持 NVIDIA GPU 与 Ascend NPU 两类后端。
- 同步 connector 面向 decode 吞吐;异步 CAM 面向 prefill,以 MoE ubatching 重叠独立阶段。
- 64A16F 较 EP64 每 die 吞吐 +11.3%(16K)/+9.0%(32K),48A16F 为负;CAM 异步使 TTFT 降约 47%。
解决什么问题
MoE 推理在每个 transformer 层内混合了两类特性迥异的工作:Attention 有状态、与请求调度和 KV cache 紧耦合;FFN/专家路径由路由专家计算与 all-to-all 通信主导。当两者共享同一 worker topology 时,serving 系统只能为需求不同的负载做一套统一的扩展与执行选择。此外 CUDA 与 Ascend 暴露不同的集合通信库、graph 运行时与 MoE 算子,需要一个稳定且后端中立的契约。本文介绍如何把 Attention 与 FFN 拆成可独立扩展、独立通信的服务,同时保留 vLLM 的请求生命周期与 OpenAI 兼容接口。
核心思路
保留 vLLM 的请求侧 Attention 路径,把 FFN 执行移到一个窄 connector 接口之后:Attention 服务保留调度器、KV cache、batching、模型生命周期与采样路径,插件自有的 model runner 把 AFD metadata 装入 forward context 并向 FFN 侧发布 data-parallel、ubatch、layer、graph 状态;FFN 服务没有请求流量、调度器或 KV cache,后台循环接收 metadata 与 activations,调用 compute_ffn_output() 后把结果送回 Attention——请求始终发给 Attention 的 API server。连接器层在每个 split layer 传输 hidden states 与执行 metadata 并回传 FFN 输出,接口 backend 中立,各后端自行实现通信与运行时优化。插件经 vllm.general_plugins entry point 与 --additional-config 接入,不需要修改 vLLM 源码树;同步 connector(GPU P2P NCCL / NPU CAMP2P)服务 decode 吞吐,异步 CAM connector 服务 prefill 并用 MoE ubatching 重叠独立阶段。
关键要点
- 为什么拆分 Attention 与 FFN — 四个系统设计挑战促成 AFD:扩展需求不同——Attention 容量随请求状态、序列长度与 KV cache 压力变化,专家容量随 token 路由与专家负载变化,应允许两条路径用不同 rank topology;运行时职责不同——Attention 需要调度、KV 协调与采样,FFN 只需要 activations、routing metadata 与回传结果的通道,拆开后 FFN 侧可跑成轻量 connector 驱动的 daemon;通信是后端相关的——CUDA 与 Ascend 的集合库、graph 运行时、MoE 算子各不相同,需要统一契约;通信与计算可重叠——异步 dispatch 与 MoE ubatching 能重叠独立阶段而非串行。
- 三段式运行时架构 — Attention 服务:worker 保留 vLLM 的调度器、KV cache、batching、model lifecycle 与采样路径,插件自有的 model runner 把 AFD metadata 装入 forward context,并向 FFN 侧发布 data-parallel、ubatch、layer 与 graph 状态。FFN 服务:没有请求流量、调度器或 KV cache,后台循环接收 metadata 与 activations,调用插件模型 wrapper 上的 compute_ffn_output() 并把结果送回 Attention;请求始终发给 Attention 的 API server。Connector 层:在每个 split layer 传输 Attention hidden states 及 FFN 所需执行 metadata,再回传计算好的 FFN 输出,接口 backend 中立。
- 刻意收窄的集成面 — 插件通过 vLLM 的 vllm.general_plugins entry point 与标准 --additional-config 通道接入,不需要改动 vLLM 源码树。vLLM 继续拥有其现有抽象最合适的 serving 控制面,插件提供 AFD workers、model runners、connectors、metadata、模型切分点以及少量版本范围的兼容性 patch。共享行为放在配置、拓扑、metadata 与 connector 契约里而非跨设备继承:GPU worker 扩展 vLLM v1 类,NPU worker 直接扩展 vLLM-Ascend 类。
- Connector 与后端矩阵 — P2pNcclAFDConnector 面向 GPU,同步 P2P,推荐用于 decode,支持 FULL_DECODE_ONLY CUDA graph;CAMP2pAFDConnector 面向 NPU,同步 CAMP2P/HCCL,同样用于 decode,支持 FULL_DECODE_ONLY ACL graph;CAMAsyncAFDConnector 面向 NPU,异步 CAM,用于 prefill,暂不支持 graph。高层交换(Attention 输出发给 FFN、FFN 输出回传)在各 connector 间一致;后端包彼此隔离,避免 CUDA graph 行为、ACL graph 行为、NCCL 通信与昇腾自定义算子互相泄漏。
- MoE 模型集成与执行路径 — 插件为 DeepSeek V2/V3 家族架构(含 DeepSeek V3.2)与 GLM MoE DSA 注册 wrapper:暴露分开的 Attention 与 FFN 计算,同时复用上游层实现。同步路径面向吞吐导向的 decode 部署,两种角色可独立扩展;异步 CAM 路径用 CAM dispatch/combine 算子解耦 prefill Attention rank 与 expert worker,配合 AFD 管理的 MoE ubatching 重叠独立的 Attention 与 FFN 阶段、减少流水线停顿,当前面向 PD 分离部署的 prefill 阶段。图执行方面,同步 GPU 与 NPU connector 支持 decode-only graph capture;Dual Batch Overlap(DBO)恰好支持两个 ubatch。
- 同步 decode 吞吐基准设置 — 在 Ascend 910C 上对 DeepSeek-V3.2 W8A8 比较常规 EP64 部署与基于 CAMP2pAFDConnector 的 AFD 部署,度量的是饱和 decode 吞吐而非在线服务延迟。三种拓扑:EP64(DP64/EP64/TP1,64 dies)、48A16F(48 Attention + 16 FFN,64 dies)、64A16F(64 Attention + 16 FFN,80 dies),吞吐按总部署 dies 归一化为 tokens/s/die。重要限制:这是受控性能结果而非精度或生产结论;机器有限,物理 48A16F/64A16F 部署模拟逻辑 192A64F/256A64F 规模;实验用确定性强制均衡路由替换自然路由专家 ID(会改变模型输出);AFDDecodeBenchConnector 供应 decode-only KV 状态;AFD 侧开启 DBO。
- 吞吐结果与解读 — 16K 固定输入:EP64 232.6、48A16F 220.3、64A16F 258.9 tokens/s/die,相对 EP64 为 -5.3% / +11.3%;32K 固定输入:EP64 168.2、48A16F 151.4、64A16F 183.3 tokens/s/die,相对为 -10.0% / +9.0%。两种输入长度下 48A16F 都低于基线,64A16F 都最高——说明 Attention 与 FFN 的配比很关键,拆分本身不保证吞吐收益。作者推断:在测试过的比例下 FFN rank 仍有算力余量而非 compute-bound,提高 Attention rank 占比可能揭示更多增益;输出为 512 到 1,536 token 的均匀分布。
- 异步 prefill 实验与当前边界 — 两个 Ascend 910C 节点、缩减到 10 层的 DeepSeek V3.2 W8A8 上的早期 CAM 异步实验:对比 DP4PCP8 TP1 基线与 Attention DP3PCP8 TP1 + FFN EP8 的 AFD 布局,同样使用强制专家均衡。在测得的各请求速率下 AFD 配置都降低 median/P50 TTFT:12 req/s 时从 15.1 秒降到 8.0 秒(约 -47%),10 与 12 req/s 两档差距均约 7.2 秒;这是对 CAM 异步执行路径的针对性验证,不是对完整 DeepSeek V3.2 或所有 AFD 拓扑的一般性能结论。当前边界:实验性项目、精确钉住 vLLM 版本、仅 model runner v1、两种角色都驻留完整权重、仅 decode-only graph 模式、DBO 恰好两个 ubatch、硬件门控的端到端测试。路线图包括更广 vLLM 兼容与上游对齐、更灵活的执行(graph 模式/ubatch 数/异步阶段)、生产级验证、更多模型与 connector、多模态(vLLM-Omni)与异构硬件低延迟方向。
关键数据
| 指标 | 数值 | 条件 |
|---|---|---|
| 16K 输入下每 die 解码吞吐 | EP64 232.6 / 48A16F 220.3 / 64A16F 258.9 tokens/s/die | DeepSeek-V3.2 W8A8、Ascend 910C、CAMP2pAFDConnector 同步 decode,固定长度输入、输出 512–1536 token 均匀分布,吞吐按总部署 dies 归一化 |
| 16K 输入下相对 EP64 的变化 | 48A16F -5.3%;64A16F +11.3% | 同上基准,AFD 部署相对 EP64 基线的归一化吞吐变化 |
| 32K 输入下每 die 解码吞吐 | EP64 168.2 / 48A16F 151.4 / 64A16F 183.3 tokens/s/die;相对变化 -10.0% / +9.0% | 同上基准的 32K 固定输入版本 |
| median TTFT 降幅 | 15.1 秒 → 8.0 秒(约 -47%) | 12 requests/s,两个 Ascend 910C 节点、缩减到 10 层的 DeepSeek V3.2 W8A8,CAMAsyncAFDConnector 异步 prefill 对比 DP4PCP8 TP1 基线 |
| 两档请求率下的 TTFT 差距 | 约 7.2 秒 | 10 与 12 requests/s 两档测得的 AFD 与基线 median TTFT 差距 |
| 基准部署拓扑与规模说明 | EP64=DP64/EP64/TP1(64 dies);48A16F=48A+16F(64 dies);64A16F=64A+16F(80 dies);物理部署模拟逻辑 192A64F/256A64F | Ascend 910C 同步 decode 基准;因机器有限用强制均衡路由,属受控性能结果 |
| 目标 vLLM 版本与 Python 要求 | vLLM 0.19.1;Python 3.10–3.13 | 当前 AFD Plugin 实现的版本要求 |
“EP64 achieves **232.6 tokens/s/die**, 48A16F achieves **220.3 tokens/s/die**, and 64A16F achieves **258.9 tokens/s/die**.”
“Relative to EP64, the AFD results are **-5.3%** for 48A16F and **+11.3%** for 64A16F.”
“EP64 achieves **168.2 tokens/s/die**, 48A16F achieves **151.4 tokens/s/die**, and 64A16F achieves **183.3 tokens/s/die**. Relative to EP64, the AFD results are **-10.0%** for 48A16F and **+9.0%** for 64A16F.”
前置知识
MoE 模型基础:expert routing、all-to-all 通信 · vLLM V1 架构:model runner、forward context、scheduler 与 KV cache 的分工 · prefill/decode 分离(PD disaggregation)概念,异步路径面向分离部署的 prefill 阶段 · 并行策略:DP/TP/EP 及其组合(如 DP64/EP64/TP1) · CUDA graph 与 Ascend ACL graph 的作用与限制 · NCCL/HCCL 等集合通信库的基本概念
术语表
AFD (Attention-FFN Disaggregation): 把 transformer 的 Attention 与 FFN(专家路径)拆成独立部署、独立扩展的服务,经 connector 层通信的架构。MoE (Mixture-of-Experts): 每层只激活路由到的部分专家网络的模型结构,FFN 侧计算由 token 路由与专家负载决定。Connector: AFD 中在每个 split layer 传输 Attention hidden states 与执行 metadata、并回传 FFN 输出的 backend 中立接口,各后端自行实现通信与运行时优化。CAM: Ascend 上的异步通信机制,提供 dispatch 与 combine 算子,CAMAsyncAFDConnector 用它解耦 prefill Attention rank 与 expert worker。CAMP2P: Ascend NPU 上经 HCCL 的同步点对点通信方式,CAMP2pAFDConnector 用它同步交换 Attention activations 与 FFN 输出。DBO (Dual Batch Overlap): 双批重叠:把 batch 拆成两个 ubatch 交错计算与通信;本插件恰好支持两个 ubatch。FULL_DECODE_ONLY graph: 仅在 decode 阶段捕获执行的图模式,分别对应 CUDA graph 与 ACL graph。EP (Expert Parallelism): 把 MoE 专家切分到不同 rank 的并行方式,常规 MoE 部署(如 EP64)的基准拓扑。Ascend NPU: 华为昇腾神经网络加速器,本插件与 NVIDIA GPU 并列支持的后端,经 vLLM-Ascend 类扩展。
Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving
We are excited to introduce vLLM AFD Plugin, an experimental external plugin that brings Attention-FFN Disaggregation (AFD) to vLLM.
vLLM AFD Plugin brings AFD into Mixture-of-Experts (MoE) models by separating Attention and FFN into independently deployed services. The plugin preserves vLLM's existing request lifecycle and OpenAI-compatible serving interface, while allowing the Attention and FFN paths to scale independently.
The project currently supports NVIDIA GPUs and Ascend NPUs, synchronous and asynchronous connectors, DeepSeek V2/V3-family model wrappers, and eager, graph, and dual-batch execution paths within clearly validated limits.
[!NOTE] This project is still experimental and needs more large-scale testing across different hardware backends.
Why Attention-FFN Disaggregation?
Mixture-of-Experts (MoE) inference combines two very different kinds of work inside every transformer layer. Attention is stateful and closely coupled to request scheduling and the KV cache, while the FFN or expert path is dominated by routed expert computation and all-to-all communication. When both paths share the same worker topology, the serving system must make one set of scaling and execution choices for workloads with very different requirements.
Making this separation practical requires addressing several system design challenges:
- Attention and FFN have different scaling requirements. Attention capacity follows request state, sequence length, and KV-cache pressure. Expert capacity follows token routing and expert load. The serving system should support independent scaling by allowing both paths to use different rank topologies, instead of requiring one shared layout.
- Attention and FFN have different runtime responsibilities. Attention needs scheduling, KV-cache coordination, and sampling. FFN execution only needs activations, routing metadata, and a way to return expert outputs. Splitting the services lets the FFN side run as a lightweight connector-driven daemon.
- Communication is backend-specific. CUDA and Ascend expose different collective libraries, graph runtimes, and optimized MoE operators. A common connector contract keeps the model-facing flow stable while allowing each backend to own its data path.
- Communication and computation benefit from overlap. Asynchronous dispatch and MoE ubatching can overlap independent stages instead of serializing all expert work behind the Attention path.
Together, these challenges define the core design goal of AFD: keep vLLM's request-facing Attention path intact, while moving FFN execution behind a narrow connector interface that can scale, communicate, and execute independently.
Inside the Architecture
The plugin integrates through vLLM's vllm.general_plugins entry point and the standard --additional-config channel. It does not require edits to the vLLM source tree.
The runtime has three main parts:
- Attention service. The Attention worker retains vLLM's scheduler, KV cache, batching, model lifecycle, and sampling path. A plugin-owned model runner installs AFD metadata into the forward context and publishes data-parallel, ubatch, layer, and graph state to the FFN side.
- FFN service. The FFN worker has no request traffic, scheduler, or KV cache. A background loop receives metadata and activations, invokes
compute_ffn_output()on the plugin-owned model wrapper, and sends the result back to Attention. Requests are always sent to the Attention API server. - Connector layer. At each split layer, the connector transfers Attention hidden states together with the execution metadata required by the FFN service, then returns the computed FFN outputs. A backend-neutral connector interface defines this exchange while allowing each backend to implement its own communication and runtime optimizations.
This integration surface is designed to be intentionally small. vLLM continues to own the serving control plane where its existing abstractions fit, while the plugin provides the implementations of AFD workers, model runners, connectors, metadata, model split points, and a small set of version-scoped compatibility patches.
Connector and backend support
| Connector | Backend | Execution | Recommended stage | Graph support |
|---|---|---|---|---|
P2pNcclAFDConnector |
GPU | Synchronous P2P | Decode | FULL_DECODE_ONLY CUDA graph |
CAMP2pAFDConnector |
NPU | Synchronous CAMP2P/HCCL | Decode | FULL_DECODE_ONLY ACL graph |
CAMAsyncAFDConnector |
NPU | Asynchronous CAM | Prefill | Not currently supported |
The same high-level exchange - Attention output to FFN, FFN output back to Attention - is shared across connectors. Backend packages remain separate so CUDA graph behavior, ACL graph behavior, NCCL communication, and Ascend custom operators do not leak into one another.
Supported features
- Native vLLM serving surface. Existing vLLM users still launch with
vllm serve, send requests to an OpenAI-compatible endpoint, and configure the runtime through--additional-config. - GPU and NPU implementations. GPU workers extend vLLM v1 classes, while NPU workers extend vLLM-Ascend classes directly. Shared behavior lives in configuration, topology, metadata, and connector contracts rather than cross-device inheritance.
- Synchronous AFD for decode throughput.
P2pNcclAFDConnectorandCAMP2pAFDConnectorsynchronously exchange Attention activations and FFN outputs, allowing the two roles to scale independently in throughput-oriented decode deployments. Their current graph paths useFULL_DECODE_ONLYsemantics on CUDA and ACL, respectively. - Asynchronous AFD for prefill.
CAMAsyncAFDConnectoruses CAM asynchronous dispatch and combine operators to decouple prefill Attention ranks from expert workers. Together with AFD-managed MoE ubatching, it overlaps independent Attention and FFN stages to reduce pipeline stalls. This path currently targets the prefill stage in a prefill/decode-disaggregated deployment and does not yet support graph execution. - MoE model integration. The plugin registers wrappers for DeepSeek V2/V3-family architectures, including DeepSeek V3.2, and GLM MoE DSA. The wrapper exposes separate Attention and FFN computations while reusing upstream layer implementations.
- Graph and ubatching paths. The synchronous GPU and NPU connectors support decode-only graph capture. Dual Batch Overlap is supported with exactly two ubatches, and CAM async provides AFD-managed MoE ubatching for its prefill path.
A Performance Snapshot
Synchronous AFD Decode Throughput with CAMP2pAFDConnector
The synchronous decode recipe in vllm-project/afd-plugin#67 compares a conventional EP64 deployment with CAMP2pAFDConnector-based AFD deployments for DeepSeek-V3.2 W8A8 on Ascend 910C. The benchmark measures saturated decode throughput rather than online-serving latency.
| Deployment | Physical topology | Total dies |
|---|---|---|
| EP64 | DP64, EP64, TP1 | 64 |
| 48A16F | 48 Attention ranks, 16 FFN ranks | 64 |
| 64A16F | 64 Attention ranks, 16 FFN ranks | 80 |
[!NOTE] These are controlled performance results, not accuracy or production-serving results. Due to limited machine availability, the physical 48A16F and 64A16F deployments simulate logical 192A64F and 256A64F scales. The experiment replaces natural routed expert IDs with a deterministic forced-balancing cycle, which changes model outputs.
AFDDecodeBenchConnectorsupplies the decode-only KV state, and DBO is enabled for AFD.
Throughput is normalized by the total number of deployed dies:
tokens/s/die = aggregate output token throughput / total deployed dies
Both workloads use fixed-length inputs and uniformly distributed outputs from 512 to 1,536 tokens.
16K fixed input
EP64 achieves 232.6 tokens/s/die, 48A16F achieves 220.3 tokens/s/die, and 64A16F achieves 258.9 tokens/s/die. Relative to EP64, the AFD results are -5.3% for 48A16F and +11.3% for 64A16F.
32K fixed input
EP64 achieves 168.2 tokens/s/die, 48A16F achieves 151.4 tokens/s/die, and 64A16F achieves 183.3 tokens/s/die. Relative to EP64, the AFD results are -10.0% for 48A16F and +9.0% for 64A16F.
Across both input lengths, 48A16F is below the EP64 baseline, while 64A16F delivers the highest normalized throughput: +11.3% at 16K and +9.0% at 32K. This result shows that the Attention-to-FFN allocation matters; disaggregation alone does not guarantee a throughput gain.
Due to limited machine availability, we did not evaluate deployments with higher Attention-to-FFN ratios. The observed trend suggests that, at the ratios tested, the FFN ranks still have compute headroom rather than being compute-bound. Increasing the proportion of Attention ranks may therefore reveal further throughput gains.
Asynchronous AFD Prefill Performance with CAMAsyncAFDConnector
The repository includes an early CAM async experiment on two Ascend 910C nodes using a DeepSeek V3.2 W8A8 model reduced to 10 layers. The comparison uses forced expert balancing and contrasts a DP4PCP8 TP1 baseline with an AFD layout consisting of Attention DP3PCP8 TP1 plus FFN EP8.
Across the measured request rates, the AFD configuration lowers median/P50 time to first token. At 12 requests per second, median TTFT decreases from 15.1 seconds to 8.0 seconds, a reduction of approximately 47%. At both 10 and 12 requests per second, the measured gap is about 7.2 seconds.
Note: These numbers are a focused validation of the CAM async execution path, not a general performance claim for full DeepSeek V3.2 or every AFD topology. The performance gains may also vary across workloads.
Getting Started
The current implementation requires Python 3.10–3.13 and targets vLLM 0.19.1.
Install
Check out the installation steps in our README for details.
Deployment Recipes
Deployment commands depend on the backend, connector, model, and rank topology. Instead of duplicating configurations here, use the maintained AFD Plugin recipes:
- GPU synchronous AFD: the DeepSeek V2 Lite P2P NCCL recipes cover decode-oriented colocated and prefill/decode-disaggregated deployments, eager and CUDA graph execution, and multiple DP/TP layouts.
- NPU asynchronous prefill AFD: the DeepSeek V3.2 CAM async recipe documents the required environment, topology, AFD configuration, benchmark setup, and current limitations.
Refer to the repository README and recipe directory for the latest supported connector matrix, configuration fields, and complete launch commands.
Current Scope and Roadmap
The project intentionally exposes its current boundaries: exact vLLM version pinning, model runner v1 only, full weights on both roles, decode-only graph modes, exactly two ubatches for DBO, and hardware-gated end-to-end testing.
The next phase of development will focus on:
- Broader vLLM compatibility and upstream alignment: track newer vLLM releases, evaluate model runner v2, keep compatibility patches minimal, and contribute generally useful abstractions upstream as they mature.
- More flexible execution: extend graph modes, ubatch counts, asynchronous stages, and validated rank topologies.
- Production-scale validation: publish repeatable accuracy, latency, throughput, stability, and multi-node results on full models and realistic workloads.
- Expanded model and connector coverage: add MoE architectures and backend transports through the existing model-wrapper and connector interfaces, together with corresponding deployment recipes for each newly supported model and connector.
- Multimodal and vLLM-Omni integration: explore how AFD can integrate with vLLM-Omni and heterogeneous multimodal pipelines, including its application within autoregressive (AR), Diffusion Transformer (DiT), and other stages that can benefit from independently scaled Attention and FFN execution.
- Heterogeneous hardware and low-latency serving: explore deploying Attention and FFN roles across different accelerator types and interconnects, together with connector, scheduling, placement, and computation-communication overlap optimizations that reduce time to first token and inter-token latency.
Join the Community
vLLM AFD Plugin is at an early stage, and feedback from model, serving, and hardware communities will shape its direction.
- Code and documentation: github.com/vllm-project/afd-plugin
- Runtime design docs: GPU Attention/FFN and Ascend Attention/FFN designs
- Issues and feature requests: GitHub Issues
Let's build a more composable and hardware-aware future for MoE serving together.