要点速览
- v0.5.0 引入 DFlash 训练支持:block diffusion 单次前向生成整块 token,不同于 Eagle 3 自回归。
- DFlash 块内用非因果 attention,块结构由 attention mask 实现;训练随机选 anchor 位置控制块数。
- 在线/离线训练统一走 vLLM 原生 hidden states 提取,经 REST API 通信,摆脱对 vLLM 内部 API 的依赖。
- Gemma 4 31B DFlash speculator 在推理与代码任务接受率出色,延迟优于 Eagle 3 与 FP8 verifier。
- DFlash 模型自 vllm>=0.20.0 可直接 serve,speculators_config 描述目标模型与算法。
解决什么问题
Speculators 此前只支持 Eagle 3 这类自回归式 drafter 训练,缺少单次前向生成整块 token 的新一代算法支持。同时旧版隐藏态提取依赖 vLLM 的底层工具,使 vLLM 成为训练管线的直接 python 依赖,内部 API 在版本更新间频繁变化,需要手动与上游同步。v0.5.0 需要同时解决算法扩展与训练管线解耦两个问题。
核心思路
v0.5.0 新增 DFlash 训练支持:drafter 对每个前缀用 block diffusion 一次生成长度 B 的 token 块,块结构与多块并行训练完全通过 attention mask 实现,并从参与 loss 的位置中随机选取 anchor 来限制预测块数量,使训练可扩展到长上下文。训练管线统一迁移到 vLLM v0.18.0 引入的原生 hidden states 提取系统:在线训练边推理边提取,离线训练预生成并缓存到磁盘,两者数据格式一致且可混合使用。训练进程通过标准 REST API 与运行中的 vLLM server 通信,彻底移除 vLLM 作为直接 python 依赖。基于该支持训练并开源了 Gemma 4 31B DFlash speculator 验证效果。
关键要点
- DFlash 与 Eagle 3 的本质差异 — Eagle 3 通过多次前向自回归地生成 draft token,而 DFlash 采用 block diffusion 在单次前向中生成全部 draft token,对更长草稿序列的开销降低尤其明显。drafter 为每个前缀产出长度 B 的 token 块,这一块结构完全由 attention mask 实现。另一个关键差异是 DFlash 使用非因果(non-causal)attention 模式:同一块内的 query 可以关注块内所有其他 token。
- Anchor 采样控制训练成本 — 训练时多个预测块并行训练。直观做法是在序列每个可能位置后都启动预测块,但长序列会让 attention mask 极度膨胀,显存与计算都不可行。DFlash 改为从真正参与训练 loss 的位置中随机选取一组较小的 anchor 位置,预测块只挂载在这些 anchor 上。这样预测块数量与序列长度无关,训练得以扩展到更长上下文,同时保持 attention mask 可控。
- DFlash 训练的关键参数与流程 — 训练流程与 Eagle 3 的在线工作流类似(torchrun 启动,指定 verifier 模型与 vLLM endpoint),差异在 speculator 专属参数:--speculator-type 必须指定 dflash,--block-size 定义每个 diffusion block 生成的 token 数,--max-anchors 限制训练时的最大 anchor 数。示例命令还展示了 --draft-vocab-size 8192、--num-layers 5、--target-layer-ids "2 18 33"、--epochs 5、--lr 1e-4 等 drafter 结构与训练超参,官方提供完整教程。
- Gemma 4 31B DFlash 实测结果 — 基于新的算法支持,训练了 Gemma 4 31B DFlash speculator(RedHatAI/gemma-4-31B-it-speculator.dflash)并在多种任务类型上评估接受率,推理与代码生成任务表现尤佳。其 inter-token latency 优于 Eagle 3 和单独的 FP8 量化 verifier;将 DFlash 与 FP8 量化 verifier 组合还能带来更大收益。
- 统一在线/离线训练与 hidden states 提取 — 两种模式都改用 vLLM(v0.18.0 引入)的原生 hidden states 提取系统。在线训练流程:vLLM server 以基础模型初始化,训练 prompt 发送给 vLLM 推理,hidden states 被提取并临时写入磁盘(或 ram disk),训练进程加载后删除文件,再在提取的 states 上训练 speculator。离线训练则预生成并缓存 hidden states 到磁盘。两种方式耦合紧密,可混合使用:离线生成一部分、在线训练加载已有并补齐缺失;或在线任务首轮生成后不清文件,后续 epoch 直接加载。
- 与 vLLM 的解耦及 serving 集成 — 旧版用 vLLM 底层工具提取 hidden states,导致训练管线与 vLLM 内部 API 紧耦合,版本升级需手动同步。新版通过标准 REST API 与运行中的 vLLM server 通信,继承了 vLLM 的内存管理、batching 策略与硬件加速能力,同时团队可独立升级 vLLM。推理侧,DFlash 模型自 PR #38300(vllm>=0.20.0)起与 vLLM 推测解码基础设施无缝集成:模型 config.json 内含 speculators_config(目标模型、speculative tokens、算法名等),一条 vllm serve 命令即可部署。此外 v0.5.0 还更新了文档站,新增算法介绍、训练教程、新算法接入指南与 API 参考。
关键数据
| 指标 | 数值 | 条件 |
|---|---|---|
| 示例训练配置:block 大小 / anchor 上限 / draft 词表 | block-size 8;max-anchors 3072;draft-vocab-size 8192 | Qwen3-8B 作为 verifier 的 DFlash 在线训练示例命令(2 卡 torchrun) |
| 示例训练配置:drafter 层数与目标层 | 5 层;target-layer-ids 为 2/18/33 | Qwen3-8B verifier 的 DFlash drafter 结构选择 |
| 示例训练配置:epoch 数与学习率 | 5 epochs;lr 1e-4 | Qwen3-8B verifier 的 DFlash 在线训练示例 |
| Gemma 4 DFlash speculator 规模 | 31B(Gemma 4 31B) | 用 v0.5.0 DFlash 支持训练的 speculator,评估接受率与 inter-token latency |
| DFlash 模型在 vLLM 中的支持版本 | vllm>=0.20.0(PR #38300 合入) | vLLM 推测解码基础设施对 DFlash 的原生集成 |
| hidden states 提取系统的引入版本 | vLLM v0.18.0 | Speculators v0.5.0 在线/离线训练统一依赖的原生提取系统 |
“--draft-vocab-size 8192 \ --block-size 8 \ --max-anchors 3072”
“--num-layers 5 \ --target-layer-ids "2 18 33"”
“--epochs 5 --lr 1e-4”
前置知识
推测解码基本原理:drafter 起草 + 目标模型验证 · Eagle 3 自回归起草方式与 Speculators 库的既有训练流程 · block diffusion 与非因果(non-causal)attention 的概念 · vLLM serving 与 REST API 的基本使用 · hidden states 在 drafter 训练中的作用(输入特征)
术语表
DFlash: 基于 block diffusion 的推测解码算法,单次前向生成整块 draft token,采用块内非因果 attention,是 Eagle 3 之后的下一代方案。Block diffusion: 块扩散,把生成过程组织为按 token 块并行预测的建模范式,块结构通过 attention mask 实现。Eagle 3: 自回归式推测解码算法,需多次前向逐个生成 draft token,是 DFlash 的对照范式。Anchor position: anchor 位置,训练时从参与 loss 的位置中随机选取的预测块挂载点,用于把预测块数量固定住、控制 attention mask 规模。Speculators: vLLM 生态的推测解码模型训练库,支持 Eagle 3、DFlash 等算法的在线与离线训练。Speculator: 经训练得到的 draft 模型及其 checkpoint,与目标模型搭配做推测解码。Hidden states extraction: vLLM v0.18.0 引入的原生隐藏态提取系统,让训练管线通过标准接口获取目标模型 hidden states,替代旧的底层工具。Online / Offline training: 在线训练边推理边提取 hidden states(临时落盘后加载即删);离线训练预先批量生成并缓存到磁盘再训练,两者可混合。Acceptance rate: 接受率,目标模型验证时接受 draft token 的比例,衡量 speculator 质量的核心指标。Inter-token latency: 相邻 token 之间的生成延迟,推测解码与量化的核心优化目标之一。
Speculators v0.5.0: DFlash Support and Online Training
The v0.5.0 release brings significant architectural improvements to speculative decoding model training, introducing DFlash algorithm support, fully unified online training capabilities, and a complete migration to vLLM's native hidden states extraction system. This release represents a major step forward in both training flexibility and production readiness for speculative decoding workflows.
Key features include:
- DFlash algorithm support - Single-pass draft token generation with block diffusion
- Gemma 4 DFlash results
- vLLM-native online and offline training with unified hidden states extraction
- Updated documentation and examples outlining key workflows
DFlash Algorithm Support
v0.5.0 introduces training support for the DFlash speculative decoding algorithm, a fundamentally different approach to draft token generation compared to the autoregressive Eagle 3 models. While Eagle 3 generates draft tokens autoregressively through multiple forward passes, DFlash employs block diffusion to generate all draft tokens in a single forward pass.
The single-pass nature of DFlash can dramatically reduce the overhead of speculative decoding, particularly for longer draft sequences. The drafter produces a block of tokens of length B for each prefix. This block structure is entirely accomplished using the attention mask. Another key difference from Eagle3 is that DFlash uses a non-causal attention pattern where queries within a block can attend to all other tokens within the same block.
During training, multiple predicted blocks are trained on in parallel. A straightforward approach would be to start a prediction block after every possible point in the sequence. For a long sequence, however, this causes the attention mask to grow extremely large, making training impractical in both memory usage and compute cost. To avoid this, we do not start blocks everywhere. Instead, we randomly choose a smaller set of “anchor” positions from locations that actually contribute to the training loss. Predicted blocks are only attached to these anchors. This keeps the number of predicted blocks fixed regardless of sequence length, allowing training to scale to much longer contexts while keeping the attention mask manageable.
Training a DFlash Speculator
Training a DFlash model follows a similar online workflow to Eagle 3. A complete tutorial is provided here.
The key difference from Eagle 3 is the speculator-specific parameters in the training command, as shown below:
torchrun --standalone --nproc_per_node 2 scripts/train.py \
--verifier-name-or-path "Qwen/Qwen3-8B" \
--vllm-endpoint "http://localhost:8000/v1" \
--speculator-type dflash \
--draft-vocab-size 8192 \
--block-size 8 \
--max-anchors 3072 \
--num-layers 5 \
--target-layer-ids "2 18 33" \
--epochs 5 --lr 1e-4
DFlash-specific parameters include:
--block-size # Number of tokens generated per diffusion block
--max-anchors # Maximum anchor points for speculation during training
--speculator-type # Must specify dflash
Gemma 4 DFlash Speculator
Using the DFlash algorithmic support, a Gemma 4 31B DFlash speculator was trained and acceptance rates were evaluated across diverse task types. The results demonstrate strong performance particularly on reasoning and code generation tasks:
Figure 1: Gemma 4 DFlash acceptance rates across diverse task types.
Gemma 4 DFlash achieves better inter-token latency than both Eagle 3 and a standalone FP8 quantized verifier. Combining DFlash with an FP8 quantized verifier yields even greater gains, as shown below:
Figure 2: Gemma 4 DFlash inter-token latency comparison.
Serving DFlash models in vLLM
DFlash models integrate seamlessly with vLLM's speculative decoding infrastructure, as of PR #38300, which is included in vllm>=0.20.0.
Similar to the Eagle 3 models, DFlash models contain a speculators_config in their config.json which contain details on the target model, speculative tokens, the name of the speculative algorithm, etc. With this config, models can be served using a basic vllm serve command, as shown below.
vllm serve -tp 2 RedHatAI/gemma-4-31B-it-speculator.dflash
Unified Online and Offline Training Support
v0.5.0 adds native support for both online and offline training modes through vLLM's hidden states extraction system (introduced in vLLM v0.18.0). Previous versions of Speculators extracted hidden states using lower-level utilities from vLLM, requiring vLLM to be a direct python dependency. This approach tightly coupled the training pipeline to vLLM's internal APIs, which often changes between vLLM version updates and required manual synchronization with upstream changes. This integration removes the previous custom data generation pipeline and eliminates vLLM as a direct python dependency.
Both training modes now use the same vLLM-based extraction path:
- Online training: Extract hidden states on-the-fly during training
- Offline training: Pre-generate and cache hidden states to disk, then train
By leveraging vLLM's native hidden states extraction, Speculators inherits all of vLLM's inference optimizations including efficient memory management, batching strategies, and hardware acceleration support. Training now communicates with a running vLLM server via its standard REST API, decoupling the training infrastructure from vLLM's internal implementation details. This architectural shift provides better version stability and makes it easier for teams to update vLLM independently of the Speculators training framework.
What happens during online training:
- vLLM server initializes with the base model (and some special configuration)
- Training prompts are sent to vLLM for inference
- Hidden states are extracted and temporarily written to disk (or ram disk)
- The training process loads the extracted hidden states and deletes the file
- Speculator model trains on extracted states
This tutorial provides more information on the online training workflow.
Offline data generation has also been updated to use the same hidden states extraction system and data format as online. New scripts have been developed to saturate the running vLLM server with requests and write them to disk. The two approaches are so tightly coupled that you can even run a combination of them. For example, you can partially generate hidden states offline and then run training and load the existing hidden states, while generating any that are missing. You can also run an online training job that does not clear the files after generating them, allowing you to generate once on the first epoch and then load the files on subsequent ones.
This tutorial goes into more detail on the offline training workflow.
Added Comprehensive Documentation
Another feature worth highlighting is the updated documentation site. We’ve added concise introductions to the speculative decoding algorithms supported by Speculators, along with detailed tutorial walkthroughs for training speculator models. For developers, we also introduced a guide covering how to add new speculative decoding algorithms to the Speculators library, as well as a comprehensive API reference.