要点速览
- EAGLE 3.1 主打 robustness：修复换 chat template、长上下文、OOD system prompt 时的性能退化。
- 根因 attention drift：drafter 注意力从 sink tokens 漂向自生成 token，深层推测不稳定。
- 方案：每个 target hidden state 后加 FC normalization，post-norm hidden states 反馈给下一步。
- 长上下文 acceptance length 最高达 2×；vLLM 兼容 EAGLE 3 checkpoints，随 v0.22.0 发布。
- Kimi K2.6 实测（GB200、TP=4、SPEED-Bench）：并发 1 单用户吞吐 2.03×，C=4/16 时 1.71×/1.66×。
解决什么问题
推测解码在受控环境中表现良好，但换用不同 chat template、面对长上下文输入或 out-of-distribution system prompt 时，性能常常退化，实验室里的加速难以直接迁移到生产。这篇文章定位该脆弱性的根源（EAGLE 团队称之为 attention drift），给出架构级修复方案 EAGLE 3.1，并说明其在 vLLM 与 TorchSpec 中的工程落地。
核心思路
EAGLE 团队识别出两个底层问题：高层 hidden states 在 fused 输入表示中占比日益失衡，以及未归一化的 residual path 使 hidden-state 幅值随推测步数增长。EAGLE 3.1 因此在每个 target hidden state 之后、FC 层之前加入 normalization，并把 post-norm hidden states 送入下一个解码步，让方法的行为更像跨步递归调用 drafter 而非给 target 模型追加层。vLLM 侧以 config-driven 扩展合入该实现并保持对既有 EAGLE 3 checkpoints 的向后兼容；TorchSpec 提供高效训练支持，三方协作训练并开源了 Kimi K2.6 的 EAGLE 3.1 draft model，且给出了 GB200 上的基准数据。
关键要点
- 问题：真实部署下的性能退化 — 推测解码在受控设置下表现很好，但在不同 chat templates、长上下文输入或 out-of-distribution system prompts 下性能往往下降。这种对部署环境的敏感性使推测解码的收益难以在多样化生产场景中稳定复现。
- 根因：attention drift 与两个底层机制 — EAGLE 团队把脆弱性溯源到 attention drift：随着 speculation depth 增加，drafter 的注意力逐渐从 sink tokens 偏移到自己生成的 token。背后有两个问题——其一是 fused input representation 越来越失衡（高层 hidden states 主导 drafter 输入），其二是未归一化的 residual path 使 hidden-state 幅值跨推测步增长——两者叠加让 drafter 在更深的推测深度上越来越不稳定。
- 方案：FC normalization + post-norm 反馈 — EAGLE 3.1 引入两项架构改进：在每个 target hidden state 之后、FC 层之前做 FC normalization；并把 post-norm hidden states 送入下一个解码步。直观上，post-norm 设计使方法更像跨解码步递归调用 drafter，而不是简单给 target 模型追加额外层，从而在深层推测中保持稳定。
- 四方面鲁棒性收益 — 与 EAGLE 3 相比，EAGLE 3.1 展现出：更好的训练到推理外推（training-time to inference-time extrapolation）、更强的长上下文鲁棒性、对 chat template 与 system prompt 变化更高的韧性，以及在多样 serving 环境下更稳定的 acceptance length。长上下文负载下 acceptance length 最高可达 EAGLE 3 的 2 倍。
- TorchSpec 训练支持与 Kimi K2.6 开源模型 — TorchSpec 现已为 EAGLE 3.1 及未来推测解码算法提供高效训练支持，通过降低训练开销、简化实验流程加速研究迭代。基于 TorchSpec 与 vLLM，三方还训练并开源了 Kimi K2.6 的 EAGLE 3.1 draft 模型（lightseekorg/kimi-k2.6-eagle3.1-mla），作为在真实 serving 模型上部署 EAGLE 3.1 的范例。
- vLLM 集成：config-driven 且向后兼容 — EAGLE 3.1 在 vLLM 中以 config-driven 扩展的方式落在现有 EAGLE 3 实现之上，内容包括 FC normalization 支持、post-norm hidden-state 反馈、以及移除对 target hidden states 的硬编码假设。对既有 EAGLE 3 checkpoints 完全向后兼容，EAGLE 3.1 draft 模型可直接通过同一推测解码代码路径接入，例如对 Kimi-K2.6-NVFP4 用 --speculative-config 指定 method 为 eagle3、num_speculative_tokens 为 3。该支持已合入 vLLM main 分支，将随 nightly 与 v0.22.0 发布。
- 基准：单用户吞吐最高 2.03× — 在 Kimi-K2.6-NVFP4 上以 vLLM（TP=4、GB200、非 disagg）运行 SPEED-Bench coding 数据集：EAGLE 3.1 相对无推测基线在并发 1 时单用户输出吞吐提升 2.03×；并发 4 时 1.71×、并发 16 时 1.66×，随并发扩展仍保持有意义的加速。
- 跨团队开源协作模式 — EAGLE 团队持续演进推测解码算法，vLLM 把这些创新大规模带入生产推理系统，TorchSpec 提供高效训练与快速实验基础设施，NVIDIA 的 GPU 支持则保障了开发、验证与基准测试。这是算法研究、系统优化与训练基础设施三方开源协作的范例，目标共同抬高推测解码的整体基线。
关键数据
| 指标 | 数值 | 条件 |
|---|---|---|
| 长上下文 acceptance length 提升 | 最高 2×（相对 EAGLE 3） | 长上下文（long-context）负载下，EAGLE 3.1 与 EAGLE 3 对比 |
| 单用户输出吞吐加速（并发 1） | 2.03× | Kimi-K2.6-NVFP4 + vLLM，TP=4，GB200，非 disagg，SPEED-Bench coding 数据集，EAGLE 3.1-MLA 对比无推测基线 |
| 并发扩展下的加速保持 | 并发 4 时 1.71×，并发 16 时 1.66× | 同上：Kimi-K2.6-NVFP4 + vLLM（TP=4，GB200），SPEED-Bench coding |
| vLLM 发布版本 | v0.22.0（及 nightly） | EAGLE 3.1 支持已合入 vLLM main 分支，随 nightly 与后续版本提供 |
“In long-context workloads, **EAGLE 3.1 achieves up to 2× longer acceptance length compared with EAGLE 3**.”
“EAGLE 3.1 delivers **2.03× higher per-user output throughput at concurrency 1**, and the speedup stays meaningful as concurrency scales (1.71× at C=4, 1.66× at C=16).”
“the speedup stays meaningful as concurrency scales (1.71× at C=4, 1.66× at C=16).”
前置知识
推测解码基本原理与 acceptance length 概念（建议先读路径前两篇） · EAGLE 系列（尤其 EAGLE 3）的 drafter 结构：以 target hidden states 为输入起草 · Transformer 的 residual stream 与 normalization（pre-norm/post-norm 区别） · vLLM --speculative-config 的用法 · 了解即可：NVFP4 量化格式、MLA（Multi-head Latent Attention）注意力机制
术语表
Attention Drift: 注意力漂移：随推测深度增加，drafter 的注意力逐渐从 sink tokens 转向自己生成的 token，导致深层推测不稳定的现象。Sink Tokens: 注意力汇聚 token（通常位于序列开头），模型注意力倾向于长期聚集在这些 token 上。FC Normalization: EAGLE 3.1 在每个 target hidden state 之后、FC 层之前加入的归一化，用于抑制高层 hidden states 主导 fused 输入造成的失衡。Post-norm: 将归一化后的 hidden states 反馈给下一个解码步的设计，使方法更像递归调用 drafter 而非给 target 模型追加层。Drafter: draft 侧的推测生成器：以 target hidden states 与 token ids 为输入，自回归产出候选 token 的轻量模型。Acceptance Length: 接受长度：每个推测步平均被 verifier 接受的 token 数量，直接决定推测解码的加速幅度。TorchSpec: 推测解码训练库，为 EAGLE 3.1 及未来算法提供高效训练支持，降低训练开销并简化实验流程。NVFP4: NVIDIA 的 4 比特浮点量化格式；文中的 Kimi-K2.6-NVFP4 即以该格式量化的 verifier 模型。MLA: Multi-head Latent Attention，Kimi K2.6 采用的潜空间注意力机制，对应开源 draft 模型名中的 -mla 后缀。SPEED-Bench: 文中用于评测的编码（coding）类基准数据集。
EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec
The EAGLE series — including EAGLE 1, EAGLE 2, and EAGLE 3 — has become one of the most widely adopted and practically deployed families of speculative decoding algorithms across both research and production systems.
Today, the EAGLE team, vLLM team, and TorchSpec team are excited to jointly introduce EAGLE 3.1 — a major step forward in speculative decoding robustness, efficiency, and deployability.
EAGLE 3.1 Innovations
While speculative decoding performs well in controlled settings, performance often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts.
The EAGLE team traced this fragility to a phenomenon we call attention drift — as speculation depth increases, the drafter gradually shifts attention away from sink tokens and toward its own generated tokens.
We identified two underlying issues. First, the fused input representation becomes increasingly imbalanced as higher-layer hidden states dominate the drafter input. Second, hidden-state magnitude grows across speculation steps due to the unnormalized residual path. Together, these effects make the drafter progressively less stable at deeper speculation depths.
Figure 1: EAGLE 3 vs. EAGLE 3.1 architecture comparison. EAGLE 3.1 adds FC normalization after each target hidden state and feeds post-norm hidden states into the next decoding step.
To address this issue, EAGLE 3.1 introduces two key architectural improvements:
- FC normalization after each target hidden state and before the FC layer
- Feeding post-norm hidden states into the next decoding step
Intuitively, the post-norm design makes the method behave more like recursively invoking the drafter across decoding steps, rather than simply appending additional layers to the target model.
These changes significantly improve robustness across deployment scenarios. Compared with EAGLE 3, EAGLE 3.1 demonstrates:
- Better training-time to inference-time extrapolation
- Stronger long-context robustness
- Higher resilience to chat template and system prompt variation
- More stable acceptance length across diverse serving environments
In long-context workloads, EAGLE 3.1 achieves up to 2× longer acceptance length compared with EAGLE 3.
EAGLE 3.1 Training with TorchSpec
TorchSpec now provides efficient training support for EAGLE 3.1 and future speculative decoding algorithms.
By lowering training overhead and simplifying experimentation workflows, TorchSpec helps accelerate iteration and exploration for next-generation speculative decoding research and deployment.
Based on TorchSpec and vLLM, we also trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6:
https://huggingface.co/lightseekorg/kimi-k2.6-eagle3.1-mla
The model serves as an example of deploying EAGLE 3.1 with TorchSpec training and vLLM serving support on a real-world serving model.
EAGLE 3.1 Integration with vLLM
EAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation.
The integration includes:
- FC normalization support
- Post-norm hidden-state feedback
- Removal of hardcoded assumptions around target hidden states
At the same time, backward compatibility with existing EAGLE 3 checkpoints is fully preserved. As a result, EAGLE 3.1 draft models can be plugged directly through the same speculative-decoding code path, for example:
vllm serve nvidia/Kimi-K2.6-NVFP4 \
--trust-remote-code \
--tensor-parallel-size 4 \
--tool-call-parser kimi_k2 \
--enable-auto-tool-choice \
--reasoning-parser kimi_k2 \
--attention-backend tokenspeed_mla \
--speculative-config '{"model":"lightseekorg/kimi-k2.6-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}' \
--language-model-only
This makes draft-model upgrades in production vLLM serving smooth and easy.
The support has already been merged into the current main branch of vLLM and will be available via vLLM's nightly release as well as the upcoming v0.22.0 release.
As an early data point, we benchmarked the Kimi K2.6 EAGLE 3.1 draft model on Kimi-K2.6-NVFP4 with vLLM (TP=4, GB200, non-disagg) on the SPEED-Bench coding dataset. EAGLE 3.1 delivers 2.03× higher per-user output throughput at concurrency 1, and the speedup stays meaningful as concurrency scales (1.71× at C=4, 1.66× at C=16).
Figure 2: Per-user output throughput (TPS) on Kimi-K2.6-NVFP4 with vLLM, TP=4, GB200 on SPEED-Bench coding. EAGLE 3.1-MLA vs. no-spec baseline.
Open-Source Collaboration Across the Ecosystem
This collaboration between the EAGLE team, vLLM team, TorchSpec team represents a strong example of open-source collaboration across algorithm research, system optimization, and training infrastructure.
The EAGLE team continues advancing speculative decoding algorithms, vLLM helps bring these innovations into production inference systems at scale, and TorchSpec enables efficient training and rapid experimentation for future speculative decoding algorithms.
We are also grateful to NVIDIA for their GPU support and continued partnership. This support has played an important role in enabling the development, validation, and benchmarking efforts required to bring EAGLE 3.1 from algorithmic innovation to practical deployment.
Together, we hope to continue raising the overall baseline for speculative decoding and driving further improvements in token efficiency across the broader LLM ecosystem.