📏 长上下文
1M token 级别的上下文服务。
Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM
本文是『PD 分离服务』路径中 KV 传输优化的进阶篇：应先掌握 P-D disaggregation 基础与默认 NIXL pull connector 的工作流，再读本文的 push 改造。它与双向 KV 传输一文（2026-10-12）互为对照——push 优化单请求首响传输路径但不支持双向传输，两篇合看构成 connector 选型全景。
Improve TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM
本文是『PD 分离服务』路径中面向多轮/agentic 场景的进阶篇，应在掌握 P-D 分离与默认 NIXL pull connector 后阅读。它与 push connector 一文（2026-10-14）互补：push 优化单请求首响传输路径，本文解决跨轮 KV 复用，且双向能力目前构建在 pull 模式的 NixlConnector 之上。
DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0
Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.
How we trained the fastest DSpark for Kimi-K3 using GB300 NVL72
本文是『推测解码』学习路径中的 DSpark 训练篇:应在掌握推测解码基础与 DFlash/DSpark 算法原理之后阅读,聚焦训练数据(hidden states)如何跨节点规模化生产。它排在两篇自适应验证文章之前,为理解 confidence head 驱动的硬件感知验证调度提供背景与动机。
Tiered KV Cache Offloading in vLLM
A host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers — reducing recomputation and increasing serving capacity.
vLLM x AgentX: Optimizing for Real-World Agentic Serving
How vLLM optimizes KV cache management, parallelism, scheduling, and P/D disaggregation for agentic workloads, validated on SemiAnalysis AgentX with up to 130K tokens per GPU-second and a 14.6x-106x serving-cost advantage over Opus 5.
GLM 5.3 Optimizations, Part 1: Hybrid HiSparse Offloading in vLLM
vLLM integrates HiSparse as a pressure-driven memory tier that composes with the Hybrid Memory Allocator and KV offloading, letting GLM 5.3 requests keep decoding when their KV no longer fits in GPU memory, so concurrency stays high.
Announcing Day-0 Support for NVIDIA Nemotron 3.5 Lightning on vLLM
How vLLM serves NVIDIA Nemotron 3.5 Lightning with OpenAI-compatible APIs, speculative decoding, and BF16/NVFP4 checkpoints across NVIDIA GPUs and edge systems.
Efficient Decode Context Parallelism with vLLM for Long Context Workloads
Decode Context Parallelism (DCP) in vLLM shards KV cache across GPUs by sequence dimension, enabling 3× higher throughput on long-context agentic workloads compared to standard tensor parallelism.
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across NVIDIA and AMD GPUs.
From Day 0 to Production SLAs: Serving GLM-5.2 on 24 NVIDIA B300 GPUs with vLLM
本文是『PD 分离服务』学习路径末尾三篇生产规模实战之一，以 SLA 合规而非峰值吞吐为第一目标，与两篇 Qwen 吞吐导向文章形成方法论对照。建议在读完 Qwen3.5（25K TPS）篇建立 NVL72 + 混合架构 PD 基础后阅读，重点体会约束驱动的配置取舍与边界问题定位。
A Preview of Production-Scale Kimi K3 Support on vLLM
A preview of production-scale Kimi K3 support in vLLM, including KDA-aware prefix caching, fused kernels, optimized MXFP4 MoE, multimodal integration, and initial NVIDIA and AMD paths.
Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router
vLLM Semantic Router is expanding from intelligent routing into a system for building, evaluating, and running Mixture-of-Models.
TML Inkling on vLLM: Day-0 Support with Optimized Performance
vLLM brings day-0 support to TML Inkling, a 1T-parameter multimodal model, with MTP, long-context serving, parallelism, and up to 380 tokens per second per user on NVIDIA GB200 GPUs.
EAGLE3 Speculative Decoding on AMD Instinct GPUs: Training and Serving with vLLM and AMD Quark
『推测解码』学习路径中的动手实践深化篇:在掌握 speculative decoding 基本原理与 EAGLE 思路后阅读,可学到 EAGLE3 从训练、量化到服务落地的完整工程闭环;之后再读 2026-07-28 并行起草一文,了解超越 EAGLE-3 自回归限制的 P-EAGLE、DFlash、DSpark。
vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan
How HPC-Ops integrates Hopper-optimized attention and FP8 MoE backends into vLLM for Tencent Hunyuan Hy3, improving mixed-length decode, MoE latency, TTFT, and TPOT on NVIDIA H20.
Micro-Agent: Beat Frontier Models with Collaboration inside Model API
How vLLM Semantic Router turns vllm-sr/auto into a bounded micro-agent runtime for Confidence, Ratings, ReMoM, Fusion, Workflows, and benchmark-shaped collaboration.
MiniMax M3 in vLLM: Day-0 Serving for 1M-Token Multimodal Reasoning
How vLLM serves MiniMax M3 with MiniMax Sparse Attention, multimodal and reasoning parsers, MXFP8 weights, and long-context deployment recipes.
vLLM Semantic Router v0.3 Themis: From Signals to Stateful Production Routing
What vLLM Semantic Router v0.3 Themis adds for production routing: canonical config, inspectable signal-decision-policy flows, safer operations, CLI/dashboard/Kubernetes alignment, and replayable routing behavior.
Announcing Day-0 Support for NVIDIA Nemotron 3 Ultra on vLLM
How to serve NVIDIA Nemotron 3 Ultra with vLLM for long-running agentic reasoning, including BF16 and NVFP4 checkpoints, supported GPU configurations, OpenAI-compatible deployment, and NeMo RL integration.
Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents
How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cache-aware switch pricing, and replayable traces.
vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation
How to run vLLM on NVIDIA DGX Spark and GB10 systems, including unified memory behavior, NVFP4 Nemotron-3-Super serving, Docker deployment, Prometheus metrics, and local evaluation results.
EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec
『推测解码』路径的第三篇（进阶收尾，也是当前最新进展）：默认读者已掌握前两篇的推测解码原理与 Eagle3 hidden-state 机制，聚焦 EAGLE 3.1 对 attention drift 的架构级修复与生产落地，建议最后阅读。
Elastic Expert Parallelism in vLLM
How Elastic Expert Parallelism lets vLLM scale Mixture-of-Experts serving up or down at runtime by changing data-parallel workers, redistributing experts, and coordinating live topology changes without server restarts.
A First Comprehensive Study of TurboQuant: Accuracy and Performance
A vLLM study comparing TurboQuant KV-cache quantization with BF16 and FP8 across long-context and reasoning workloads, showing where 4-bit variants help, where accuracy drops, and why FP8 remains the default choice.
Serving Agentic Workloads at Scale with vLLM x Mooncake
How vLLM integrates Mooncake Store as a distributed KV cache for agentic workloads, reusing shared prefixes across turns and instances to improve throughput, TTFT, end-to-end latency, and multi-GPU scaling.
Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM
How to serve NVIDIA Nemotron 3 Nano Omni with vLLM for multimodal agentic AI, including BF16, FP8, and NVFP4 checkpoints, vision/audio/video inputs, supported GPUs, OpenAI-compatible APIs, and deployment recipes.
DeepSeek V4 in vLLM: Efficient Long-context Attention
A first-principles walkthrough of DeepSeek V4's long-context attention, and how we implemented it in vLLM.
The State of FP8 KV-Cache and Attention Quantization in vLLM
What vLLM FP8 KV-cache validation found across Hopper and Blackwell, covering attention quantization, Flash Attention 3 fixes, memory savings, decode speedups, and layers to skip.
Announcing Gemma 4 on vLLM: Byte for byte, the most capable open models
How vLLM supports Google's Gemma 4 open models across NVIDIA, AMD, Intel, and TPU backends, with multimodal inputs, agentic workflows, long context, function calling, and deployment recipes.
P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM
推测解码学习路径中的深度原理压轴篇:建议先读 Laguna XS.2 + DFlash 案例建立直观认识,再读 Speculators v0.5.0 掌握训练生态,最后读本文深入并行起草的架构、训练算法与 vLLM 内核级实现。
Run Highly Efficient and Accurate Multi-Agent AI with NVIDIA Nemotron 3 Super Using vLLM
How to serve NVIDIA Nemotron 3 Super with vLLM for multi-agent AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, Thinking Budget, MTP, supported GPUs, and OpenAI-compatible deployment.
vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System Brain
What vLLM Semantic Router v0.2 Athena adds: refreshed multilingual and multimodal routing models, ONNX and ROCm acceleration, safety and memory signals, long-context handling, and ClawOS orchestration.
vLLM Triton Attention Backend Deep Dive
A technical walkthrough of the vLLM Triton attention backend, covering performance-portable paged attention kernels, backend selection, autotuning, CUDA graph behavior, benchmarks, and NVIDIA, AMD, and Intel support.
Beyond Porting: How vLLM Orchestrates High-Performance Inference on AMD ROCm
How vLLM orchestrates high-performance inference on AMD ROCm with multiple attention backends, workload-aware prefill, extend, and decode routing, AITER primitives, MLA support, and MI300X-class benchmarks.
Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon Bedrock
How vLLM serves many fine-tuned MoE and dense models with Multi-LoRA, including fused MoE LoRA kernels, Triton compiler fixes, Split-K and CTA swizzling optimizations, and SageMaker AI and Bedrock tuning.
DeepSeek-V3.2 on GB300: Performance Breakthrough
What DeepSeek-V3.2 and DeepSeek-R1 benchmark results show on NVIDIA GB300 with vLLM, covering NVFP4 quantization, TP and EP deployment, throughput, and reproducible setup details.
vLLM Large Scale Serving: DeepSeek @ 2.2k tok/s/H200 with Wide-EP
How vLLM reaches 2.2k tokens per second per H200 for DeepSeek-style MoE serving with Wide-EP, async scheduling, dual-batch overlap, disaggregated serving, CUDA graphs, DeepGEMM, and expert load balancing.
AMD × vLLM Semantic Router: Building the System Intelligence Together
How AMD and vLLM Semantic Router build GPU-accelerated Mixture-of-Models routing with signals, semantic caching, response storage, PII, jailbreak, and hallucination guardrails.
Run Highly Efficient and Accurate AI Agents with NVIDIA Nemotron 3 Nano on vLLM
How to serve NVIDIA Nemotron 3 Nano with vLLM for efficient agentic AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, hybrid MoE architecture, Thinking Budget, supported GPUs, and OpenAI-compatible deployment.
Token-Level Truth: Real-Time Hallucination Detection for Production LLMs
How HaluGate adds token-level hallucination detection to vLLM Semantic Router by verifying assistant claims against tool outputs and grounding context in real time without LLM-as-judge overhead.
Shared Memory IPC Caching: Accelerating Data Transfer in LLM Inference Systems
How shared memory IPC caching in vLLM reduces redundant data transfers for multimodal and multi-process inference, improving prefill throughput and TTFT by sharing large inputs across coordinator and worker processes.
Fast and Affordable LLMs serving on Intel Arc Pro B-Series GPUs with vLLM
How vLLM serves LLMs on Intel Arc Pro B-Series GPUs with MoE optimizations, persistent kernels, multi-GPU scaling, LoRA, speculative decoding, structured outputs, and mixed-precision recipes.
Run Multimodal Reasoning Agents with NVIDIA Nemotron on vLLM
How to serve NVIDIA Nemotron Nano 2 VL with vLLM for multimodal reasoning agents, including video understanding, document intelligence, Efficient Video Sampling, 128K context, and OpenAI-compatible deployment.
From Monolithic to Modular: Scaling Semantic Routing with Extensible LoRA
How vLLM Semantic Router refactors its Rust classification layer with modular model support, Qwen3-Embedding, EmbeddingGemma, LoRA-based multi-task classification, and concurrent routing execution.
Zero-Reload Model Switching with vLLM Sleep Mode
How vLLM Sleep Mode enables fast model switching by hibernating weights to CPU RAM or discarding them while preserving process state, CUDA graphs, allocators, and kernel warmup to avoid full reloads.
Now Serving NVIDIA Nemotron with vLLM
How vLLM serves NVIDIA Nemotron Nano 2 for agentic reasoning, including hybrid Transformer-Mamba architecture, thinking budget control, open weights and data, throughput benefits, and deployment commands.
DeepSeek-V3.2-Exp in vLLM: Fine-Grained Sparse Attention in Action
How vLLM supports DeepSeek-V3.2-Exp with DeepSeek Sparse Attention, lightning indexer caches, separate prefill and decode paths, FlashMLA sparse attention, DeepGEMM kernels, and Blackwell deployment.
vLLM Now Supports Qwen3-Next: Hybrid Architecture with Extreme Efficiency
How vLLM supports Qwen3-Next with hybrid attention, Gated DeltaNet, full attention, high-sparsity MoE, multi-token prediction, hybrid KV cache management, Triton kernels, and CUDA graphs.
MiniMax-M1 Hybrid Architecture Meets vLLM: Long Context, Fast Inference
How vLLM serves MiniMax-M1's hybrid MoE architecture for long-context inference, covering model deployment, memory management, batched serving, backend optimizations, and Docker-based setup.
Llama 4 in vLLM
How vLLM serves Meta Llama 4 Scout and Maverick multimodal MoE models with long-context support, tensor parallel deployment, H100 and H200 guidance, FP8 variants, and performance tips.
Introducing AIBrix: A Scalable, Cost-Effective Control Plane for vLLM
What AIBrix adds as a Kubernetes control plane for vLLM: LoRA management, LLM gateway routing, autoscaling, unified runtime, distributed inference, distributed KV cache, heterogeneous serving, and GPU failure detection.
Structured Decoding in vLLM: a gentle introduction
How structured decoding works in vLLM, covering JSON outputs, grammar-guided generation, outlines, XGrammar, TPOT improvements, constrained decoding, and agentic workflow use cases.
vLLM 2024 Retrospective and 2025 Vision
A vLLM 2024 retrospective and 2025 roadmap covering community growth, model and hardware support, production adoption, office hours, ecosystem partnerships, and the path toward universal open-source serving.
Serving LLMs on AMD MI300X: Best Practices
Best practices for serving LLMs with vLLM on AMD MI300X, covering ROCm setup, Llama 3.1 70B and 405B benchmarks, chunked prefill, multi-step scheduling, prefix caching, graph capture, and AMD tuning.
Announcing Llama 3.1 Support in vLLM
How vLLM supports Meta Llama 3.1 models, including 128K context, Llama 3.1 405B serving, chunked prefill, FP8 quantization, tensor and pipeline parallelism, CPU offloading, and early performance results.