📊 性能评测
AgentX、MLPerf 等基准与 Pareto 分析。
Finish RL Training Faster: Bitwise-Consistent, High-Performance, Agent-Driven
We make vLLM rollout and the training forward give bitwise identical logprobs on DeepSeek-V4-Flash and Nemotron-3.5-Lightning-30B-A3B (NVFP4), keep 91% of rollout throughput on DeepSeek-V4-Flash, and open-source the agent skill.
Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM
本文是『PD 分离服务』路径中 KV 传输优化的进阶篇：应先掌握 P-D disaggregation 基础与默认 NIXL pull connector 的工作流，再读本文的 push 改造。它与双向 KV 传输一文（2026-10-12）互为对照——push 优化单请求首响传输路径但不支持双向传输，两篇合看构成 connector 选型全景。
Adaptive Verification Without a Confidence Head
推测解码路径中自适应验证系列的第二篇:上一篇(2026-08-14 DSpark AV)依赖草稿模型自带的 confidence head,本篇用在线拟合的接受概率估计器去掉该依赖,使任意既有草稿 checkpoint 免重训接入 AV;可继续跟进 AV 在更多 speculator 上的扩展。
Improve TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM
本文是『PD 分离服务』路径中面向多轮/agentic 场景的进阶篇，应在掌握 P-D 分离与默认 NIXL pull connector 后阅读。它与 push connector 一文（2026-10-14）互补：push 优化单请求首响传输路径，本文解决跨轮 KV 复用，且双向能力目前构建在 pull 模式的 NixlConnector 之上。
vLLM Support for NVIDIA Vera Rubin NVL72: 7.8x Throughput over GB200 NVL72
vLLM now runs on NVIDIA Vera Rubin NVL72 with day-0 model support, Rubin-tuned FlashInfer kernels, and locality-aware MoE, delivering 7.8x the per-GPU throughput of GB200 NVL72 on AgentX.
DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0
Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.
Taking vLLM Apart: A Practical Guide to Disaggregated Serving
『PD 分离服务』学习路径的核心实操文章:建议在理解 prefill/decode 差异并读过路由层介绍(如 vLLM Router 一文)之后阅读,再延伸到 Qwen3.8-2.4T PD 容量规划与 bidirectional KV transfer 等专题文章。
Watermarking in vLLM
How vLLM implements distribution-preserving Gumbel-max text watermarking with efficient GPU kernels, statistical detection, speculative decoding, and repeated-context safeguards.
Announcing vllm-metal: Concurrent Serving on Apple Silicon
vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M5 prefill acceleration.
PD Serving of Qwen3.8-2.4T
本文是『PD 分离服务』路径末尾三篇生产实战的收官篇，开篇即引用 Qwen3.5（25K TPS）篇的结果并把目标从“冲吞吐”扩展为完整 Pareto 前沿。建议三篇中最后阅读，重点吸收其分步测量、先估容量再选拓扑的可复用调优方法论。
Scaling Multi-GPU Video Captioning with PyNvVideoCodec and vLLM
How to leverage NVIDIA Hardware Video Decoders to Achieve Multi-GPU Scaling in Video Captioning and Description tasks.
vLLM x Novita AI: Chord, Faster INT4 MoE for Kimi K2.x. Up to 1.3x on H200, 2.15x on Untuned B300
Novita AI has open-sourced Chord, a high-performance W4A16 MoE CUDA kernel for Kimi K2.x serving shapes, with a Humming-compatible indexed path and grouped SM90 operators.
How we trained the fastest DSpark for Kimi-K3 using GB300 NVL72
本文是『推测解码』学习路径中的 DSpark 训练篇:应在掌握推测解码基础与 DFlash/DSpark 算法原理之后阅读,聚焦训练数据(hidden states)如何跨节点规模化生产。它排在两篇自适应验证文章之前,为理解 confidence head 驱动的硬件感知验证调度提供背景与动机。
Kimi K3 Performance Optimizations in vLLM: The Road to 2.8× Throughput
Kimi K3 serving optimizations across scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels.
Tiered KV Cache Offloading in vLLM
A host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers — reducing recomputation and increasing serving capacity.
Following the Bottleneck: Optimizing MiniMax M3 on AMD Instinct MI355X
A performance model for LLM serving: inspect local shapes, remove repeated work, verify data movement and dispatch, then follow the queue.
vLLM x AgentX: Optimizing for Real-World Agentic Serving
How vLLM optimizes KV cache management, parallelism, scheduling, and P/D disaggregation for agentic workloads, validated on SemiAnalysis AgentX with up to 130K tokens per GPU-second and a 14.6x-106x serving-cost advantage over Opus 5.
GLM 5.3 Optimizations, Part 1: Hybrid HiSparse Offloading in vLLM
vLLM integrates HiSparse as a pressure-driven memory tier that composes with the Hybrid Memory Allocator and KV offloading, letting GLM 5.3 requests keep decoding when their KV no longer fits in GPU memory, so concurrency stays high.
MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3
How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.
Exploring Speculative Decoding in vLLM on AMD GPUs
本文适合放在『推测解码』学习路径的中后段:在掌握推测解码基本原理与 vLLM 基础后,作为方法选型、配置与调优的参考手册随时查阅;附录的逐位置接受率热图与逐模型 serving 命令适合在实际部署前对照使用。
Large-Scale Sharded Weight Transfer with Ray Direct Transport (RDT) in vLLM
We implement a native sharded weight transfer engine in vLLM utilizing Ray Direct Transport (RDT), achieving weight transfer for the Kimi K2 model in BF16 on 48 8xH100 nodes in 7.53s
IsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRL
IsoExec unifies numerical execution across SkyRL's vLLM and Megatron runtimes, reducing the average rollout-versus-training logprob difference below 1e-6 on Qwen3.5-35B-A3B with 25% overhead.
VeRL-Omni v0.2.0: Faster Diffusion RL and Stable Omni Training
A release focused on higher-throughput diffusion rollout, reusable omni adapters, and broader recipe coverage.
Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni
Distributed Layerwise Offload shards and streams DiT weights across devices, serving a measured 124 GB Cosmos3 model on 64 GB HBM and estimating a path toward 200B+ models.
Adaptive Verification in vLLM: DSpark confidence-scheduled verification
本文是『推测解码』路径中自适应验证主题的第一篇，讲 DSpark 如何用置信度头按存活概率调度验证预算；下一篇是无 confidence head 的在线估计器方案，建议对比阅读以理解两类自适应验证的取舍。
Day 0 Support for Qwen3.8-2.4T-A95B on vLLM
Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.
Announcing Day-0 Support for NVIDIA Nemotron 3.5 Lightning on vLLM
How vLLM serves NVIDIA Nemotron 3.5 Lightning with OpenAI-compatible APIs, speculative decoding, and BF16/NVFP4 checkpoints across NVIDIA GPUs and edge systems.
Efficient Decode Context Parallelism with vLLM for Long Context Workloads
Decode Context Parallelism (DCP) in vLLM shards KV cache across GPUs by sequence dimension, enabling 3× higher throughput on long-context agentic workloads compared to standard tensor parallelism.
vLLM Reaches 25K Total TPS/GPU on Qwen3.5
本文是『PD 分离服务』路径末尾三篇生产规模实战中建议最先读的一篇：它是 Qwen3.8 篇的直接前置（后者开篇引用本文结果），也是 GB200/GB300 NVL72 + 混合 GDN 架构 PD serving 的基础认知入口，可再顺读 GLM-5.2 篇对照 SLA 导向的取舍。
Optimizing vLLM on Arm CPUs
An overview of Arm CPU enablement and inference performance optimizations in vLLM.
Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding
『推测解码』学习路径中的进阶演进篇:先读 speculative decoding 基础与 EAGLE-3 实践(如 2026-07-13 的 EAGLE3 on AMD Instinct 一文),再读本文理解 P-EAGLE、DFlash、DSpark 如何突破自回归起草的串行上限,可作为该主题的前沿收尾综述。
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across NVIDIA and AMD GPUs.
Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving
『PD 分离服务』路径的横向扩展篇:在掌握按阶段拆分的 PD 分离之后阅读,理解按组件(Attention vs FFN)拆分的另一维度;其异步 connector 本就部署在 PD 分离之上,与 hybrid SSM 篇(同一 PD 框架下的模型适配)互为补充。
From Day 0 to Production SLAs: Serving GLM-5.2 on 24 NVIDIA B300 GPUs with vLLM
本文是『PD 分离服务』学习路径末尾三篇生产规模实战之一，以 SLA 合规而非峰值吞吐为第一目标，与两篇 Qwen 吞吐导向文章形成方法论对照。建议在读完 Qwen3.5（25K TPS）篇建立 NVL72 + 混合架构 PD 基础后阅读，重点体会约束驱动的配置取舍与边界问题定位。
A Preview of Production-Scale Kimi K3 Support on vLLM
A preview of production-scale Kimi K3 support in vLLM, including KDA-aware prefix caching, fused kernels, optimized MXFP4 MoE, multimodal integration, and initial NVIDIA and AMD paths.
Keeping vLLM Production Quality: A Look Inside CI, Benchmarking, and the Release Process
How vLLM maintains production quality with extensive CI across diverse accelerators, nightly performance benchmark and accuracy evaluation, and a two-week release process.
TML Inkling on vLLM: Day-0 Support with Optimized Performance
vLLM brings day-0 support to TML Inkling, a 1T-parameter multimodal model, with MTP, long-context serving, parallelism, and up to 380 tokens per second per user on NVIDIA GB200 GPUs.
vLLM x TileRT: Specialized Decode for Latency-Critical Serving
『PD 分离服务』路径的生态应用篇:先理解 PD 分离动机与 vLLM V1 connector 接口,再读本文看 decode 侧可插拔的实际落地;三篇中最易读,适合作为路径收官时对“分离改变推理栈形态”的总结案例。
EAGLE3 Speculative Decoding on AMD Instinct GPUs: Training and Serving with vLLM and AMD Quark
『推测解码』学习路径中的动手实践深化篇:在掌握 speculative decoding 基本原理与 EAGLE 思路后阅读,可学到 EAGLE3 从训练、量化到服务落地的完整工程闭环;之后再读 2026-07-28 并行起草一文,了解超越 EAGLE-3 自回归限制的 P-EAGLE、DFlash、DSpark。
vime + ROCm: End-to-End RL Post-Training on AMD Instinct™ GPUs
Announcing ROCm support for vime, now running end-to-end on AMD Instinct MI355X GPUs with prebuilt container.
vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan
How HPC-Ops integrates Hopper-optimized attention and FP8 MoE backends into vLLM for Tencent Hunyuan Hy3, improving mixed-length decode, MoE latency, TTFT, and TPOT on NVIDIA H20.
Experience and Lessons Learned from Serving Multi-Stage Qwen3-Omni in vLLM-Omni
How vLLM-Omni serves and optimizes Qwen3-Omni with staged Thinker-Talker-Code2Wav execution, batching, CUDA Graphs, async chunk, async output, replicas, hot-path cleanup, and perf validation.
Micro-Agent: Beat Frontier Models with Collaboration inside Model API
How vLLM Semantic Router turns vllm-sr/auto into a bounded micro-agent runtime for Confidence, Ratings, ReMoM, Fusion, Workflows, and benchmark-shaped collaboration.
Engineering TTS Inference in vLLM-Omni
How vLLM-Omni supports and optimizes Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro with staged serving, batching, CUDA Graphs, and model-specific kernels.
Beyond One Model: Fusion in vLLM Semantic Router
How vLLM Semantic Router Fusion runs a panel of models, uses a judge to analyze agreement and gaps, and synthesizes one answer while preserving routing policy, traces, and OpenAI-compatible serving.
MiniMax M3 in vLLM: Day-0 Serving for 1M-Token Multimodal Reasoning
How vLLM serves MiniMax M3 with MiniMax Sparse Attention, multimodal and reasoning parsers, MXFP8 weights, and long-context deployment recipes.
Announcing vime: A Simple, Stable, and Efficient RL Framework for LLMs
vime connects slime's training stack with vLLM rollouts to provide a simple, stable, and efficient RL post-training pipeline.
Announcing Day-0 Support for NVIDIA Nemotron 3 Ultra on vLLM
How to serve NVIDIA Nemotron 3 Ultra with vLLM for long-running agentic reasoning, including BF16 and NVFP4 checkpoints, supported GPU configurations, OpenAI-compatible deployment, and NeMo RL integration.
Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI
What the DeepLearning.AI vLLM course teaches: optimizing, deploying, and benchmarking LLM inference with LLM Compressor quantization, GuideLLM, KV cache sizing, serving, and memory tradeoffs.
Accelerating vLLM-Omni Inference with AutoRound Quantization
How AutoRound integrates with vLLM-Omni to serve W4A16 quantized multimodal, diffusion, image, and video models with smaller checkpoints, preserved quality, Intel XPU acceleration, and NVIDIA GPU support.
vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation
How to run vLLM on NVIDIA DGX Spark and GB10 systems, including unified memory behavior, NVFP4 Nemotron-3-Super serving, Docker deployment, Prometheus metrics, and local evaluation results.
From Text to Multimodal Routing: Hardening Vision Signals in vLLM Semantic Router
How vLLM Semantic Router hardens multimodal routing by turning visual evidence into trustworthy signals, debugging a Rust/Candle vision-encoder parity issue, and validating image signal correctness for production policy.
Speculators v0.5.0: DFlash Support and Online Training
推测解码路径的中间篇:建议先读 Laguna XS.2 + DFlash 案例建立直观认识,再读本文掌握 DFlash 训练机制与 Speculators 训练生态,最后读 P-EAGLE 深入并行起草的架构与内核实现。
Native RL APIs in vLLM
How vLLM native RL APIs standardize weight syncing and asynchronous RL serving with NCCL and CUDA IPC transfer backends, pause mode, and fixes for fragile DPEP and disaggregated rollout deployments.
Accelerating Laguna XS.2 Inference with vLLM, Speculators, and LLM Compressor
推测解码学习路径的入门案例篇:建议最先阅读,用最短篇幅建立 DFlash 落地效果与生态协作的直观认识,再读 Speculators v0.5.0 了解训练细节,最后读 P-EAGLE 深入原理。
EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec
『推测解码』路径的第三篇（进阶收尾，也是当前最新进展）：默认读者已掌握前两篇的推测解码原理与 Eagle3 hidden-state 机制，聚焦 EAGLE 3.1 对 attention drift 的架构级修复与生产落地，建议最后阅读。
vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache
How PegaFlow integrates with vLLM as an external KV cache service, using a Rust daemon, CUDA IPC, RDMA, SSD caching, and the external KV connector to improve startup, sharing, throughput, and cache lifecycle.
Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models
How VeRL-Omni extends verl with vLLM-Omni for reinforcement learning post-training of diffusion and multimodal generative models, including efficient rollouts, reward inference, trainers, hardware support, and recipes.
Elastic Expert Parallelism in vLLM
How Elastic Expert Parallelism lets vLLM scale Mixture-of-Experts serving up or down at runtime by changing data-parallel workers, redistributing experts, and coordinating live topology changes without server restarts.
vLLM Tops the Artificial Analysis Leaderboard
How vLLM achieved leading Artificial Analysis results for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B using open-source kernel fusion, speculative decoding, Blackwell optimizations, and model-specific serving work.
A First Comprehensive Study of TurboQuant: Accuracy and Performance
A vLLM study comparing TurboQuant KV-cache quantization with BF16 and FP8 across long-context and reasoning workloads, showing where 4-bit variants help, where accuracy drops, and why FP8 remains the default choice.
Serving Agentic Workloads at Scale with vLLM x Mooncake
How vLLM integrates Mooncake Store as a distributed KV cache for agentic workloads, reusing shared prefixes across turns and instances to improve throughput, TTFT, end-to-end latency, and multi-GPU scaling.
Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM
How to serve NVIDIA Nemotron 3 Nano Omni with vLLM for multimodal agentic AI, including BF16, FP8, and NVFP4 checkpoints, vision/audio/video inputs, supported GPUs, OpenAI-compatible APIs, and deployment recipes.
The State of FP8 KV-Cache and Attention Quantization in vLLM
What vLLM FP8 KV-cache validation found across Hopper and Blackwell, covering attention quantization, Flash Attention 3 fixes, memory savings, decode speedups, and layers to skip.
Disaggregated Serving for Hybrid SSM Models in vLLM
『PD 分离服务』路径的进阶深化篇:建议先读 vLLM 基础 PD 分离文章(介绍 NIXL connector 的 v0-disagg-prefill)掌握标准流程,再读本文理解 hybrid SSM 模型带来的底层改造;可与 AFD Plugin(按组件拆分)、TileRT(decode 可插拔)两篇生态文对照阅读。
vLLM Korea Meetup 2026 Wrap-Up
What the vLLM Korea Meetup 2026 covered: community growth, vLLM V1 updates, production stack adoption, accelerator integration, vllm-playground, and real-world LLM serving.
Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation
How single-node prefill/decode disaggregation in vLLM uses AMD MORI-IO on an 8-GPU MI300X node to separate prefill and decode, transfer KV cache efficiently, stabilize ITL, and improve goodput.
Announcing Gemma 4 on vLLM: Byte for byte, the most capable open models
How vLLM supports Google's Gemma 4 open models across NVIDIA, AMD, Intel, and TPU backends, with multimodal inputs, agentic workflows, long context, function calling, and deployment recipes.
Extracting hidden states from vLLM
How vLLM extracts verifier hidden states through dummy draft models and KV Connector APIs for speculative decoding, enabling offline and online Speculators training without patching vLLM internals.
Model Runner V2: A Modular and Faster Core for vLLM
How Model Runner V2 reworks vLLM's execution core with modular model logic, GPU-native input preparation, stable persistent batching, async-first scheduling, and no API changes.
P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM
推测解码学习路径中的深度原理压轴篇:建议先读 Laguna XS.2 + DFlash 案例建立直观认识,再读 Speculators v0.5.0 掌握训练生态,最后读本文深入并行起草的架构、训练算法与 vLLM 内核级实现。
Run Highly Efficient and Accurate Multi-Agent AI with NVIDIA Nemotron 3 Super Using vLLM
How to serve NVIDIA Nemotron 3 Super with vLLM for multi-agent AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, Thinking Budget, MTP, supported GPUs, and OpenAI-compatible deployment.
vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System Brain
What vLLM Semantic Router v0.2 Athena adds: refreshed multilingual and multimodal routing models, ONNX and ROCm acceleration, safety and memory signals, long-context handling, and ClawOS orchestration.
vLLM Triton Attention Backend Deep Dive
A technical walkthrough of the vLLM Triton attention backend, covering performance-portable paged attention kernels, backend selection, autotuning, CUDA graph behavior, benchmarks, and NVIDIA, AMD, and Intel support.
Beyond Porting: How vLLM Orchestrates High-Performance Inference on AMD ROCm
How vLLM orchestrates high-performance inference on AMD ROCm with multiple attention backends, workload-aware prefill, extend, and decode routing, AITER primitives, MLA support, and MI300X-class benchmarks.
Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon Bedrock
How vLLM serves many fine-tuned MoE and dense models with Multi-LoRA, including fused MoE LoRA kernels, Triton compiler fixes, Split-K and CTA swizzling optimizations, and SageMaker AI and Bedrock tuning.
DeepSeek-V3.2 on GB300: Performance Breakthrough
What DeepSeek-V3.2 and DeepSeek-R1 benchmark results show on NVIDIA GB300 with vLLM, covering NVFP4 quantization, TP and EP deployment, throughput, and reproducible setup details.
Driving vLLM WideEP and Large-Scale Serving Toward Maturity on Blackwell (Part I)
How vLLM improves WideEP and large-scale DeepSeek-style MoE serving on NVIDIA GB200 with NVFP4 and FP8 kernels, fusion, prefill/decode disaggregation, weight offloading, and reduced chunking overhead.
GPT-OSS Performance Optimizations on NVIDIA Blackwell: Pushing the Pareto Frontier
How vLLM and NVIDIA optimized GPT-OSS on Blackwell with FlashInfer, torch.compile fusion, FP8 KV cache, async scheduling, stream interval tuning, and deployment recipes that improve throughput and interactivity.
Streaming Requests & Realtime API in vLLM
How vLLM supports streamable inputs and a Realtime WebSocket API for audio, video, robotics, and low-latency applications that need incremental input processing instead of complete prompts.
Building Mixture-of-Models on AMD GPUs with vLLM-SR
How vLLM Semantic Router builds a Mixture-of-Models system on AMD MI300X and MI355X GPUs, routing across specialized models with signals, decisions, safety checks, semantic caching, and live MoM deployment.
Inside vLLM’s New KV Offloading Connector: Smarter Memory Transfer for Maximizing Inference Throughput
How vLLM's asynchronous KV offloading connector stores KV cache in CPU memory to reduce recomputation, improve throughput under memory pressure, and support pluggable offload backends.
vLLM Semantic Router v0.1 Iris: The First Major Release
What vLLM Semantic Router v0.1 Iris introduces: signal-decision plugin architecture, model selection, safety filtering, semantic caching, hallucination detection, LoRA-based routing models, and production-ready MoM routing.
Introducing vLLM Playground: A Modern Web Interface for Managing and Interacting with vLLM Servers
How vLLM Playground provides a web UI for starting, configuring, testing, and monitoring vLLM servers across local macOS, Linux GPU or CPU, Kubernetes, and OpenShift environments.
vLLM-Omni Diffusion Cache Acceleration
How vLLM-Omni speeds up diffusion model inference with Cache-DiT and TeaCache, reusing intermediate computations across timesteps to deliver 1.5x to 2x image generation speedups with minimal quality loss.
vLLM Large Scale Serving: DeepSeek @ 2.2k tok/s/H200 with Wide-EP
How vLLM reaches 2.2k tokens per second per H200 for DeepSeek-style MoE serving with Wide-EP, async scheduling, dual-batch overlap, disaggregated serving, CUDA graphs, DeepGEMM, and expert load balancing.
AMD × vLLM Semantic Router: Building the System Intelligence Together
How AMD and vLLM Semantic Router build GPU-accelerated Mixture-of-Models routing with signals, semantic caching, response storage, PII, jailbreak, and hallucination guardrails.
Encoder Disaggregation for Scalable Multimodal Model Serving
How vLLM EPD separates visual encoders from text prefill and decode, covering LMM serving, GPU resource scaling, multimodal interference, and pipelined execution.
Run Highly Efficient and Accurate AI Agents with NVIDIA Nemotron 3 Nano on vLLM
How to serve NVIDIA Nemotron 3 Nano with vLLM for efficient agentic AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, hybrid MoE architecture, Thinking Budget, supported GPUs, and OpenAI-compatible deployment.
Token-Level Truth: Real-Time Hallucination Detection for Production LLMs
How HaluGate adds token-level hallucination detection to vLLM Semantic Router by verifying assistant claims against tool outputs and grounding context in real time without LLM-as-judge overhead.
vLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale Serving
『PD 分离服务』学习路径的入门/生态篇:篇幅短,快速建立『P/D 分离在生产中需要专门编排层』的认知;建议先于深度实操文章(如 disaggregated serving guide)阅读,为理解 router/proxy 在 P/D 架构中的角色打基础。
Diving into speculative decoding training support for vLLM with Speculators v0.3.0
『推测解码』路径的第二篇（工程进阶）：承接第一篇对 EAGLE/Medusa 的概念铺垫，深入 Eagle3 draft model 的训练数据构造、训练技术与 vLLM 部署细节，建议在读过入门篇后、读 EAGLE 3.1 之前阅读。
Advancing Low‑Bit Quantization for LLMs: AutoRound x LLM Compressor
How Intel AutoRound integrates with LLM Compressor to produce low-bit quantized checkpoints for vLLM, using tuning-based PTQ, W4A16 and related formats, compressed-tensors compatibility, and lightweight calibration.
Tracing Hanging and Complicated GPU Kernels Down To The Source Code
How vLLM developers debug hanging and complex CUDA kernels by triggering GPU core dumps, identifying stuck kernels, and mapping failures back to source code lines for faster kernel debugging.
Announcing vLLM-Omni: Easy, Fast, and Cheap Omni-Modality Model Serving
What vLLM-Omni adds to the vLLM ecosystem: omni-modality serving for text, image, video, and audio, diffusion and non-autoregressive generation support, disaggregated stages, OpenAI-compatible APIs, and pipelined execution.
Building Clean, Maintainable vLLM Modifications Using the Plugin System
How the vLLM plugin system helps teams customize scheduling, KV-cache behavior, hardware integrations, and model execution without long-lived forks, monkey patches, or brittle internal modifications.
Docker Model Runner Integrates vLLM for High-Throughput Inferencing
How Docker Model Runner integrates vLLM as an inference backend, letting developers run safetensors models with high-throughput serving, PagedAttention, streaming, and OpenAI-compatible APIs from Docker workflows.
Shared Memory IPC Caching: Accelerating Data Transfer in LLM Inference Systems
How shared memory IPC caching in vLLM reduces redundant data transfers for multimodal and multi-process inference, improving prefill throughput and TTFT by sharing large inputs across coordinator and worker processes.
Fast and Affordable LLMs serving on Intel Arc Pro B-Series GPUs with vLLM
How vLLM serves LLMs on Intel Arc Pro B-Series GPUs with MoE optimizations, persistent kernels, multi-GPU scaling, LoRA, speculative decoding, structured outputs, and mixed-precision recipes.
No More Train-Inference Mismatch: Bitwise Consistent On-Policy Reinforcement Learning with vLLM and TorchTitan
How vLLM and TorchTitan demonstrate bitwise consistent on-policy RL by matching training and inference numerics, using batch-invariant kernels to reduce train-inference mismatch and stabilize reinforcement learning.
Run Multimodal Reasoning Agents with NVIDIA Nemotron on vLLM
How to serve NVIDIA Nemotron Nano 2 VL with vLLM for multimodal reasoning agents, including video understanding, document intelligence, Efficient Video Sampling, 128K context, and OpenAI-compatible deployment.
Chasing 100% Accuracy: A Deep Dive into Debugging Kimi K2's Tool-Calling on vLLM
How vLLM debugged Kimi K2 tool-calling accuracy, covering chat-template compatibility, add_generation_prompt handling, schema validation failures, benchmark fixes, and tool-use reliability.
From Monolithic to Modular: Scaling Semantic Routing with Extensible LoRA
How vLLM Semantic Router refactors its Rust classification layer with modular model support, Qwen3-Embedding, EmbeddingGemma, LoRA-based multi-task classification, and concurrent routing execution.
Zero-Reload Model Switching with vLLM Sleep Mode
How vLLM Sleep Mode enables fast model switching by hibernating weights to CPU RAM or discarding them while preserving process state, CUDA graphs, allocators, and kernel warmup to avoid full reloads.
Now Serving NVIDIA Nemotron with vLLM
How vLLM serves NVIDIA Nemotron Nano 2 for agentic reasoning, including hybrid Transformer-Mamba architecture, thinking budget control, open weights and data, throughput benefits, and deployment commands.
No More Retokenization Drift: Returning Token IDs via the OpenAI Compatible API Matters in Agent RL
How vLLM's OpenAI-compatible API can return prompt and response token IDs to prevent retokenization drift in agent reinforcement learning, preserving exact sampled sequences for stable on-policy updates.
vLLM TPU: A New Unified Backend Supporting PyTorch and JAX on TPU
How the redesigned vLLM TPU backend uses tpu-inference, JAX-to-XLA lowering, Torchax, ragged paged attention, and unified PyTorch and JAX support to improve TPU performance and model coverage.
SemiAnalysis InferenceMAX: vLLM and NVIDIA Accelerate Blackwell Inference
How vLLM and NVIDIA optimize Blackwell inference for SemiAnalysis InferenceMAX, improving gpt-oss 120B and Llama 3.3 70B throughput with FP4 kernels, scheduling work, and Pareto-frontier benchmarking.
DeepSeek-V3.2-Exp in vLLM: Fine-Grained Sparse Attention in Action
How vLLM supports DeepSeek-V3.2-Exp with DeepSeek Sparse Attention, lightning indexer caches, separate prefill and decode paths, FlashMLA sparse attention, DeepGEMM kernels, and Blackwell deployment.
The First vLLM Meetup in Korea
What the first vLLM Korea meetup covered: community adoption, llm-d, TPU integration, contribution workflows, hardware plugin architecture, Rebellions NPU work, and production inference lessons.
vLLM Semantic Router: Next Phase in LLM inference
How vLLM Semantic Router routes requests by intent, covering semantic classification, smart reasoning-path selection, Rust and Candle execution, and Kubernetes Envoy integration for efficient inference.
vLLM Now Supports Qwen3-Next: Hybrid Architecture with Extreme Efficiency
How vLLM supports Qwen3-Next with hybrid attention, Gated DeltaNet, full attention, high-sparsity MoE, multi-token prediction, hybrid KV cache management, Triton kernels, and CUDA graphs.
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
How vLLM's inference engine works, covering PagedAttention, continuous batching, prefix caching, speculative decoding, multi-GPU serving, scheduling, and benchmarking for high-throughput LLM workloads.
Introduction to torch.compile and How It Works with vLLM
How torch.compile works inside vLLM, including TorchDynamo graph capture, TorchInductor code generation, custom compiler passes, graph breaks, model compilation strategy, and performance optimization.
GLM-4.5 Meets vLLM: Built for Intelligent Agents
How to run GLM-4.5 and GLM-4.5V with vLLM for intelligent agents, including hybrid reasoning modes, FP8 and BF16 serving, multimodal support, and NVIDIA Blackwell and Hopper deployment.
CUDA Core Dump: An Effective Tool to Debug Memory Access Issues and Beyond
How to debug vLLM CUDA illegal memory access errors with CUDA core dumps, environment variables, cuda-gdb, and GPU state inspection when Python stack traces or CUDA_LAUNCH_BLOCKING are insufficient.
vLLM Now Supports gpt-oss
How vLLM supports gpt-oss 20B and 120B on NVIDIA Blackwell, Hopper, and AMD GPUs, with MXFP4 MoE kernels, efficient attention, hybrid KV cache allocation, and built-in tool support.
MiniMax-M1 Hybrid Architecture Meets vLLM: Long Context, Fast Inference
How vLLM serves MiniMax-M1's hybrid MoE architecture for long-context inference, covering model deployment, memory management, batched serving, backend optimizations, and Docker-based setup.
Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU
How vLLM hardware plugins decouple backend integrations from core vLLM, using Platform, Executor, Worker, ModelRunner, AttentionBackend, and Communicator hooks to support Ascend NPU and IBM Spyre.
Accelerating RLHF with vLLM, Best Practice from OpenRLHF
How OpenRLHF uses vLLM, Ray, ZeRO-3, AutoTP, Ray placement groups, and weight synchronization to accelerate PPO and RLHF sample generation for reasoning models with long chain-of-thought outputs.
Transformers modeling backend integration in vLLM
How vLLM integrates the Hugging Face Transformers modeling backend to serve more model architectures efficiently, including text and vision-language models through model_impl=\"transformers\".
Llama 4 in vLLM
How vLLM serves Meta Llama 4 Scout and Maverick multimodal MoE models with long-context support, tensor parallel deployment, H100 and H200 guidance, FP8 variants, and performance tips.
PTPC-FP8: Boosting vLLM Performance on AMD ROCm
How PTPC-FP8 quantization improves vLLM performance on AMD ROCm by combining per-token activation scaling and per-channel weight scaling for near-BF16 accuracy with FP8 speed.
Introducing AIBrix: A Scalable, Cost-Effective Control Plane for vLLM
What AIBrix adds as a Kubernetes control plane for vLLM: LoRA management, LLM gateway routing, autoscaling, unified runtime, distributed inference, distributed KV cache, heterogeneous serving, and GPU failure detection.
Distributed Inference with vLLM
A guide to distributed inference in vLLM, covering tensor parallelism, pipeline parallelism, multi-GPU and multi-node serving, KV cache challenges, speculative decoding, communication kernels, and control-plane design.
vLLM V1: A Major Upgrade to vLLM's Core Architecture
What changed in vLLM V1: a re-architected engine with a simpler scheduler, near-zero-overhead prefix caching, cleaner tensor parallelism, multiprocessing API server, and default optimizations for higher-throughput serving.
Introducing vLLM Inference Provider in Llama Stack
How Llama Stack integrates vLLM as an inference provider through remote and inline providers, enabling OpenAI-compatible vLLM serving for local and Kubernetes generative AI application deployments.
High Performance and Easy Deployment of vLLM in K8S with vLLM production-stack
What vLLM production-stack adds for Kubernetes serving: prefix-aware routing, LMCache-backed KV cache sharing, autoscaling, observability, fault tolerance, and cluster deployment with higher throughput and lower latency.
Structured Decoding in vLLM: a gentle introduction
How structured decoding works in vLLM, covering JSON outputs, grammar-guided generation, outlines, XGrammar, TPOT improvements, constrained decoding, and agentic workflow use cases.
vLLM 2024 Retrospective and 2025 Vision
A vLLM 2024 retrospective and 2025 roadmap covering community growth, model and hardware support, production adoption, office hours, ecosystem partnerships, and the path toward universal open-source serving.
Installing and Developing vLLM with Ease
How to install and develop vLLM using stable releases, nightly wheels, uv, source builds, Python and C++/CUDA workflows, and version tracking for production deployments.
Serving LLMs on AMD MI300X: Best Practices
Best practices for serving LLMs with vLLM on AMD MI300X, covering ROCm setup, Llama 3.1 70B and 405B benchmarks, chunked prefill, multi-step scheduling, prefix caching, graph capture, and AMD tuning.
How Speculative Decoding Boosts vLLM Performance by up to 2.8x
『推测解码』学习路径的第一篇（入门）：先读它建立 draft/target 协作、n-gram、Medusa/EAGLE 等基础概念和 vLLM 系统架构认知，再进入后两篇分别学习 Eagle3 draft model 的训练工程与 EAGLE 3.1 的算法改进。
vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction
What changed in vLLM v0.6.0 to improve throughput and latency, including API server isolation, reduced CPU overhead, multi-step scheduling, async execution, and benchmarks against earlier vLLM versions.
vLLM’s Open Governance and Performance Roadmap
vLLM's open governance and performance roadmap, covering LF AI and Data incubation, public benchmarks, optimized kernels, async scheduling, API frontend overhead, torch.compile, disaggregated prefill, and community research.
Announcing Llama 3.1 Support in vLLM
How vLLM supports Meta Llama 3.1 models, including 128K context, Llama 3.1 405B serving, chunked prefill, FP8 quantization, tensor and pipeline parallelism, CPU offloading, and early performance results.
Notes on vLLM v.s. DeepSpeed-FastGen
A performance comparison of vLLM and DeepSpeed-FastGen, explaining when Dynamic SplitFuse helps, where vLLM is faster, and how memory allocation, output length, and workload shape affect throughput.
vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
What the original vLLM launch announced: PagedAttention for KV cache management, up to 24x throughput over Hugging Face Transformers, and lower-cost high-throughput LLM serving.