🎬 多模态 / Omni
视觉、语音、视频与 vLLM-Omni 全模态栈。
Finish RL Training Faster: Bitwise-Consistent, High-Performance, Agent-Driven
We make vLLM rollout and the training forward give bitwise identical logprobs on DeepSeek-V4-Flash and Nemotron-3.5-Lightning-30B-A3B (NVFP4), keep 91% of rollout throughput on DeepSeek-V4-Flash, and open-source the agent skill.
vLLM Support for NVIDIA Vera Rubin NVL72: 7.8x Throughput over GB200 NVL72
vLLM now runs on NVIDIA Vera Rubin NVL72 with day-0 model support, Rubin-tuned FlashInfer kernels, and locality-aware MoE, delivering 7.8x the per-GPU throughput of GB200 NVL72 on AgentX.
Taking vLLM Apart: A Practical Guide to Disaggregated Serving
『PD 分离服务』学习路径的核心实操文章:建议在理解 prefill/decode 差异并读过路由层介绍(如 vLLM Router 一文)之后阅读,再延伸到 Qwen3.8-2.4T PD 容量规划与 bidirectional KV transfer 等专题文章。
Watermarking in vLLM
How vLLM implements distribution-preserving Gumbel-max text watermarking with efficient GPU kernels, statistical detection, speculative decoding, and repeated-context safeguards.
Announcing vllm-metal: Concurrent Serving on Apple Silicon
vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M5 prefill acceleration.
Scaling Multi-GPU Video Captioning with PyNvVideoCodec and vLLM
How to leverage NVIDIA Hardware Video Decoders to Achieve Multi-GPU Scaling in Video Captioning and Description tasks.
vime × RL-Kernel × AMD: Bitwise Train–Rollout Consistency on ROCm
vime and RL-Kernel align selected-token logprobs bit for bit across Megatron training and vLLM rollout on AMD Instinct MI300X, with zero mismatches across 200 GRPO steps.
Following the Bottleneck: Optimizing MiniMax M3 on AMD Instinct MI355X
A performance model for LLM serving: inspect local shapes, remove repeated work, verify data movement and dispatch, then follow the queue.
Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin
Tenstorrent accelerators join vLLM as an out-of-tree platform plugin, driven by mesh-architecture choices: phase-based scheduling, single-process data parallelism on Galaxy, on-device sampling with host fallback, and async decode overlap.
MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3
How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.
VeRL-Omni v0.2.0: Faster Diffusion RL and Stable Omni Training
A release focused on higher-throughput diffusion rollout, reusable omni adapters, and broader recipe coverage.
Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni
Distributed Layerwise Offload shards and streams DiT weights across devices, serving a measured 124 GB Cosmos3 model on 64 GB HBM and estimating a path toward 200B+ models.
Announcing Day-0 Support for NVIDIA Nemotron 3.5 Lightning on vLLM
How vLLM serves NVIDIA Nemotron 3.5 Lightning with OpenAI-compatible APIs, speculative decoding, and BF16/NVFP4 checkpoints across NVIDIA GPUs and edge systems.
vLLM Reaches 25K Total TPS/GPU on Qwen3.5
本文是『PD 分离服务』路径末尾三篇生产规模实战中建议最先读的一篇：它是 Qwen3.8 篇的直接前置（后者开篇引用本文结果），也是 GB200/GB300 NVL72 + 混合 GDN 架构 PD serving 的基础认知入口，可再顺读 GLM-5.2 篇对照 SLA 导向的取舍。
Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding
『推测解码』学习路径中的进阶演进篇:先读 speculative decoding 基础与 EAGLE-3 实践(如 2026-07-13 的 EAGLE3 on AMD Instinct 一文),再读本文理解 P-EAGLE、DFlash、DSpark 如何突破自回归起草的串行上限,可作为该主题的前沿收尾综述。
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across NVIDIA and AMD GPUs.
Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving
『PD 分离服务』路径的横向扩展篇:在掌握按阶段拆分的 PD 分离之后阅读,理解按组件(Attention vs FFN)拆分的另一维度;其异步 connector 本就部署在 PD 分离之上,与 hybrid SSM 篇(同一 PD 框架下的模型适配)互为补充。
A Preview of Production-Scale Kimi K3 Support on vLLM
A preview of production-scale Kimi K3 support in vLLM, including KDA-aware prefix caching, fused kernels, optimized MXFP4 MoE, multimodal integration, and initial NVIDIA and AMD paths.
Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router
vLLM Semantic Router is expanding from intelligent routing into a system for building, evaluating, and running Mixture-of-Models.
TML Inkling on vLLM: Day-0 Support with Optimized Performance
vLLM brings day-0 support to TML Inkling, a 1T-parameter multimodal model, with MTP, long-context serving, parallelism, and up to 380 tokens per second per user on NVIDIA GB200 GPUs.
vime + ROCm: End-to-End RL Post-Training on AMD Instinct™ GPUs
Announcing ROCm support for vime, now running end-to-end on AMD Instinct MI355X GPUs with prebuilt container.
Experience and Lessons Learned from Serving Multi-Stage Qwen3-Omni in vLLM-Omni
How vLLM-Omni serves and optimizes Qwen3-Omni with staged Thinker-Talker-Code2Wav execution, batching, CUDA Graphs, async chunk, async output, replicas, hot-path cleanup, and perf validation.
Micro-Agent: Beat Frontier Models with Collaboration inside Model API
How vLLM Semantic Router turns vllm-sr/auto into a bounded micro-agent runtime for Confidence, Ratings, ReMoM, Fusion, Workflows, and benchmark-shaped collaboration.
Engineering TTS Inference in vLLM-Omni
How vLLM-Omni supports and optimizes Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro with staged serving, batching, CUDA Graphs, and model-specific kernels.
MiniMax M3 in vLLM: Day-0 Serving for 1M-Token Multimodal Reasoning
How vLLM serves MiniMax M3 with MiniMax Sparse Attention, multimodal and reasoning parsers, MXFP8 weights, and long-context deployment recipes.
DiffusionGemma: The First Diffusion LLM (dLLM) Natively Supported in vLLM
How vLLM supports DiffusionGemma, the first native diffusion language model in vLLM, using Model Runner V2 state hooks, iterative denoising, bidirectional attention, and reused speculative decoding paths.
Announcing vime: A Simple, Stable, and Efficient RL Framework for LLMs
vime connects slime's training stack with vLLM rollouts to provide a simple, stable, and efficient RL post-training pipeline.
vLLM Semantic Router v0.3 Themis: From Signals to Stateful Production Routing
What vLLM Semantic Router v0.3 Themis adds for production routing: canonical config, inspectable signal-decision-policy flows, safer operations, CLI/dashboard/Kubernetes alignment, and replayable routing behavior.
Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI
What the DeepLearning.AI vLLM course teaches: optimizing, deploying, and benchmarking LLM inference with LLM Compressor quantization, GuideLLM, KV cache sizing, serving, and memory tradeoffs.
Accelerating vLLM-Omni Inference with AutoRound Quantization
How AutoRound integrates with vLLM-Omni to serve W4A16 quantized multimodal, diffusion, image, and video models with smaller checkpoints, preserved quality, Intel XPU acceleration, and NVIDIA GPU support.
Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents
How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cache-aware switch pricing, and replayable traces.
From Text to Multimodal Routing: Hardening Vision Signals in vLLM Semantic Router
How vLLM Semantic Router hardens multimodal routing by turning visual evidence into trustworthy signals, debugging a Rust/Candle vision-encoder parity issue, and validating image signal correctness for production policy.
Speculators v0.5.0: DFlash Support and Online Training
推测解码路径的中间篇:建议先读 Laguna XS.2 + DFlash 案例建立直观认识,再读本文掌握 DFlash 训练机制与 Speculators 训练生态,最后读 P-EAGLE 深入并行起草的架构与内核实现。
Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models
How VeRL-Omni extends verl with vLLM-Omni for reinforcement learning post-training of diffusion and multimodal generative models, including efficient rollouts, reward inference, trainers, hardware support, and recipes.
Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM
How to serve NVIDIA Nemotron 3 Nano Omni with vLLM for multimodal agentic AI, including BF16, FP8, and NVFP4 checkpoints, vision/audio/video inputs, supported GPUs, OpenAI-compatible APIs, and deployment recipes.
vLLM Korea Meetup 2026 Wrap-Up
What the vLLM Korea Meetup 2026 covered: community growth, vLLM V1 updates, production stack adoption, accelerator integration, vllm-playground, and real-world LLM serving.
Announcing Gemma 4 on vLLM: Byte for byte, the most capable open models
How vLLM supports Google's Gemma 4 open models across NVIDIA, AMD, Intel, and TPU backends, with multimodal inputs, agentic workflows, long context, function calling, and deployment recipes.
Model Runner V2: A Modular and Faster Core for vLLM
How Model Runner V2 reworks vLLM's execution core with modular model logic, GPU-native input preparation, stable persistent batching, async-first scheduling, and no API changes.
vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System Brain
What vLLM Semantic Router v0.2 Athena adds: refreshed multilingual and multimodal routing models, ONNX and ROCm acceleration, safety and memory signals, long-context handling, and ClawOS orchestration.
vLLM Triton Attention Backend Deep Dive
A technical walkthrough of the vLLM Triton attention backend, covering performance-portable paged attention kernels, backend selection, autotuning, CUDA graph behavior, benchmarks, and NVIDIA, AMD, and Intel support.
Streaming Requests & Realtime API in vLLM
How vLLM supports streamable inputs and a Realtime WebSocket API for audio, video, robotics, and low-latency applications that need incremental input processing instead of complete prompts.
Building Mixture-of-Models on AMD GPUs with vLLM-SR
How vLLM Semantic Router builds a Mixture-of-Models system on AMD MI300X and MI355X GPUs, routing across specialized models with signals, decisions, safety checks, semantic caching, and live MoM deployment.
Introducing vLLM Playground: A Modern Web Interface for Managing and Interacting with vLLM Servers
How vLLM Playground provides a web UI for starting, configuring, testing, and monitoring vLLM servers across local macOS, Linux GPU or CPU, Kubernetes, and OpenShift environments.
vLLM-Omni Diffusion Cache Acceleration
How vLLM-Omni speeds up diffusion model inference with Cache-DiT and TeaCache, reusing intermediate computations across timesteps to deliver 1.5x to 2x image generation speedups with minimal quality loss.
AMD × vLLM Semantic Router: Building the System Intelligence Together
How AMD and vLLM Semantic Router build GPU-accelerated Mixture-of-Models routing with signals, semantic caching, response storage, PII, jailbreak, and hallucination guardrails.
Encoder Disaggregation for Scalable Multimodal Model Serving
How vLLM EPD separates visual encoders from text prefill and decode, covering LMM serving, GPU resource scaling, multimodal interference, and pipelined execution.
Diving into speculative decoding training support for vLLM with Speculators v0.3.0
『推测解码』路径的第二篇（工程进阶）：承接第一篇对 EAGLE/Medusa 的概念铺垫，深入 Eagle3 draft model 的训练数据构造、训练技术与 vLLM 部署细节，建议在读过入门篇后、读 EAGLE 3.1 之前阅读。
Advancing Low‑Bit Quantization for LLMs: AutoRound x LLM Compressor
How Intel AutoRound integrates with LLM Compressor to produce low-bit quantized checkpoints for vLLM, using tuning-based PTQ, W4A16 and related formats, compressed-tensors compatibility, and lightweight calibration.
Announcing vLLM-Omni: Easy, Fast, and Cheap Omni-Modality Model Serving
What vLLM-Omni adds to the vLLM ecosystem: omni-modality serving for text, image, video, and audio, diffusion and non-autoregressive generation support, disaggregated stages, OpenAI-compatible APIs, and pipelined execution.
Shared Memory IPC Caching: Accelerating Data Transfer in LLM Inference Systems
How shared memory IPC caching in vLLM reduces redundant data transfers for multimodal and multi-process inference, improving prefill throughput and TTFT by sharing large inputs across coordinator and worker processes.
Run Multimodal Reasoning Agents with NVIDIA Nemotron on vLLM
How to serve NVIDIA Nemotron Nano 2 VL with vLLM for multimodal reasoning agents, including video understanding, document intelligence, Efficient Video Sampling, 128K context, and OpenAI-compatible deployment.
Zero-Reload Model Switching with vLLM Sleep Mode
How vLLM Sleep Mode enables fast model switching by hibernating weights to CPU RAM or discarding them while preserving process state, CUDA graphs, allocators, and kernel warmup to avoid full reloads.
vLLM TPU: A New Unified Backend Supporting PyTorch and JAX on TPU
How the redesigned vLLM TPU backend uses tpu-inference, JAX-to-XLA lowering, Torchax, ragged paged attention, and unified PyTorch and JAX support to improve TPU performance and model coverage.
Serving Geospatial, Vision, and Beyond: Enabling Multimodal Output Processing in vLLM
How vLLM expands beyond text generation to serve geospatial, vision, and other non-autoregressive models with pooling-model support, TerraTorch integration, raw tensor handling, and flexible IO processors.
GLM-4.5 Meets vLLM: Built for Intelligent Agents
How to run GLM-4.5 and GLM-4.5V with vLLM for intelligent agents, including hybrid reasoning modes, FP8 and BF16 serving, multimodal support, and NVIDIA Blackwell and Hopper deployment.
Transformers modeling backend integration in vLLM
How vLLM integrates the Hugging Face Transformers modeling backend to serve more model architectures efficiently, including text and vision-language models through model_impl=\"transformers\".
Llama 4 in vLLM
How vLLM serves Meta Llama 4 Scout and Maverick multimodal MoE models with long-context support, tensor parallel deployment, H100 and H200 guidance, FP8 variants, and performance tips.
PTPC-FP8: Boosting vLLM Performance on AMD ROCm
How PTPC-FP8 quantization improves vLLM performance on AMD ROCm by combining per-token activation scaling and per-channel weight scaling for near-BF16 accuracy with FP8 speed.
Introducing AIBrix: A Scalable, Cost-Effective Control Plane for vLLM
What AIBrix adds as a Kubernetes control plane for vLLM: LoRA management, LLM gateway routing, autoscaling, unified runtime, distributed inference, distributed KV cache, heterogeneous serving, and GPU failure detection.
vLLM V1: A Major Upgrade to vLLM's Core Architecture
What changed in vLLM V1: a re-architected engine with a simpler scheduler, near-zero-overhead prefix caching, cleaner tensor parallelism, multiprocessing API server, and default optimizations for higher-throughput serving.
Introducing vLLM Inference Provider in Llama Stack
How Llama Stack integrates vLLM as an inference provider through remote and inline providers, enabling OpenAI-compatible vLLM serving for local and Kubernetes generative AI application deployments.
vLLM 2024 Retrospective and 2025 Vision
A vLLM 2024 retrospective and 2025 roadmap covering community growth, model and hardware support, production adoption, office hours, ecosystem partnerships, and the path toward universal open-source serving.