全部文章

在线拟合的逻辑回归从草稿 logits 估计接受概率,让无 confidence head 的草稿模型免训接入 AV。

push 模式让 prefill 直接把 KV 写入 decode 预注册显存，TTFT 最高降 3 倍

We make vLLM rollout and the training forward give bitwise identical logprobs on DeepSeek-V4-Flash and Nemotron-3.5-Lightning-30B-A3B (NVFP4), keep 91% of rollo

双向 KV 传输让 prefill 复用 decode 保留的历史 KV，多轮场景 TTFT 最高降 3.3 倍

vLLM now runs on NVIDIA Vera Rubin NVL72 with day-0 model support, Rubin-tuned FlashInfer kernels, and locality-aware MoE, delivering 7.8x the per-GPU throughpu

Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounde

vLLM 分离式服务实战:拆分 prefill/decode 与 CPU 前端,用 KV cache 传输换取 goodput 与稳定尾延迟。

How vLLM implements distribution-preserving Gumbel-max text watermarking with efficient GPU kernels, statistical detection, speculative decoding, and repeated-c

vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M

GB300 上 Qwen3.8-2.4T PD serving Pareto 前沿：5K TPS/GPU、180 tok/s/user。

How to leverage NVIDIA Hardware Video Decoders to Achieve Multi-GPU Scaling in Video Captioning and Description tasks.

Speculators+Mooncake 多节点训练 Kimi K3 DSpark,单流提速至 ~435 tok/s

Novita AI has open-sourced Chord, a high-performance W4A16 MoE CUDA kernel for Kimi K2.x serving shapes, with a Humming-compatible indexed path and grouped SM90

vime and RL-Kernel align selected-token logprobs bit for bit across Megatron training and vLLM rollout on AMD Instinct MI300X, with zero mismatches across 200 G

Kimi K3 serving optimizations across scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU ke

A performance model for LLM serving: inspect local shapes, remove repeated work, verify data movement and dispatch, then follow the queue.

A host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers — reducing recomputation and increasing serving c

vLLM integrates HiSparse as a pressure-driven memory tier that composes with the Hybrid Memory Allocator and KV offloading, letting GLM 5.3 requests keep decodi

How vLLM optimizes KV cache management, parallelism, scheduling, and P/D disaggregation for agentic workloads, validated on SemiAnalysis AgentX with up to 130K

Tenstorrent accelerators join vLLM as an out-of-tree platform plugin, driven by mesh-architecture choices: phase-based scheduling, single-process data paralleli

How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.

vLLM 在 AMD GPU 上的推测解码实战大全:五种 draft 方法、配置调优与基准数据

We implement a native sharded weight transfer engine in vLLM utilizing Ray Direct Transport (RDT), achieving weight transfer for the Kimi K2 model in BF16 on 48

IsoExec unifies numerical execution across SkyRL's vLLM and Megatron runtimes, reducing the average rollout-versus-training logprob difference below 1e-6 on Qwe

A release focused on higher-throughput diffusion rollout, reusable omni adapters, and broader recipe coverage.

Distributed Layerwise Offload shards and streams DiT weights across devices, serving a measured 124 GB Cosmos3 model on 64 GB HBM and estimating a path toward 2

DSpark 用置信度调度草稿验证预算，vLLM 一套配置从并发 1 到 256 守住吞吐/延迟前沿。

Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quanti

How vLLM serves NVIDIA Nemotron 3.5 Lightning with OpenAI-compatible APIs, speculative decoding, and BF16/NVFP4 checkpoints across NVIDIA GPUs and edge systems.

Decode Context Parallelism (DCP) in vLLM shards KV cache across GPUs by sequence dimension, enabling 3× higher throughput on long-context agentic workloads comp

GB200 NVL72 上分离服务 Qwen3.5-397B，vLLM 每 GPU 吞吐达 25K TPS。

An overview of Arm CPU enablement and inference performance optimizations in vLLM.

P-EAGLE、DFlash、DSpark 三种并行起草算法获 vLLM 支持:单次前向起草整个候选 token 块

vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N

24 卡 B300 以 4P+1D 分离部署 GLM-5.2，TPOT 从 40ms 降至 17ms 满足生产 SLA。

AFD 插件把 MoE 的 Attention/FFN 拆为独立扩展的服务,经 connector 接入 vLLM。

A preview of production-scale Kimi K3 support in vLLM, including KDA-aware prefix caching, fused kernels, optimized MXFP4 MoE, multimodal integration, and initi

vLLM Semantic Router is expanding from intelligent routing into a system for building, evaluating, and running Mixture-of-Models.

How vLLM maintains production quality with extensive CI across diverse accelerators, nightly performance benchmark and accuracy evaluation, and a two-week relea

vLLM brings day-0 support to TML Inkling, a 1T-parameter multimodal model, with MTP, long-context serving, parallelism, and up to 380 tokens per second per user

vLLM prefill 搭配 TileRT 专用 decode:零修改 vLLM、按流量路由,主打每用户低延迟。

AMD Instinct 上 EAGLE3:Quark MXFP4 量化 + vLLM 训练与推理,提速至 2.00x

Announcing ROCm support for vime, now running end-to-end on AMD Instinct MI355X GPUs with prebuilt container.

How HPC-Ops integrates Hopper-optimized attention and FP8 MoE backends into vLLM for Tencent Hunyuan Hy3, improving mixed-length decode, MoE latency, TTFT, and

How vLLM-Omni serves and optimizes Qwen3-Omni with staged Thinker-Talker-Code2Wav execution, batching, CUDA Graphs, async chunk, async output, replicas, hot-pat

How vLLM Semantic Router turns vllm-sr/auto into a bounded micro-agent runtime for Confidence, Ratings, ReMoM, Fusion, Workflows, and benchmark-shaped collabora

How vLLM-Omni supports and optimizes Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro with staged serving, batching, CUDA Graphs, and model-specific k

How vLLM Semantic Router Fusion runs a panel of models, uses a judge to analyze agreement and gaps, and synthesizes one answer while preserving routing policy,

How vLLM serves MiniMax M3 with MiniMax Sparse Attention, multimodal and reasoning parsers, MXFP8 weights, and long-context deployment recipes.

How vLLM supports DiffusionGemma, the first native diffusion language model in vLLM, using Model Runner V2 state hooks, iterative denoising, bidirectional atten

vime connects slime's training stack with vLLM rollouts to provide a simple, stable, and efficient RL post-training pipeline.

What vLLM Semantic Router v0.3 Themis adds for production routing: canonical config, inspectable signal-decision-policy flows, safer operations, CLI/dashboard/K

How to serve NVIDIA Nemotron 3 Ultra with vLLM for long-running agentic reasoning, including BF16 and NVFP4 checkpoints, supported GPU configurations, OpenAI-co

What the DeepLearning.AI vLLM course teaches: optimizing, deploying, and benchmarking LLM inference with LLM Compressor quantization, GuideLLM, KV cache sizing,

How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cach

How AutoRound integrates with vLLM-Omni to serve W4A16 quantized multimodal, diffusion, image, and video models with smaller checkpoints, preserved quality, Int

How to run vLLM on NVIDIA DGX Spark and GB10 systems, including unified memory behavior, NVFP4 Nemotron-3-Super serving, Docker deployment, Prometheus metrics,

Laguna XS.2 配 DFlash 推测解码与多精度量化,token 生成快 2-3× 且质量无损。

How vLLM native RL APIs standardize weight syncing and asynchronous RL serving with NCCL and CUDA IPC transfer backends, pause mode, and fixes for fragile DPEP

Speculators v0.5.0 带来 DFlash 训练支持与统一在线/离线训练,并改用 vLLM 原生提取。

How vLLM Semantic Router hardens multimodal routing by turning visual evidence into trustworthy signals, debugging a Rust/Candle vision-encoder parity issue, an

EAGLE 3.1 修复 attention drift，长上下文 acceptance length 最高达 2×

How PegaFlow integrates with vLLM as an external KV cache service, using a Rust daemon, CUDA IPC, RDMA, SSD caching, and the external KV connector to improve st

How Elastic Expert Parallelism lets vLLM scale Mixture-of-Experts serving up or down at runtime by changing data-parallel workers, redistributing experts, and c

How VeRL-Omni extends verl with vLLM-Omni for reinforcement learning post-training of diffusion and multimodal generative models, including efficient rollouts,

A vLLM study comparing TurboQuant KV-cache quantization with BF16 and FP8 across long-context and reasoning workloads, showing where 4-bit variants help, where

How vLLM achieved leading Artificial Analysis results for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B using open-source kernel fusion, speculative decoding,

How vLLM integrates Mooncake Store as a distributed KV cache for agentic workloads, reusing shared prefixes across turns and instances to improve throughput, TT

How to serve NVIDIA Nemotron 3 Nano Omni with vLLM for multimodal agentic AI, including BF16, FP8, and NVFP4 checkpoints, vision/audio/video inputs, supported G

A first-principles walkthrough of DeepSeek V4's long-context attention, and how we implemented it in vLLM.

What vLLM FP8 KV-cache validation found across Hopper and Blackwell, covering attention quantization, Flash Attention 3 fixes, memory savings, decode speedups,

vLLM 用双描述符视图与 DS 布局把 NIXL PD 分离扩展到 hybrid SSM,零开销传 Mamba 状态。

What the vLLM Korea Meetup 2026 covered: community growth, vLLM V1 updates, production stack adoption, accelerator integration, vllm-playground, and real-world

How single-node prefill/decode disaggregation in vLLM uses AMD MORI-IO on an 8-GPU MI300X node to separate prefill and decode, transfer KV cache efficiently, st

How vLLM supports Google's Gemma 4 open models across NVIDIA, AMD, Intel, and TPU backends, with multimodal inputs, agentic workflows, long context, function ca

How vLLM extracts verifier hidden states through dummy draft models and KV Connector APIs for speculative decoding, enabling offline and online Speculators trai

How Model Runner V2 reworks vLLM's execution core with modular model logic, GPU-native input preparation, stable persistent batching, async-first scheduling, an

P-EAGLE 单次前向并行生成全部 draft token,消除自回归起草瓶颈,最高提速 1.69×。

How to serve NVIDIA Nemotron 3 Super with vLLM for multi-agent AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, Thinking Budget, MTP, supported

What vLLM Semantic Router v0.2 Athena adds: refreshed multilingual and multimodal routing models, ONNX and ROCm acceleration, safety and memory signals, long-co

A technical walkthrough of the vLLM Triton attention backend, covering performance-portable paged attention kernels, backend selection, autotuning, CUDA graph b

How vLLM orchestrates high-performance inference on AMD ROCm with multiple attention backends, workload-aware prefill, extend, and decode routing, AITER primiti

How vLLM serves many fine-tuned MoE and dense models with Multi-LoRA, including fused MoE LoRA kernels, Triton compiler fixes, Split-K and CTA swizzling optimiz

What DeepSeek-V3.2 and DeepSeek-R1 benchmark results show on NVIDIA GB300 with vLLM, covering NVFP4 quantization, TP and EP deployment, throughput, and reproduc

How vLLM improves WideEP and large-scale DeepSeek-style MoE serving on NVIDIA GB200 with NVFP4 and FP8 kernels, fusion, prefill/decode disaggregation, weight of

How vLLM and NVIDIA optimized GPT-OSS on Blackwell with FlashInfer, torch.compile fusion, FP8 KV cache, async scheduling, stream interval tuning, and deployment

How vLLM supports streamable inputs and a Realtime WebSocket API for audio, video, robotics, and low-latency applications that need incremental input processing

How vLLM Semantic Router builds a Mixture-of-Models system on AMD MI300X and MI355X GPUs, routing across specialized models with signals, decisions, safety chec

How vLLM's asynchronous KV offloading connector stores KV cache in CPU memory to reduce recomputation, improve throughput under memory pressure, and support plu

What vLLM Semantic Router v0.1 Iris introduces: signal-decision plugin architecture, model selection, safety filtering, semantic caching, hallucination detectio

How vLLM Playground provides a web UI for starting, configuring, testing, and monitoring vLLM servers across local macOS, Linux GPU or CPU, Kubernetes, and Open

What changed on the new vllm.ai website for vLLM users: installation guidance, events pages, Slack and X community channels, vLLM Daily updates, and a clearer p

How vLLM-Omni speeds up diffusion model inference with Cache-DiT and TeaCache, reusing intermediate computations across timesteps to deliver 1.5x to 2x image ge

How vLLM reaches 2.2k tokens per second per H200 for DeepSeek-style MoE serving with Wide-EP, async scheduling, dual-batch overlap, disaggregated serving, CUDA

How AMD and vLLM Semantic Router build GPU-accelerated Mixture-of-Models routing with signals, semantic caching, response storage, PII, jailbreak, and hallucina

How to serve NVIDIA Nemotron 3 Nano with vLLM for efficient agentic AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, hybrid MoE architecture, T

How vLLM EPD separates visual encoders from text prefill and decode, covering LMM serving, GPU resource scaling, multimodal interference, and pipelined executio

How HaluGate adds token-level hallucination detection to vLLM Semantic Router by verifying assistant claims against tool outputs and grounding context in real t

Speculators v0.3.0 打通 Eagle3 draft model 端到端训练与 vLLM 一键部署

vLLM Router:Rust 实现的状态感知负载均衡器,原生编排 prefill/decode 分离,大规模下性能领先 llm-d 与 K8s 方案。

How Intel AutoRound integrates with LLM Compressor to produce low-bit quantized checkpoints for vLLM, using tuning-based PTQ, W4A16 and related formats, compres

How vLLM developers debug hanging and complex CUDA kernels by triggering GPU core dumps, identifying stuck kernels, and mapping failures back to source code lin

What vLLM-Omni adds to the vLLM ecosystem: omni-modality serving for text, image, video, and audio, diffusion and non-autoregressive generation support, disaggr

How Ray symmetric-run simplifies multi-node vLLM serving by launching the same entrypoint on every Ray cluster node, matching HPC and parallel SSH workflows for

How the vLLM plugin system helps teams customize scheduling, KV-cache behavior, hardware integrations, and model execution without long-lived forks, monkey patc

How Docker Model Runner integrates vLLM as an inference backend, letting developers run safetensors models with high-throughput serving, PagedAttention, streami

How vLLM Semantic Router replaces fixed domain classification with signal-decision architecture, combining multi-dimensional signals, AND/OR decision logic, mod

How shared memory IPC caching in vLLM reduces redundant data transfers for multimodal and multi-process inference, improving prefill throughput and TTFT by shar

How vLLM serves LLMs on Intel Arc Pro B-Series GPUs with MoE optimizations, persistent kernels, multi-GPU scaling, LoRA, speculative decoding, structured output

How vLLM and TorchTitan demonstrate bitwise consistent on-policy RL by matching training and inference numerics, using batch-invariant kernels to reduce train-i

How to serve NVIDIA Nemotron Nano 2 VL with vLLM for multimodal reasoning agents, including video understanding, document intelligence, Efficient Video Sampling

How vLLM debugged Kimi K2 tool-calling accuracy, covering chat-template compatibility, add_generation_prompt handling, schema validation failures, benchmark fix

How vLLM Semantic Router refactors its Rust classification layer with modular model support, Qwen3-Embedding, EmbeddingGemma, LoRA-based multi-task classificati

How vLLM Sleep Mode enables fast model switching by hibernating weights to CPU RAM or discarding them while preserving process state, CUDA graphs, allocators, a

How vLLM serves NVIDIA Nemotron Nano 2 for agentic reasoning, including hybrid Transformer-Mamba architecture, thinking budget control, open weights and data, t

How vLLM's OpenAI-compatible API can return prompt and response token IDs to prevent retokenization drift in agent reinforcement learning, preserving exact samp

How the redesigned vLLM TPU backend uses tpu-inference, JAX-to-XLA lowering, Torchax, ragged paged attention, and unified PyTorch and JAX support to improve TPU

How vLLM and NVIDIA optimize Blackwell inference for SemiAnalysis InferenceMAX, improving gpt-oss 120B and Llama 3.3 70B throughput with FP4 kernels, scheduling

How vLLM supports DeepSeek-V3.2-Exp with DeepSeek Sparse Attention, lightning indexer caches, separate prefill and decode paths, FlashMLA sparse attention, Deep

What the first vLLM Korea meetup covered: community adoption, llm-d, TPU integration, contribution workflows, hardware plugin architecture, Rebellions NPU work,

How vLLM supports Qwen3-Next with hybrid attention, Gated DeltaNet, full attention, high-sparsity MoE, multi-token prediction, hybrid KV cache management, Trito

How vLLM Semantic Router routes requests by intent, covering semantic classification, smart reasoning-path selection, Rust and Candle execution, and Kubernetes

How vLLM's inference engine works, covering PagedAttention, continuous batching, prefix caching, speculative decoding, multi-GPU serving, scheduling, and benchm

How vLLM expands beyond text generation to serve geospatial, vision, and other non-autoregressive models with pooling-model support, TerraTorch integration, raw

How torch.compile works inside vLLM, including TorchDynamo graph capture, TorchInductor code generation, custom compiler passes, graph breaks, model compilation

How to run GLM-4.5 and GLM-4.5V with vLLM for intelligent agents, including hybrid reasoning modes, FP8 and BF16 serving, multimodal support, and NVIDIA Blackwe

How to debug vLLM CUDA illegal memory access errors with CUDA core dumps, environment variables, cuda-gdb, and GPU state inspection when Python stack traces or

How vLLM supports gpt-oss 20B and 120B on NVIDIA Blackwell, Hopper, and AMD GPUs, with MXFP4 MoE kernels, efficient attention, hybrid KV cache allocation, and b

How vLLM serves MiniMax-M1's hybrid MoE architecture for long-context inference, covering model deployment, memory management, batched serving, backend optimiza

How vLLM hardware plugins decouple backend integrations from core vLLM, using Platform, Executor, Worker, ModelRunner, AttentionBackend, and Communicator hooks

How OpenRLHF uses vLLM, Ray, ZeRO-3, AutoTP, Ray placement groups, and weight synchronization to accelerate PPO and RLHF sample generation for reasoning models

How vLLM integrates the Hugging Face Transformers modeling backend to serve more model architectures efficiently, including text and vision-language models thro

How vLLM serves Meta Llama 4 Scout and Maverick multimodal MoE models with long-context support, tensor parallel deployment, H100 and H200 guidance, FP8 variant

How PTPC-FP8 quantization improves vLLM performance on AMD ROCm by combining per-token activation scaling and per-channel weight scaling for near-BF16 accuracy

What AIBrix adds as a Kubernetes control plane for vLLM: LoRA management, LLM gateway routing, autoscaling, unified runtime, distributed inference, distributed

A guide to distributed inference in vLLM, covering tensor parallelism, pipeline parallelism, multi-GPU and multi-node serving, KV cache challenges, speculative

How Llama Stack integrates vLLM as an inference provider through remote and inline providers, enabling OpenAI-compatible vLLM serving for local and Kubernetes g

What changed in vLLM V1: a re-architected engine with a simpler scheduler, near-zero-overhead prefix caching, cleaner tensor parallelism, multiprocessing API se

What vLLM production-stack adds for Kubernetes serving: prefix-aware routing, LMCache-backed KV cache sharing, autoscaling, observability, fault tolerance, and

How structured decoding works in vLLM, covering JSON outputs, grammar-guided generation, outlines, XGrammar, TPOT improvements, constrained decoding, and agenti

How to install and develop vLLM using stable releases, nightly wheels, uv, source builds, Python and C++/CUDA workflows, and version tracking for production dep

A vLLM 2024 retrospective and 2025 roadmap covering community growth, model and hardware support, production adoption, office hours, ecosystem partnerships, and

Best practices for serving LLMs with vLLM on AMD MI300X, covering ROCm setup, Llama 3.1 70B and 405B benchmarks, chunked prefill, multi-step scheduling, prefix

vLLM 推测解码入门：小模型提议、大模型并行验证，低 QPS 下最高 2.8x 加速

What changed in vLLM v0.6.0 to improve throughput and latency, including API server isolation, reduced CPU overhead, multi-step scheduling, async execution, and

vLLM's open governance and performance roadmap, covering LF AI and Data incubation, public benchmarks, optimized kernels, async scheduling, API frontend overhea

How vLLM supports Meta Llama 3.1 models, including 128K context, Llama 3.1 405B serving, chunked prefill, FP8 quantization, tensor and pipeline parallelism, CPU

A performance comparison of vLLM and DeepSpeed-FastGen, explaining when Dynamic SplitFuse helps, where vLLM is faster, and how memory allocation, output length,

What the original vLLM launch announced: PagedAttention for KV cache management, up to 24x throughput over Hugging Face Transformers, and lower-cost high-throug