要点速览
- vLLM Router 是从 SGLang model gateway fork 而来的 Rust 负载均衡器,专为大规模 vLLM 生产部署设计,可运行在 K8s 或裸金属 GPU 集群。
- 提供 Consistent Hashing、Power of Two、Round Robin、Random 等策略,把同会话请求粘到持有其 KV cache 的 worker 副本,最大化缓存复用、降低延迟。
- 原生承担 P/D 分离编排:把新请求路由到 prefill worker 组,完成后再把请求状态转给 decode worker,支持 NIXL 与 NCCL(带 ZMQ discovery)两种后端。
- 内置企业级能力:K8s 服务发现(label selector 自动发现 worker pod)、指数退避+抖动的重试、熔断器与 Prometheus /metrics 可观测性。
- 基准测试:Llama 3.1 8B(8+8 pods)下吞吐比 llm-d 高 25%、比 K8s 原生高 100%;DeepSeek V3 下 TTFT 比 llm-d 与 K8s 原生快 2000 ms。
解决什么问题
大规模生产 vLLM 部署需要把请求分发到一队模型副本,但标准负载均衡器不理解 LLM 推理的有状态性(例如 KV cache),也无法管理 prefill/decode 分离这类复杂服务模式。缺少一个高性能、状态感知且具备生产级弹性与可观测性的路由层,是 vLLM 集群规模化的关键缺口。
核心思路
vLLM Router 用 Rust 编写以保证极低自身开销,从 SGLang model gateway fork 后针对 vLLM 修改和简化而来。一方面提供多种智能负载均衡策略,其中一致哈希让相同 routing key(会话/用户 ID)的请求粘在同一 worker 副本,以最大化 KV cache 复用;另一方面原生承担 P/D 分离的编排职责:先把请求路由到 prefill 组,prefill 完成后把请求状态转给 decode 组,并支持 NIXL 与 NCCL(ZMQ discovery)两种分离后端。配套 K8s 服务发现、重试、熔断与 Prometheus 指标,最后用两组基准证明其在规模下的性能优势。
关键要点
- 为什么需要专用 router — 标准负载均衡器把 vLLM worker 当作无状态副本,既不知道哪个 worker 已持有某会话的 KV cache,也无法处理 P/D 分离的两阶段工作流。vLLM Router 用 Rust 实现以压低自身开销,作为客户端与 vLLM worker 集群之间的智能、状态感知负载均衡器,支持 Kubernetes 与裸金属 GPU 集群两种环境。它源自 SGLang model gateway 的 fork,经修改和简化适配 vLLM,未来可能并入 vLLM 主仓库。
- 智能负载均衡策略与 KV cache 亲和 — 对会话型负载而言,把同一用户的后续请求路由到已持有其 KV cache 的 worker 是降低延迟的关键。Consistent Hashing 是其中最重要的策略:相同 routing key(如 session ID 或 user ID)的请求保持粘性,稳定落在同一副本上,最大化 KV cache 复用;Power of Two 是低开销的随机选择策略,负载分布表现优秀;Round Robin 与 Random 则面向无状态的普通分发。
- 原生 P/D 分离编排 — 在 P/D 分离架构中,计算密集的 prefill 与访存密集的 decode 由两组专门的 worker 承担,router 是这套架构的编排层。它先把新请求智能路由到 prefill worker 组;prefill 完成后,再把请求状态转给合适的 decode worker 做生成。同时支持 NIXL 与 NCCL(配合 ZMQ discovery)两种分离后端的服务发现与路由,这使它区别于把所有 pod 当作相同副本的通用负载均衡器。
- 企业级弹性与可观测性 — Kubernetes 模式下,router 可通过 label selector 自动发现、监控并路由到 vLLM worker pod。故障容忍方面,提供可配置的重试逻辑(指数退避加抖动)与熔断器:worker 未通过健康检查会被立即移出路由池并重试请求,防止级联故障。可观测性方面,内置 Prometheus 端点(/metrics)导出请求量、延迟、错误率与各 worker 健康状况指标,提供对整个服务集群的完整可见性。
- 基准结果:规模下的吞吐与 TTFT 优势 — 与 llm-d(默认 queue-aware 负载均衡)和 K8s 原生负载均衡(基本 round-robin,不感知 P/D 状态)对比:Llama 3.1 8B(8 个 prefill pod + 8 个 decode pod)下,router 的 Req/S 吞吐比 llm-d 高 25%、比 K8s 原生高 100%,TTFT 与 K8s 原生接近而比 llm-d 快约 1200 ms;DeepSeek V3(1 个 TP8 prefill pod + 1 个 TP8 decode pod)下,吞吐与 llm-d 接近、比 K8s 原生高 100%,TTFT 比 llm-d 与 K8s 原生都快 2000 ms。基准还排除了 vLLM 内置 DP/EP coordinator(vLLM 官方推荐的外部负载均衡方案之一):其吞吐仅为其他方案的 1/8,源于一个已知性能问题。
- 定位与演进路线 — router 把服务架构从一组独立实例转变为单一、统一且有弹性的集群,是生产规模运行 vLLM 的关键组件。它 fork 自 SGLang model gateway,官方预期会进一步分化:一方面探索把 router 并入 vLLM 主仓库,另一方面也可能与 SGLang gateway 开发者合作统一大规模部署的网关功能。
关键数据
| 指标 | 数值 | 条件 |
|---|---|---|
| 吞吐提升(Llama 3.1 8B) | 比 llm-d 高 25%,比 K8s 原生负载均衡高 100% | Llama 3.1 8B,8 个 prefill pod + 8 个 decode pod 的分离部署 |
| TTFT 优势(Llama 3.1 8B) | 比 llm-d 快 1200 ms,与 K8s 原生负载均衡接近 | Llama 3.1 8B,8 个 prefill pod + 8 个 decode pod 的分离部署 |
| 吞吐(DeepSeek V3) | 与 llm-d 接近,比 K8s 原生负载均衡高 100% | DeepSeek V3,1 个 prefill pod(TP8)+ 1 个 decode pod(TP8) |
| TTFT 优势(DeepSeek V3) | 比 llm-d 与 K8s 原生负载均衡都快 2000 ms | DeepSeek V3,1 个 prefill pod(TP8)+ 1 个 decode pod(TP8) |
| 被排除方案的吞吐(vLLM 内置 DP/EP coordinator) | 仅为其他方案的 1/8 | 基准中因已知性能问题(github issue #24461)被排除,该方案本是 vLLM 集群推荐的外部负载均衡选项 |
“vLLM Router (blue line) Req/S throughput is 25% higher than llm-d (purple line) and 100% higher than K8s-native load balancer (orange line).”
“vLLM Router’s TTFT is close to K8s-native load balancer and 1200 ms faster than llm-d.”
“vLLM Router (blue line) Req/S throughput is close to llm-d (purple line) and 100% higher than K8s-native load balancer (orange line).”
前置知识
负载均衡基础:round robin、随机、一致性哈希等策略 · KV cache 与会话亲和的概念:为何同会话请求落在同一副本能复用缓存 · prefill/decode 分离(P/D disaggregation)的基本架构与两阶段工作流 · Kubernetes 服务发现与 label selector 机制 · 可观测性与可靠性模式:Prometheus 指标、重试(指数退避)、熔断器
术语表
vLLM Router: vLLM 团队推出的 Rust 负载均衡器,状态感知并原生支持 prefill/decode 分离编排,定位为大规模生产部署的路由层。Consistent Hashing: 一致性哈希:相同 routing key 的请求稳定映射到同一 worker 副本,用于最大化 KV cache 复用。Power of Two (PoT): 低开销的随机选择负载均衡策略,提供优秀的负载分布。Prefill/Decode disaggregation: 把计算密集的 prefill 与访存密集的 decode 交给两组专门 worker 处理的分离式服务架构。NIXL: P/D 分离的后端之一,vLLM 原生 remote prefill/decode 与 KV-transfer 所用的传输组件。NCCL with ZMQ discovery: 另一种 P/D 分离后端:NCCL 通信配合 ZMQ 完成服务发现。Circuit breaker: 熔断器:健康检查失败的 worker 被立即移出路由池并触发请求重试,防止级联故障。Kubernetes Service Discovery: router 的 K8s 原生模式:通过 label selector 自动发现、监控并路由到 vLLM worker pod。Prometheus /metrics: router 内置的指标导出端点,输出请求量、延迟、错误率与各 worker 健康状况。llm-d: Kubernetes 原生的 LLM 路由/推理框架,基准中作为对比对象,采用默认 queue-aware 负载均衡。TP (Tensor Parallelism): 张量并行:DeepSeek V3 基准中每个 prefill/decode pod 跨 8 卡(TP8)部署。
vLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale Serving
Efficiently managing request distribution across a fleet of model replicas is a critical requirement for large-scale, production vLLM deployments. Standard load balancers often fall short as they lack awareness of the stateful nature of LLM inference (e.g., KV cache) and cannot manage complex serving patterns like prefill/decode disaggregation.
To address this, we are introducing the vLLM Router (Github repo), a high-performance, lightweight load balancer engineered specifically for vLLM. Built in Rust for minimal overhead, the router acts as an intelligent, state-aware load balancer that sits between clients and a fleet of vLLM workers, either in a K8s or a bare metal GPU cluster.
The vLLM Router is derived from a fork of the SGLang model gateway, modified and simplified to work with vLLM. Further divergence is anticipated as we explore merging this router into the vLLM main repo. On the other hand, the gateway functionalities for large-scale deployment may be unified in collaboration with SGLang model gateway developers.
Core Architecture and Capabilities
The vllm-router is designed to solve two primary challenges in large-scale serving: intelligent load balancing and support for prefill/decode disaggregation.
1. Intelligent Load Balancing Strategies
Unlike a simple round-robin, the vLLM Router provides multiple, sophisticated load balancing algorithms to optimize for performance and stateful affinity. For conversational workloads, routing subsequent requests from the same user to the same worker that holds their KV cache is critical for minimizing latency.
The router supports several policies to this end:
- Consistent Hashing: This is the key policy for maximizing performance. It ensures that requests with the same routing key (e.g., a session ID or user ID) are "sticky" and consistently routed to the same worker replica, maximizing KV cache reuse.
- Power of Two (PoT): A low-overhead random-choice policy that provides excellent load distribution.
- Round Robin & Random: Standard policies for stateless load distribution.
2. Native Support for Prefill/Decode Disaggregation
The router is designed as the orchestration layer for vLLM's most advanced serving architecture: prefill/decode (P/D) disaggregation.
In this architecture, the compute-intensive prefill step and the memory-intensive decode step are handled by separate, specialized worker groups. The vLLM Router manages this complex workflow:
- It intelligently routes new requests to the prefill worker group.
- Upon completion, it directs the request state to the appropriate decode worker for token generation.
- It supports discovery and routing for both NIXL and NCCL-based (with ZMQ discovery) disaggregation backends.
Enterprise-Grade Resiliency and Observability
The vllm-router is built with production-grade features for maintaining high availability in large-scale environments.
- Kubernetes Service Discovery: The router can operate in a Kubernetes-native mode, automatically discovering, monitoring, and routing to vLLM worker pods using label selectors.
- Fault Tolerance: It includes configurable retry logic (with exponential backoff and jitter) and circuit breakers. If a worker fails health checks, the router immediately removes it from the routing pool and will retry requests, preventing cascading failures.
- Observability: A built-in Prometheus endpoint (
/metrics) exports detailed metrics on request volume, latency, error rates, and the health of individual workers, providing complete visibility into the serving fleet.
Benchmark Analysis: The Most Performant Choice at Scale
We benchmarked the new vLLM Router against two widely used alternatives:
- llm-d: A Kubernetes-native routing framework that utilizes default queue-aware load balancing.
- vLLM-native: The standard K8s native load balancer, which employs a basic round-robin strategy. Crucially, this option is not aware of Prefill/Decode states, treating all pods as identical vLLM replicas.
Note on Exclusion: We excluded the vLLM built-in DP/EP coordinator—the recommended External Load Balancing solution for vLLM clusters—from the benchmark. Its throughput was only 1/8 of the others due to a known performance issue.
Llama 3.1 8B with 8 Prefill pods and 8 Decode pods
- vLLM Router (blue line) Req/S throughput is 25% higher than llm-d (purple line) and 100% higher than K8s-native load balancer (orange line).
- vLLM Router’s TTFT is close to K8s-native load balancer and 1200 ms faster than llm-d.
Deepseek V3 with 1 Prefill pod (TP8) and 1 Decode pod (TP8)
- vLLM Router (blue line) Req/S throughput is close to llm-d (purple line) and 100% higher than K8s-native load balancer (orange line).
- vLLM Router’s TTFT is 2000 ms faster than llm-d and K8s-native.
Summary
The vLLM Router is an essential component for operating vLLM at production scale. It transitions the serving architecture from a collection of individual instances to a single, unified, and resilient fleet. By providing intelligent load balancing and native support for prefill/decode disaggregation, it unlocks new levels of performance and operational efficiency.
Acknowledgements
- Thanks to Phi and the AWS team for providing technical support and the test clusters.
- Special thanks to Naman Lalit for driving the comprehensive performance and correctness benchmarking efforts.
- We also acknowledge the SGLang Model Gateway team. By forking their established API implementation and service framework, we were able to significantly accelerate our design and implementation process while maintaining alignment with open standards.
- Finally, we thank Tyler Michael Smith and Robert Shaw for sharing llm-d expertise and receipts which unblocked performance optimizations and benchmarks.