Basic
Minimal multi-node inference on LeaderWorkerSet with vLLM and SGLang.
LeaderWorkerSet
Multi-node inference with LeaderWorkerSet, including autoscaling and topology-aware placement.
These guides deploy a distributed, multi-node inference service with
LeaderWorkerSet, spreading tensor and pipeline parallelism across the leader and
worker pods. Each guide isolates one feature and ships both a vllm.yaml and a
sglang.yaml.
- Basic: minimal multi-node deployment (no Ray).
- Autoscaling: scale replica groups with a HorizontalPodAutoscaler.
- Topology-aware scheduling: pin each group to one
topology domain with
exclusive-topology.
Autoscaling
Scale LeaderWorkerSet replica groups with a HorizontalPodAutoscaler.
Topology-aware scheduling
Pin each replica group to a single topology domain, with native placement or Kueue.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.