vLLM
An example of using vLLM with LWS
Examples
This section contains examples of using LWS with or without specific inference runtime.
Use the pages in this section for runtime-specific LeaderWorkerSet deployments (vLLM, SGLang, TensorRT-LLM, and others). For multi-role disaggregated inference (prefill/decode and related patterns), start with the DisaggregatedSet examples.
TensorRT-LLM
An example of using TensorRT-LLM with LWS
llama.cpp
An example of using llama.cpp with LWS
SGLang
An example of using SGLang with LWS
Horizontal Pod Autoscaler (HPA)
An example of using Horizontal Pod Autoscaler with LeaderWorkerSet
Topology Aware Scheduling with Kueue
An example on using topology aware scheduling with LWS and Kueue, using vLLM
DisaggregatedSet Examples
Minimal working examples for DisaggregatedSet — from a simple 2-role nginx setup to a 3-role LLM inference pattern.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.