Welcome to production-stack!

K8S-native cluster-wide deployment for vLLM.

Star Watch Fork

vLLM Production Stack project provides a reference implementation on how to build an inference stack on top of vLLM, which allows you to:

• 🚀 Scale from single vLLM instance to distributed vLLM deployment without changing any application code

• 💻 Monitor the metrics through a web dashboard

• 😄 Enjoy the performance benefits brought by request routing and KV cache offloading

• 📈 Easily deploy the stack on AWS, GCP, or any other cloud provider

Documentation

Getting Started

• Prerequisite
  • Table of Contents
  • Prerequisites
  • Steps
  • Conclusion
• Quick Start
  • Table of Contents
  • Prerequisites
  • Steps
• FAQ
  • Installation & Setup
  • Deployment & Configuration
  • Performance & Optimization
  • Troubleshooting

Deployment

• Deployment Overview
  • Deployment Options

Use Cases

• KV Cache Aware Routing
  • Table of Contents
  • Prerequisites
  • Step 1: Deploy with KV Cache Aware Routing
  • Step 2: Port Forwarding
  • Step 3: Testing KV Cache Aware Routing
  • Step 4: Clean Up
  • Conclusion
• Prefix Aware Routing
  • Table of Contents
  • Prerequisites
  • Step 1: Deploy with Prefix Aware Routing
  • Step 2: Port Forwarding
  • Step 3: Testing Prefix Aware Routing
  • Step 4: Clean Up
  • Conclusion
• Disaggregated Prefill
  • Prerequisites
  • Kubernetes Deployment
• Sharing KV Cache Across Instances
  • Table of Contents
  • Prerequisites
  • Step 1: Configuring Remote KV Cache Storage
  • Step 2: Deploying the Helm Chart
  • Step 3: Verifying the Installation
  • Benchmark the Performance Gain of Remote Shared Storage (Work in Progress)
  • Conclusion
• Benchmarking
• Distributed Tracing
  • Table of Contents
  • Prerequisites
  • Step 1: Preparing the Jaeger Configuration File
  • Step 2: Preparing the OpenTelemetry Collector File
  • Step 3: Configuring Model and Monitoring
  • Conclusion
• Tool Enabled Installation
  • Prerequisites
  • Steps
• Pipeline Parallelism with KubeRay
  • Table of Contents
  • Prerequisites
  • Step 1: Basic explanation of Ray and Kuberay
  • Step 2: Preparing the Configuration File
  • Step 3: Applying the Configuration
  • Step 4: Verifying the Deployment
  • Conclusion
• Sleep and Wakeup Mode
  • Table of Contents
  • Prerequisites
  • Step 1: Deploy with Sleep Mode Enable
  • Step 2: Port Forwarding
  • Step 3: Testing sleep Mode Feature for the Engine
  • Step 4: Clean Up
  • Conclusion
• Autoscaling with KEDA
  • Table of Contents
  • Prerequisites
  • Steps
  • Additional Resources
• Intelligent Semantic Routing
  • What is vLLM Semantic Router?
  • Benefits of Integration
  • Table of Contents
  • Prerequisites
  • Step 1: Deploy the vLLM Production Stack
  • Step 2: Deploy vLLM Semantic Router
  • Step 3: Test the Deployment
  • Troubleshooting
  • Conclusion

Developer Guide

• Contributing
  • Submitting a Proposal
  • Opening a Pull Request
  • Code Quality and Validation
  • Adding Examples and Tests
  • DCO and Signed-off-by
  • What to Expect for the Reviews
• Docker Guide
  • Build Docker Image

Community

• Community Meetings
  • Community Events