What the lmdeploy-distserve is included:

1. A Simple Router for PD Scheduling
2. A Transfer Engine for KVCache Migration
3. A Migration Scheduler for Overlaping Migration and Decode

State of lmdeploy-distserve:

1. pass the functional validation (in H800 with RoCE Supported).

Next Step

• substitute the router to LMDeploy Proxy.
• Support multi Prefill and multi Decode Engines and cross node LLM like deepseek v3.
• We Learned a lot from https://github.com/bytedance/InfiniStore when build our minimum implementation. @thesues. Besides, we have strong motivation to adapte infinStore because of its pythonic and well python async support.
• Some micro benchmark.

Initialization

The PD Consolidation process outlines the sequential steps for establishing peer-to-peer (P2P) connections between system components. The process begins with the Router , which acts as the central orchestrator. First, the Router initiates the connection setup by sending a p2p_initialize message to both the Prefill Server and the Decode Server . This ensures that all necessary components are prepared for the subsequent connection phase.

Once the initialization phase is complete for both the Prefill Server and the Decode Server , the Router proceeds to establish the actual P2P connections. It sends a p2p_connect message to the Prefill Server to finalize the connection, followed by another p2p_connect message to the Decode Server . This systematic approach ensures that all components are properly initialized before any connections are established, forming a cohesive network during the system's startup phase.

Control Plane

The diagram illustrates the workflow and interactions between various components involved in the system's prefill and decode processes. This process is designed to manage tasks efficiently, ensuring smooth operation and scalability.

Prefill Process:

The Prefill Server initiates the prefill process by sending a Prefill Message to the Prefill Engine .
The Prefill Engine processes the request and generates an EngineOutput, which includes details such as FirstToken and Cahce BlockIds.
The Prefill Scheduler receives the output from the Prefill Engine and manages task scheduling. Tasks are placed into a Waiting Queue with a status of Status.WAITING.
Once ready, the tasks are forwarded to the Forward Executor , which processes them with a status of Status.RUNNING. The status will be converted to Status.ToBeMigrated and will be free when decode engine migration done.

Decode Process:

The Decode Server sends requests to the Decode Engine , which processes the input and generates an EngineOutput. This output may include details like GenToken. The Decode Scheduler manages the decoded tasks and places them into a Migration Queue with a status of Status.WaitingMigration. The Migration Executor processes these tasks, transitioning their status to Status.Running. Completed tasks are then sent back to the Forward Executor for further processing (Prefill Engine cache_free).

Key Features

• Task Management: Both the Prefill Scheduler and Decode Scheduler play crucial roles in managing task queues and ensuring efficient execution.
• Executor Roles: The Forward Executor and Migration Executor handle the execution of tasks in different stages of the pipeline.
• Scalability and Monitoring (Future work): The inclusion of load balancing, offline profiling, online monitoring, and scaling mechanisms ensures the system remains robust and adaptable under varying workloads.

This structured approach enables seamless coordination between components, facilitating efficient task execution and system control within the Control Plane .

Data Plane

The diagram illustrates the workflow and interactions between key components responsible for managing cache operations, migration, and load balancing. This process is designed to optimize data handling and ensure efficient resource utilization.

Prefill CacheEngine:

The Prefill CacheEngine handles caching operations for prefill tasks. It interacts with the MigrationBackend.Store to store cached data, which can be migrated or loaded as needed.

Decode CacheEngine:

The Decode CacheEngine manages caching operations for decode tasks. It interacts with the MigrationBackend.Load to retrieve cached data when required.

Optional Store Component:

An optional Store component is included, which can be utilized for additional storage needs.This component may interact with the MigrationBackend.Store to manage persistent storage or backup mechanisms.

Migration Operations:

Both the Prefill CacheEngine and Decode CacheEngine utilize the MigrationBackend.Migrate functionality to migrate cached data as necessary. This ensures that cached data can be efficiently moved between different storage locations or systems, maintaining data consistency and availability.

Key Features

• Cache Management: The Prefill CacheEngine and Decode CacheEngine are responsible for managing cached data during prefill and decode operations, respectively.
• Migration and Load Balancing: The MigrationBackend facilitates data migration and loading, ensuring that cached data is appropriately managed and accessible.
• Scalability and Flexibility: The inclusion of an optional Store component allows for additional storage flexibility, while the Router (Load Balance) ensures efficient distribution of workloads.

This structured approach enables seamless coordination between components, facilitating efficient data handling and system control within the Data Plane .

How to build

pip install dlslime==0.0.1.post2
pip install -v -e .

How to Run

• in Server Side
  Step 1. Start Prefill Engine

lmdeploy serve api_server Qwen/Qwen2.5-72B --server-port 23333 --role Prefill --tp 8 --cache-block-seq-len 256

Step 2. Start Decode Engine

lmdeploy serve api_server Qwen/Qwen2.5-72B --server-port 23333 --role Prefill --tp 8 --cache-block-seq-len 256

Step 3. Start Router

python -m lmdeploy.disagg.router --host 0.0.0.0 --port 5000 --prefill-endpoint http://10.130.8.139:23333 --decode-endpoint http://10.130.8.139:23334

• in Client Side

curl -X POST "http://localhost:5000/v1/completions" \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-72B", "prompt": "Shanghai is a city that", "max_tokens": 32, "stream": true}'