Skip to content

[feature] Support host-provided session storage for Stage routing #1304

Open

Description

@afourniernv

Problem

LiteLLM's Stage integration is now on main. Each worker or pod has its own copy of Stage's retained routing state. Turns from one session can therefore see different tier holds, and restarts lose that state.

The current registry has inactivity expiry and session-completion cleanup, but no entry-count cap.

Proposed solution

Allow hosts to supply storage for retained session state, keeping in-memory storage as the default. LiteLLM could use the customer's Redis through the Python bindings, while owning connections, credentials and the Redis implementation.

Reuse the existing serializable State where possible. Rust already supports caller-provided state, but requests with a session ID use the private registry; Python does not expose that caller-state path.

Define concurrent updates, including answer observations, expiry/capacity, session isolation and store failures. Verify that two independently constructed routers can share a session without losing updates.

Alternatives considered

Worker affinity helps keep turns together, but does not share state or preserve it across restarts.

Scope notes

Start with Stage's retained routing state and the Python integration LiteLLM uses. Keep existing callers working. This does not require a Redis dependency in libsy; the exact API still needs agreement.

Additional context

Related: #832 covers provider-owned Responses continuation state, which is separate from Stage routing state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions