You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[feature] Support host-provided session storage for Stage routing #1304
LiteLLM's Stage integration is now on main. Each worker or pod has its own copy of Stage's retained routing state. Turns from one session can therefore see different tier holds, and restarts lose that state.
The current registry has inactivity expiry and session-completion cleanup, but no entry-count cap.
Proposed solution
Allow hosts to supply storage for retained session state, keeping in-memory storage as the default. LiteLLM could use the customer's Redis through the Python bindings, while owning connections, credentials and the Redis implementation.
Reuse the existing serializable State where possible. Rust already supports caller-provided state, but requests with a session ID use the private registry; Python does not expose that caller-state path.
Define concurrent updates, including answer observations, expiry/capacity, session isolation and store failures. Verify that two independently constructed routers can share a session without losing updates.
Alternatives considered
Worker affinity helps keep turns together, but does not share state or preserve it across restarts.
Scope notes
Start with Stage's retained routing state and the Python integration LiteLLM uses. Keep existing callers working. This does not require a Redis dependency in libsy; the exact API still needs agreement.
Additional context
Related: #832 covers provider-owned Responses continuation state, which is separate from Stage routing state.
Activity