Feature Request: Session-Level KV Cache Management

Summary

Allow releasing or managing KV cache for individual conversation sessions,
rather than only global release or full restart.

Motivation

As of Splash 1.6.0, the server keeps all ended-session KV pages in a
single pool (pages_cache field in /status) without any ability to
identify or release pages belonging to a specific session.

On a Mac with 64 GB memory running a 35B MoE model, a single day of
conversations can accumulate ~5-6 GB of cached KV pages. These pages
are never reclaimed unless:

• The server restarts (manual restart, crash, or --idle-release of model
  weights — which does not free KV cache pages), or
• The memory budget is reached and the system forces eviction (unpredictable).

Users need precise control to free KV from abandoned or long-idle
conversations without affecting active or recently-used sessions.

Current Limitations

1. No session identity in KV cache

The KV cache exposes:

{
  "kv": {
    "pages_allocated": 18688,
    "pages_active": 0,
    "pages_cache": 18508,   // all ended sessions mixed together
    "pages_free": 245
  }
}

There is no field that maps pages to session IDs. Pages from
different conversations, different times, different lengths are all
lumped together in one pool. The server cannot distinguish a 100K-token
abandoned conversation from a 1K-token recent one.

2. Only global release exists

POST /v1/memory/release releases model weights only, not KV cache
pages. After calling it, the KV cache pages (pages_cache) remain
unchanged. This is expected behavior — the API documentation states it
only releases weights, but there is no equivalent for KV.

3. No way to release a single session

Workarounds are unsatisfactory:

4. Swap and compression happen silently

On a 64 GB machine running Qwen3.6-35B-A3B, memory usage can reach 59 GB
with swap and compression kicking in, making the machine feel slow, yet
there is no API-level way to proactively release stale KV before this
happens.

Proposed API

POST /v1/memory/release_session

Release all KV cache pages belonging to a specific ended session.

Request:

{
  "session_id": "bf7bc35782692fce28cc55ed",
  "all_pages": true   // optional: release ALL pages, not just this session
}

Response (success):

{
  "released_pages": 4521,
  "freed_bytes": 2885114880,
  "session_id": "bf7bc35782692fce28cc55ed"
}

Response (session not found / no pages):

{
  "released_pages": 0,
  "freed_bytes": 0,
  "session_id": "bf7bc35782692fce28cc55ed",
  "message": "no cached pages for this session"
}

GET /v1/memory/sessions

List all sessions with cached KV and their page counts.

Response:

{
  "sessions": [
    {
      "session_id": "bf7bc35782692fce28cc55ed",
      "model": "local/Qwen3.6-35B-A3B-4bit",
      "created_at": 1793957155,
      "last_token_at": 1793960000,
      "idle_minutes": 142,
      "pages_cached": 4521,
      "pages_bytes": 2885114880,
      "peak_tokens": 128450,
      "active": false
    }
  ],
  "total_pages_cached": 18508,
  "total_pages_free": 180
}

POST /v1/memory/release_stale_sessions

Release all sessions that have been idle longer than N minutes.

Request:

{
  "max_idle_minutes": 30
}

Response:

{
  "sessions_released": 5,
  "total_pages_freed": 12800,
  "total_bytes_freed": 819200000
}

POST /v1/memory/release_all_sessions

Release ALL cached KV pages (equivalent to current behavior of full
restart, but without stopping the server).

Response:

{
  "pages_freed": 18508,
  "bytes_freed": 11811160064
}

Benefits

1. No server restart needed — active sessions continue without interruption.
2. Predictable memory management — users can release stale sessions before swap kicks in.
3. Multi-session scenarios — users can target specific long-running abandoned conversations.
4. Better UX for long sessions — users don't need to restart to clear accumulated KV from old threads.
5. Monitoring — GET /v1/memory/sessions lets users see which sessions hold the most KV and take action.

Implementation Notes

Data structures needed

• Each KV page should carry a session_id tag at allocation time (this
  is likely already available internally since sessions are tracked for
  other purposes).
• A map: session_id → [page_ids] to enable lookup.
• When a session ends (request completes or is cancelled), its pages are
  moved to the session's entry in this map rather than the global pool.

Backwards compatibility

• If no session tracking data exists (pre-release sessions), pages fall
  back to the existing global pool.
• New APIs are additive — existing endpoints (/status,
  /v1/memory/release) are unchanged.

Security

• Only the authenticated client can release pages for sessions they
  initiated. No cross-session release between different API keys.
• release_all_sessions should be opt-in (not default) to avoid
  accidental data loss.

Reference

Related issues in the Splash repository:

• #644 — "decoded slows when
  6-12 GB of KV is near GPU's working-set limit" (1.6.0 fixed the GPU
  residency issue, but not the ability to free stale pages).
• #613 — capacity
  exhaustion under memory pressure.

Priority

High — this directly addresses a real user pain point (swap kicking
in silently on 64 GB machines) and enables precise memory management
without sacrificing active sessions.