You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8758cbf
Browse filesBrowse the repository at this point in the historyBrowse files
feat(server): keep pool sandbox sessions on their supervisor's gateway (NVIDIA#5052)
* feat(server): place pool sandbox sessions with their supervisor's replica
A shared supervisor pool sandbox's session now goes to the gateway
replica serving its owning member's stream, instead of the replica the
ring picks by sandbox ID. Each pool reconciliation pass publishes the
sandbox-to-replica map; sessions fall back to the ring when the parent
is unknown, not a live peer, or this replica is shutting down.
Signed-off-by: divesh <[email protected]>
* feat(server): move live pool sessions to their supervisor's replica
A pool sandbox session served by a replica other than the one serving its owning member is redirected there on its next heartbeat, reusing the shutdown handoff so the sandbox stays Ready. This re-homes sessions left on the wrong replica by a rolling restart.
Signed-off-by: divesh <[email protected]>
* refactor(server): drop gateway-side pool session placement
Removes the per-sandbox parent map and the heartbeat relocation; pool sessions will dial their supervisor's gateway directly instead.
Signed-off-by: divesh <[email protected]>
* feat(supervisor): dial pool sandbox sessions to the host's gateway replica
HostAccepted now carries the accepting replica's peer endpoint. A shared host's sandbox sessions dial it directly, so each session runs on the replica serving its supervisor. A stopping replica sheds its host streams, and redirected sessions follow the host to its next replica. ClosesNVIDIA#5051.
Signed-off-by: divesh <[email protected]>
* test(e2e): keep pool sessions on their supervisor's gateway through failover
Add the shared-supervisor-tree lane: two gateway replicas on PostgreSQL with a
two-member supervisor pool. After start, a gateway crash, a gateway rolling
restart, and a supervisor crash, every pool sandbox's session runs on the
replica serving its supervisor and exec reaches every sandbox.
Signed-off-by: divesh <[email protected]>
* fix(supervisor): move established pool sessions when their host changes gateway
A host whose stream ends, for example when its supervisor token expires, can
reattach to another replica. Its sandbox sessions now watch for that and
reconnect to the host's new replica instead of staying on the old one.
Signed-off-by: divesh <[email protected]>
* test(e2e): fold the supervisor tree checks into the shared-supervisor-ha lane
Run the shared-supervisor-ha lane with two gateway replicas on PostgreSQL and
move the gateway crash, rolling restart, and supervisor crash colocation test
into it, replacing the separate shared-supervisor-tree lane.
Signed-off-by: divesh <[email protected]>
* fix(supervisor): redial the accepting gateway replica after a host stream ends
A host's stream ends routinely, for example when its supervisor token
expires. Redialing the replica that accepted it, before falling back to the
Service, keeps the host and its sandbox sessions on that replica instead of
moving them all on every reconnect.
Signed-off-by: divesh <[email protected]>
* fix(supervisor): follow a host move that lands during session setup
An established session marked its host's current gateway as seen before
watching it, so a move that landed while the session was connecting was
never followed. Compare the current gateway first, then wait for changes.
Signed-off-by: divesh <[email protected]>
* fix(supervisor): dial the configured gateway at once when the host's replica has no peer endpoint
Validate the advertised peer endpoint scheme with the shared supervisor, and document that pool sessions follow their supervisor's gateway.
Signed-off-by: divesh <[email protected]>
* test(e2e): harden the supervisor tree check's port-forward and time limit
Signed-off-by: divesh <[email protected]>
---------
Signed-off-by: divesh <[email protected]>
|`e2e:kubernetes:shared-supervisor`| One-member shared supervisor pool for the gateway's sandboxes (gateway TLS through the port-forward) |
727
-
|`e2e:kubernetes:shared-supervisor-ha`| Two-replica shared supervisor pool: kills the member that owns a sandbox and checks the sandbox is adopted, not restarted |
727
+
|`e2e:kubernetes:shared-supervisor-ha`| Two-replica shared supervisor pool behind two gateway replicas on PostgreSQL: kills the member that owns a sandbox and checks the sandbox is adopted, not restarted; crashes and restarts gateways and checks each sandbox's session stays on its supervisor's gateway|
728
728
|`e2e:kubernetes:shared-supervisor-scaling`| One-member shared supervisor pool: scales it to two members and back with `helm upgrade`, checking sandboxes stay running and the removed member hands its sandboxes off |
0 commit comments