Skip to content

fix(server): keep old runtime status from failing sandbox restarts - #5237

Open
shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/4932-lifecycle-observation-complete/shiju-nv
Open

shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/4932-lifecycle-observation-complete/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

During a sandbox restart, a periodic status check can still see the previous run's exited container or no container at all. That stale observation can mark the new start as failed. Protect the start until its driver operation settles, then retain the replacement's actual readiness or failure.

Related Issue

Closes #4932. Related: #4981 and #5114. Rebased onto main after #5204 merged.

Continues @prekshivyas's published candidate in #4932 and shares reconciliation guards and lifecycle coverage with @TaylorMutch's #4983. This also handles atomic start ownership and legacy retries. Resolve the shared scope and landing order before merging; after #4983 lands, reduce this PR to the remaining changes.

Changes

  • Save the new start and its driver ownership together; retries retain the same generation and session.
  • Defer stale observations while preserving readiness checks and genuine later failures.
  • Settle ownership on preparation failure and keep deletion cleanup and unrelated sandboxes responsive.

Release Notes

Sandbox restarts no longer fail because a periodic check applies the previous run's state during the new start.

Testing

  • Independent review of the rebase adaptations
  • Rust formatting and diff checks
  • Unit tests added/updated (if applicable)
  • Rust regressions and E2E on the rebased head: pending hosted CI

Local Rust validation was interrupted by disk exhaustion and removal of the shared build cache.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@shiju-nv
shiju-nv marked this pull request as draft November 4, 2026 05:31
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@shiju-nv shiju-nv added the area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. label Nov 4, 2026
@shiju-nv shiju-nv changed the title fix(server): prevent stale observations from failing sandbox restarts fix(server): keep old runtime status from failing sandbox restarts Nov 4, 2026
@shiju-nv shiju-nv added the test:e2e Requires end-to-end coverage label Nov 4, 2026
@shiju-nv
shiju-nv marked this pull request as ready for review November 4, 2026 11:09
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/5237 is at fe8eaf2 while the PR head is 7a6b276. A maintainer needs to comment /ok to test 7a6b276729a83403776bb80b5e48b2765f7e4c32 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

@pimlock
pimlock deleted the branch NVIDIA:fix/stopped-driver-watch-fence-c92/shiju-nv November 10, 2026 20:25
@pimlock pimlock closed this Nov 10, 2026
@pimlock

pimlock commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

@shiju-nv FYI this got closed automatically.

It looks like it's because the #5204 was merged, which deletes the branch.
Usually what happens in this case is these stacked PRs would be updated to main, but because this PR is opened from the external fork (shiju-nv/OpenShell) that didn't happen and instead this got closed. I cannot reopen it, I think you'll need to update the base branch and open it? Please let me know if you need help.

@shiju-nv shiju-nv reopened this Nov 10, 2026
@shiju-nv
shiju-nv changed the base branch from fix/stopped-driver-watch-fence-c92/shiju-nv to main November 10, 2026 21:28
@shiju-nv
shiju-nv force-pushed the fix/4932-lifecycle-observation-complete/shiju-nv branch from 7a6b276 to a7cc8a0 Compare November 10, 2026 21:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(compute): periodic reconciliation can persist Error during an explicit start

2 participants