Repository navigation
feat: implement persistent Suspended conditions for Sandbox - #1150
kubernetes-prow[bot] merged 10 commits into
Conversation
✅ Deploy Preview for agent-sandbox canceled.
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughChangesSuspended condition lifecycle
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Sandbox
participant Controller
participant Pod
Sandbox->>Controller: update operating mode
Controller->>Pod: inspect or delete Pod
Pod-->>Controller: Pod present, terminating, or absent
Controller-->>Sandbox: update Suspended and Ready conditions
Sandbox->>Controller: resume operating mode
Controller->>Pod: recreate Pod
Controller-->>Sandbox: report Suspended=False and Ready=True
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/assign @SHRUTI6991 |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
controllers/sandbox_controller_test.go (1)
334-340: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueStandardize test assertions to use API constants instead of string literals.
This test case (and several others throughout this file) hardcodes string literals for the condition Type, Status, and Reason. For consistency with the rest of the codebase and to protect against future refactoring typos, consider standardizing on the exported API constants (
sandboxv1beta1.SandboxConditionSuspended,metav1.ConditionFalse,sandboxv1beta1.SandboxReasonSuspendedRunning, etc.) as utilized in the E2E tests.🧹 Proposed refactor (example for this occurrence)
{ - Type: "Suspended", - Status: "False", + Type: string(sandboxv1beta1.SandboxConditionSuspended), + Status: metav1.ConditionFalse, ObservedGeneration: 1, - Reason: "Running", + Reason: sandboxv1beta1.SandboxReasonSuspendedRunning, Message: "Sandbox is running.", },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@controllers/sandbox_controller_test.go` around lines 334 - 340, Standardize the condition assertions throughout the test cases in controllers/sandbox_controller_test.go to use exported API constants instead of hardcoded strings. Update Type, Status, and Reason fields in the relevant condition fixtures, including the condition shown, to use the corresponding sandboxv1beta1 and metav1 constants while preserving the existing expected values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@api/v1beta1/sandbox_types.go`:
- Around line 31-37: Restore the exported SandboxReasonSuspendedPodNotTerminated
constant in the v1beta1 API, defining it as a deprecated alias of the
replacement suspended-pod reason constant and documenting the deprecation.
Preserve the existing SandboxReasonSuspendedPodTerminated,
SandboxReasonSuspendedPodTerminating, and SandboxReasonSuspendedRunning values.
---
Nitpick comments:
In `@controllers/sandbox_controller_test.go`:
- Around line 334-340: Standardize the condition assertions throughout the test
cases in controllers/sandbox_controller_test.go to use exported API constants
instead of hardcoded strings. Update Type, Status, and Reason fields in the
relevant condition fixtures, including the condition shown, to use the
corresponding sandboxv1beta1 and metav1 constants while preserving the existing
expected values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 38f3f2ea-8dfc-409a-ab45-c47279ebcf7a
📒 Files selected for processing (8)
api/v1beta1/sandbox_types.gocontrollers/sandbox_controller.gocontrollers/sandbox_controller_test.godocs/keps/119-sandbox-suspended-state/README.mdtest/e2e/basic_test.gotest/e2e/mode_test.gotest/e2e/shutdown_test.gotest/e2e/volumeclaimtemplate_test.go
There was a problem hiding this comment.
Pull request overview
This PR updates the Sandbox controller/API to maintain a persistent Suspended condition (instead of creating/removing it), so suspension state transitions can be observed reliably (e.g., kubectl wait --for=condition=Suspended=False) and lastTransitionTime reflects suspend/resume changes.
Changes:
- Adds/updates
Suspendedcondition reason constants (e.g.,Running,PodTerminating,PodTerminated) and updatescomputeSuspendedConditionto always return a condition (defaulting toFalse/Running). - Updates controller unit tests and multiple e2e tests to assert
Suspended=Falseis present for running sandboxes, and correct transitions during suspend/resume. - Updates KEP-119 documentation to describe the new persistent condition semantics and reason mapping.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
api/v1beta1/sandbox_types.go |
Adds/adjusts Suspended condition reason constants for clearer suspension state reporting. |
controllers/sandbox_controller.go |
Makes Suspended condition persistent and populates it with Status/Reason/Message. |
controllers/sandbox_controller_test.go |
Updates unit tests to expect the persistent Suspended condition in all modes. |
test/e2e/basic_test.go |
Asserts Suspended=False/Running exists for a default running sandbox. |
test/e2e/mode_test.go |
Expands operating-mode test into a suspend → resume lifecycle assertion including Suspended transitions. |
test/e2e/shutdown_test.go |
Adds Suspended=False/Running expectations to shutdown-related e2e assertions. |
test/e2e/volumeclaimtemplate_test.go |
Adds Suspended=False/Running expectations to VCT e2e assertions. |
docs/keps/119-sandbox-suspended-state/README.md |
Updates the documented condition hierarchy and scenario table for the persistent Suspended condition. |
c182f57 to
51469e6
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@api/v1beta1/sandbox_types.go`:
- Around line 38-40: Restore the exported constant SandboxReasonSuspendedRunning
in the v1beta1 API alongside SandboxReasonNotSuspended as a deprecated alias,
preserving the existing value and downstream compilation compatibility. Add the
deprecation documentation to direct users to SandboxReasonNotSuspended.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ea400124-6144-433c-8880-a9b2f51e7685
📒 Files selected for processing (7)
api/v1beta1/sandbox_types.gocontrollers/sandbox_controller.gocontrollers/sandbox_controller_test.gotest/e2e/basic_test.gotest/e2e/mode_test.gotest/e2e/shutdown_test.gotest/e2e/volumeclaimtemplate_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- test/e2e/basic_test.go
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/mode_test.go`:
- Around line 183-186: After adding the test finalizer in the Pod update block,
register t.Cleanup to fetch the Pod with a background context, remove the
"agents.x-k8s.io/test-hold" finalizer when present, and persist the update.
Ignore NotFound errors so cleanup remains safe when the Pod is already deleted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3e713a61-d42f-4e01-a4c2-3f5b7b769c56
📒 Files selected for processing (1)
test/e2e/mode_test.go
6975290 to
5f8e037
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/mode_test.go`:
- Around line 297-355: Enhance TestSandboxPodDeletionKeepsSuspendedFalse to
observe Sandbox status continuously from Pod deletion through recreation, rather
than only checking runningReady afterward. Poll or watch the Suspended condition
during this interval and fail immediately if it becomes True, while retaining
the final recovery and Ready assertion.
- Around line 50-95: Extend the end-to-end suspend/resume assertions around
runningStatus and suspendedStatus to capture the Suspended condition after every
phase. Verify each captured condition has a nonzero lastTransitionTime and that
the timestamp advances for every True↔False transition, including the additional
phase covered by the later status assertions; keep the checks across the
controller/API-server boundary.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: f3aee166-2f38-4049-b504-40a80ffafae4
📒 Files selected for processing (2)
api/v1beta1/sandbox_types.gotest/e2e/mode_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- api/v1beta1/sandbox_types.go
|
@shrutiyam-glitch: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
…ernetes-sigs#1150 Per review: removing the stale Suspended=True condition on resume is a behavior change beyond this PR's scope, and PR kubernetes-sigs#1150 fixes the same staleness in the opposite (K8s-conventional) direction — making Suspended persistent and transitioning it to False instead of removing it. Reverting the removal keeps main's behavior unchanged here and leaves the fix to kubernetes-sigs#1150; a comment now pins that decision so the removal set and kubernetes-sigs#1150 land coherently. Ref kubernetes-sigs#1281
Proposes idle-based automatic suspension and resume-on-traffic: - spec.lifecycle.idlePolicy (idleTimeoutSeconds, action Suspend|Delete, suspendedTTLSeconds, resumeOn, workloadActivity), propagated via SandboxTemplate/SandboxClaim with an override policy - per-sandbox coordination.k8s.io Lease as the activity channel (router, SDKs, and opt-in workload renewals) - opt-in sandbox-router resume path: singleflight operatingMode patch, request held until the backing Pod is ready - source-aware expiry (ExpiryDecision) so idle reclaim and ordinary shutdownTime/TTL expiry keep distinct events and terminal condition messages Depends on KEP-119 persistent Suspended conditions (kubernetes-sigs#1150). Converges with the idle lifecycle policy discussion in kubernetes-sigs#849. Ref kubernetes-sigs#968
Proposes idle-based automatic suspension and resume-on-traffic: - spec.lifecycle.idlePolicy (idleTimeoutSeconds, action Suspend|Delete, suspendedTTLSeconds, resumeOn, workloadActivity), propagated via SandboxTemplate/SandboxClaim with an override policy - per-sandbox coordination.k8s.io Lease as the activity channel (router, SDKs, and opt-in workload renewals) - opt-in sandbox-router resume path: singleflight operatingMode patch, request held until the backing Pod is ready - source-aware expiry (ExpiryDecision) so idle reclaim and ordinary shutdownTime/TTL expiry keep distinct events and terminal condition messages Depends on KEP-119 persistent Suspended conditions (kubernetes-sigs#1150). Converges with the idle lifecycle policy discussion in kubernetes-sigs#849. Ref kubernetes-sigs#968
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: janetkuo, shrutiyam-glitch The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…es-sigs#1150) * base changes * Update the sandbox coditions * Address comment and unit tests * Add more e2e tests * Update KEP document and align API conventions * Replace "Running" reason with "NotSuspended" in KEP matrix and prose to match the implementation. * Deprecate SandboxReasonSuspendedPodNotTerminated in sandbox_types.go. ```release-note Deprecated the `PodNotTerminated` reason string in favor of `PodTerminating`. The Sandbox `Suspended` status condition is now always present after initial reconciliation (clients should inspect its boolean `status` value rather than checking for its presence). ``` * Add reason when pod is not owned * Add Unknown reason status * Update doc and address regression
Port the fix from upstream PR kubernetes-sigs#1150 (commit b9d2a10). computeSuspendedCondition now always returns a condition value (not a pointer). When operatingMode is Running, it returns Suspended: False with reason NotSuspended, clearing the stale Suspended: True left behind after a stop. This prevents the OpenShell gateway from reporting the sandbox as Stopped when it is actually Ready. New Suspended condition reasons: NotSuspended, PodTerminating, PodNotOwned, PodStateUnknown. Deprecated: PodNotTerminated. Fixes: RHBAS-19 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
On Agent Sandbox v1beta1, a `sandbox stop` then `sandbox start` left the sandbox stuck at `Starting` and `start` timed out after 300s, even though the workload pod was `Running 1/1` and the CR reported `Ready=True`. The v1beta1 controller (v0.5.0) sets `Suspended=True (PodTerminated)` on stop and does not clear it on resume, so a resumed CR carries both `Ready=True (DependenciesReady)` and a stale `Suspended=True`. The gateway's `derive_phase` treated any `Suspended=True` condition as `Stopped` before it consulted `Ready`, so a resumed-but-Ready sandbox was read as not-running and the `Starting` phase never advanced to `Ready`. `Ready=True` means the sandbox is usable through this gateway and must win over a `Suspended=True` condition. Only resolve `Stopped` from `Suspended` when `Ready` is not `True`; a genuine stop leaves `Ready` unset or `False`, so it still resolves to `Stopped`. v1alpha1 (which encodes desired state as `spec.replicas` and emits no lingering `Suspended`) is unaffected. The underlying stale condition is an upstream bug in the agent-sandbox controller, fixed by kubernetes-sigs/agent-sandbox#1150 (in v0.5.6). This gateway change keeps OpenShell resilient regardless of controller version. Fixes NVIDIA#2932 Signed-off-by: Russell Bryant <[email protected]>
…DIA#2933) On Agent Sandbox v1beta1, a `sandbox stop` then `sandbox start` left the sandbox stuck at `Starting` and `start` timed out after 300s, even though the workload pod was `Running 1/1` and the CR reported `Ready=True`. The v1beta1 controller (v0.5.0) sets `Suspended=True (PodTerminated)` on stop and does not clear it on resume, so a resumed CR carries both `Ready=True (DependenciesReady)` and a stale `Suspended=True`. The gateway's `derive_phase` treated any `Suspended=True` condition as `Stopped` before it consulted `Ready`, so a resumed-but-Ready sandbox was read as not-running and the `Starting` phase never advanced to `Ready`. `Ready=True` means the sandbox is usable through this gateway and must win over a `Suspended=True` condition. Only resolve `Stopped` from `Suspended` when `Ready` is not `True`; a genuine stop leaves `Ready` unset or `False`, so it still resolves to `Stopped`. v1alpha1 (which encodes desired state as `spec.replicas` and emits no lingering `Suspended`) is unaffected. The underlying stale condition is an upstream bug in the agent-sandbox controller, fixed by kubernetes-sigs/agent-sandbox#1150 (in v0.5.6). This gateway change keeps OpenShell resilient regardless of controller version. Fixes NVIDIA#2932 Signed-off-by: Russell Bryant <[email protected]>
What this PR does / why we need it:
This PR formally implements the
Suspendedpersistent condition forSandboxresources to accurately track the suspension state and lifecycle.Note: These changes are based out of PR #893.
Current behavior:
computeSuspendedConditionreturnsnilwheneverspec.operatingMode == Running, so the controller only sets/updates theSuspendedcondition while the Sandbox is suspended. As a result:Suspendedcondition at all.True— it never transitionsto
False, so itslastTransitionTimenever records when the Sandbox resumed.kubectl wait --for=condition=Suspended=Falsecan’t work, because thecondition is not maintained as a proper
Falsestate.This contradicts the Kubernetes convention that conditions are persistent once initialized and transition their
status(rather than appearing/disappearing), and thatlastTransitionTimeis preserved as history.Applied behavior:
The
Suspendedcondition is now persistent once initialized, properly transitioning its status (TrueorFalse) instead of appearing and disappearing. This preserves thelastTransitionTimehistory and enables standardkubectl waitcommands.States and their meaning:
False(Reason:NotSuspended): The Sandbox is active. This state is set immediately upon creation or when resumption is requested (Spec.OperatingMode = Running), even before the underlying Pod is actually provisioned.False(Reason:PodTerminating): The Sandbox is in a transient suspension phase where the underlying Pod is currently terminating.True(Reason:PodTerminated): The Sandbox is fully suspended, and the underlying Pod has been successfully terminated.Key changes:
sandbox_types.go): AddedSandboxConditionSuspendedand specific suspension reasons (PodTerminated,PodTerminating,NotSuspended).sandbox_controller.go): UpdatedcomputeSuspendedConditionto properly construct and populatemetav1.Conditionobjects for theSuspendedtype. The condition evaluates toTruewhen fully suspended (Pod is terminated) andFalsewhen active or during a transient suspension phase (e.g.,PodTerminating).keps/119-sandbox-suspended-state): Updated the condition hierarchy documentation to explicitly define theSuspendedstatus values and reasons, reflecting the new standard Kubernetes condition mechanics.Testing:
sandbox_controller_test.goand all E2E test suites (basic_test.go,shutdown_test.go,volumeclaimtemplate_test.go) to assert the presence and accuracy of the newSuspendedcondition under various scenarios.mode_test.go-TestSandboxSuspendResumeCycleWithOperatingMode,TestSandboxSuspensionWithTerminatingPod,TestSandboxPodDeletionKeepsSuspendedFalseThis enables
kubectl wait --for=condition=Suspended=Falseand preserves the suspend/resume transition history vialastTransitionTime.Note: When resume is requested (
Spec.OperatingMode = Running), theSuspendedcondition status transitions toFalse(Running) immediately, even before the underlying Pod is actually provisioned or created (pod == nil).Which issue(s) this PR is related to:
Fixes #873
Release Note
Summary by CodeRabbit
Summary