Skip to content

Sandbox enters Error phase immediately on creation across all agent types and openshell versions #2792

New issue
New issue
Closed
#2795
Closed
Sandbox enters Error phase immediately on creation across all agent types and openshell versions#2792
#2795
Labels
bugcomponent/sandboxOpenShell sandbox environment

Description

@ralphbean
ralphbean
opened on Jun 30, 2026

What happens

Sandbox creation fails on all three retry attempts with the sandbox entering
Phase: Error immediately. The error is consistent across agent types (triage,
code, review), openshell versions (0.0.63, 0.0.71), and downstream repos. All
infrastructure steps (checkout, token minting, GCP setup, agent environment)
succeed — only sandbox creation fails.

The sandbox status output is identical across all failures:

Phase: Error
Resource version: 9
Policy source: sandbox
Revision: 1

The fullsend runner retries 3 times with backoff but every attempt produces
the same immediate Error phase.

What should happen

The sandbox should reach a Ready phase within the 2-minute timeout, or the
error output should include a diagnostic message explaining why the sandbox
entered the Error phase.

Observed instances

Context

This is not transient — it is currently affecting all sandbox creation
attempts. Prior issues #1298 and #1303 described similar symptoms but were
closed as transient/needs-info. The retry logic added from #1303 is working
(3 attempts are made) but does not help because the failure is deterministic,
not transient. The Phase: Error with no accompanying diagnostic makes root-
cause investigation difficult from the runner side.

Activity

  1. ralphbean
    added
    component/sandboxOpenShell sandbox environment
    bug
    on Jun 30, 2026
  2. github-project-automation
    added this to fullsend-ai Workstreamon Jun 30, 2026
  3. github-project-automation
    moved this to Todo in fullsend-ai Workstreamon Jun 30, 2026

  4. fullsend-ai-triage commented on Jun 30, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Jun 30, 2026 – with fullsend-ai-triage

    🤖 Triage · ❌ Terminated · Started 5:53 PM UTC · Ended 6:00 PM UTC
    Commit: 104508d · View workflow run →

  5. ralphbean commented on Jun 30, 2026

    @ralphbean
    ralphbean
    on Jun 30, 2026
    MemberAuthor

    Some data from konflux-ci/.fullsend action runs. There was a healthy success streak from ~15:00–15:22 UTC, then things broke:

    Last successful streak (positions 70–75, newest first):

    Workflow Conclusion Created (UTC) Run
    Review success 15:22:27 https://github.com/konflux-ci/.fullsend/actions/runs/28455634784
    Review success 15:21:40 https://github.com/konflux-ci/.fullsend/actions/runs/28455583140
    Review success 15:16:36 https://github.com/konflux-ci/.fullsend/actions/runs/28455240810
    Review success 15:10:28 https://github.com/konflux-ci/.fullsend/actions/runs/28454829052
    Fix success 15:06:59 https://github.com/konflux-ci/.fullsend/actions/runs/28454597105
    Review success 15:02:18 https://github.com/konflux-ci/.fullsend/actions/runs/28454276272
    Review success 15:00:45 https://github.com/konflux-ci/.fullsend/actions/runs/28454172823

    First failure after that streak: ~15:27 UTC. One isolated success snuck through at 15:27 (https://github.com/konflux-ci/.fullsend/actions/runs/28455970699), but everything else since has been failing.

    The flip happened somewhere between 15:22 and 15:27 UTC.

  6. fullsend-ai-triage commented on Jun 30, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Jun 30, 2026 – with fullsend-ai-triage

    🤖 Finished Triage · ❌ Failure · Started 5:53 PM UTC · Completed 6:00 PM UTC
    Commit: 104508d · View workflow run →

  7. ralphbean commented on Jun 30, 2026

    @ralphbean
    ralphbean
    on Jun 30, 2026
    MemberAuthor

    Clarification on the earlier comment — and adding fullsend-ai/.fullsend data.

    The konflux-ci/.fullsend success streak from 15:00–15:22 UTC was not broken by sandbox errors. The first failure after the streak (Retro at 15:24, run 28455777391) was a post-script failure: 'ready-for-triage' label not found. Sandbox created fine in 4.3s.

    The first sandbox 3/3 timeout in konflux-ci/.fullsend was a Review run created at 15:32 UTC: run 28456285903

    Sandbox creation attempt 1/3 failed (not ready after 2m0s)
Sandbox creation attempt 2/3 failed (not ready after 2m0s)
Error: creating sandbox: sandbox creation failed after 3 attempts

    For fullsend-ai/.fullsend, the picture is similar. The last healthy streak was 4 consecutive successes from 12:58–13:12 UTC. Failures after that were all innocuous (validation failures, post-script issues) until the first sandbox 3/3 timeout at 15:51 UTC: run 28457538105 — same error signature.

    Summary — sandbox 3/3 timeouts started between 15:30–15:50 UTC in both orgs:

    Org First sandbox 3/3 failure Run
    konflux-ci 15:32 UTC 28456285903
    fullsend-ai 15:51 UTC 28457538105

    Everything since has been failing with the same sandbox timeout pattern.

  8. ben-alkov commented on Jun 30, 2026

    @ben-alkov
    ben-alkov
    on Jun 30, 2026
    Member

    Also these failures from fullsend-ai/.fullsend

  9. ralphbean commented on Jun 30, 2026

    @ralphbean
    ralphbean
    on Jun 30, 2026
    MemberAuthor

    Checked whether any fullsend release lines up with the onset of sandbox failures — it doesn't.

    • v0.23.0 was released 2026-06-29 13:59 UTC (~25 hours before sandbox errors started)
    • v0.24.0 was released 2026-06-30 17:27 UTC (~2 hours after sandbox errors started)
    • The ghcr.io/fullsend-ai/fullsend-code:latest image currently tagged is v0.24.0 (built at 17:28 UTC)

    In konflux-ci/.fullsend, the last successful run (28455634784) and the first sandbox-failing run (28456285903) both used the same OpenShell version (0.0.71) and the same cached sandbox image (ghcr.io/fullsend-ai/fullsend-code:latest). No code or image change occurred between them.

    This points to an infrastructure-level issue in the sandbox provisioning backend rather than a code or release change.

  10. ralphbean commented on Jun 30, 2026

    @ralphbean
    ralphbean
    on Jun 30, 2026
    MemberAuthor

    Checked the "Sandbox Images" workflow in fullsend-ai/fullsend. No image was built or pushed in the failure window:

    Time (UTC) Tag Run
    2026-06-29 13:59 v0.23.0 28377562293
    2026-06-30 17:27 v0.24.0 28463406378

    The v0.23.0 sandbox image was built ~25 hours before failures started. The v0.24.0 image was built ~2 hours after. No image rebuild or push happened in between.

    Combined with the earlier findings: same openshell version, same cached sandbox image, no fullsend release — nothing changed on our side between the last successful run and the first sandbox 3/3 timeout.

  11. ralphbean
    added a commit that references this issue on Jun 30, 2026

    fix: bump sandbox_timeout_seconds to 300 for review harness

    b95b879
  12. github-project-automation
    moved this from Todo to In progress in fullsend-ai Workstreamon Jun 30, 2026

  13. ralphbean commented on Jun 30, 2026

    @ralphbean
    ralphbean
    on Jun 30, 2026
    MemberAuthor

    Root cause identified by @waynesun09 in #2795.

    OpenShell v0.0.73 was released at 15:31 UTC — the `supervisor:latest` tag was re-pointed to a build containing NVIDIA/OpenShell#2001 (`fix(supervisor): drop sandbox child capability bounding set`). The `cap_drop_bound()` call fails with `EINVAL` in rootless Podman on GH Actions runners because `CAP_SETPCAP` is unavailable in user namespaces, crashing the supervisor immediately.

    This explains why:

    • The first sandbox 3/3 timeout in `konflux-ci/.fullsend` was at 15:32 UTC (run 28456285903) — one minute after the release
    • The first sandbox 3/3 timeout in `fullsend-ai/.fullsend` was at 15:51 UTC (run 28457538105)
    • Reverting the CLI to 0.0.63 didn't help — the gateway pulls `supervisor:latest` regardless of CLI version
    • Bumping `sandbox_timeout_seconds` to 300 didn't help either — the supervisor crashes, it doesn't just slow down

    Also reverted the `sandbox_timeout_seconds` bump in fullsend-ai/.fullsend since it's not the fix.

  14. ralphbean
    closed this as completedin #2795on Jun 30, 2026
  15. github-project-automation
    moved this from In progress to Done in fullsend-ai Workstreamon Jun 30, 2026
  16. ralphbean
    added 2 commits that reference this issue on Jun 30, 2026

    fix(ci): trigger e2e tests on action.yml changes

    d5c61f5

    fix(ci): trigger e2e tests on action.yml changes

    cbedc19
  17. ben-alkov
    added a commit that references this issue on Jun 30, 2026

    fix: pin supervisor image to match openshell CLI version

    facd02e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugcomponent/sandboxOpenShell sandbox environment

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions