Skip to content

Volume E2E tests wait for actor restoration before checking mounts - #2294

Merged
Max Smythe (maxsmythe) merged 1 commit into
agent-substrate:mainfrom
cloudonly:fix/e2e-combined-volume-resume-budget
Oct 8, 2026
Merged

Max Smythe (maxsmythe) merged 1 commit into
agent-substrate:mainfrom
cloudonly:fix/e2e-combined-volume-resume-budget

Conversation

@cloudonly

@cloudonly Luomu (cloudonly) commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

The combined-volume E2E test wakes a suspended or reverted actor with its first HTTP probe. On a busy micro-VM worker, restoration can outlast AgentGateway's five-second request-parking budget. The gateway then cancels ResumeActor and returns 504 before the test reaches its volume assertions. A two-minute client context does not extend that gateway budget. This affected main at 3230f9f and main at 0b91488d.

Resume explicitly after suspend and revert through ResumeActorAwaitCapacity, using the existing two-minute test context. Keep every image-volume, durable-directory, shared-mount, CSI reattachment, and revert-content assertion. The separate request-parking suite continues to cover traffic-triggered wakeup and budget exhaustion.

Validation:

  • Baseline Linux/kind run: adding a seven-second delay only to combined-volume micro-VM restores reproduces the same two failing subtests and their parent, with HTTP 504 and ResumeActor timed out after 5s; all 92 gVisor cases pass.
  • Fixed AgentGateway Linux/kind runs: both normal and delayed restores pass the complete gVisor and micro-VM lanes. Each run reports 92 gVisor and 92 micro-VM cases with zero failures; the existing micro-VM node-migration skip remains. All combined-volume subtests execute and pass, including both CSI checks. With the delay, suspend/resume takes 20.92s and the revert subtest 13.57s, without a 504.
  • Envoy E2E and Linux verification: both runtime lanes, race unit tests, root-gated tests, auxiliary modules, and all 14 static-verification scripts pass against the exact PR head, 8ae90df0.
  • Local router-client tests and compilation of the changed E2E package pass. Local make verify on the unmodified baseline and static verification after the change could not finish because the disk filled during compilation (ENOSPC); full verification completed in the Linux run above.

The delay is confined to the personal-fork diagnostic workflow and is absent from this PR. The upstream PR workflow currently requires maintainer approval; the linked personal-fork runs use its same jobs to validate the PR head independently.

  • Tests pass
  • Appropriate changes to documentation are included in the PR (test comments)

The router parking budget can expire while a micro-VM is restoring on a busy node. Resume within the volume test deadline so mount and snapshot assertions do not depend on that shorter budget.
@cloudonly Luomu (cloudonly) changed the title e2e: resume actors before checking restored volumes Volume E2E tests wait for actor restoration before checking mounts Oct 7, 2026
@cloudonly
Luomu (cloudonly) marked this pull request as ready for review October 7, 2026 05:53

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxsmythe
Max Smythe (maxsmythe) added this pull request to the merge queue Oct 8, 2026
Merged via the queue into agent-substrate:main with commit f9de28a Oct 8, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants