bug(e2e): file_transfer::upload_download_round_trip stalls on the k3s-ha conformance row #5520

Activity

1. matthewgrossman commented on Nov 12, 2026

   

   matthewgrossman

   on Nov 12, 2026

   Member

   More actions

   Two more occurrences on the same row, both from PR #5431. Each time, a sandbox that usually finishes in 10–20 s hangs in Kubernetes. The failing test is different each time, and all three predate #5431.

   Run	Failed test (position in suite)	Symptom
   42920206036, job 130200088405 (Nov 10)	cli::cli_stop_start_delete (2nd, about 1 minute in)	sandbox delete returned Internal error: delete sandbox failed: ServiceError: request timed out
   43231863950, job 131289925873 (re-run)	file_transfer::upload_respects_gitignore (12th)	cleanup delete timed out after 120 s (253 s total)
   same job	persistent_workspace::persistent_workspace_survives_stop_start	after the start, the new pod never became ready (614 s)

   The re-run is the only one that captured a lower-level cause. The pod's openshell-sandbox-bootstrap container failed to start:

   runc create failed: unable to start container process: unable to apply cgroup configuration:
   unable to start unit "cri-containerd-openshell-sandbox-bootstrap.scope" ...:
   Message recipient disconnected from message bus without replying
   

   The same error hit an unrelated os-supervisor-* pod at the same time. So systemd/D-Bus on the tmachine VM stopped answering runc's scope requests. That matches the delete and stop hangs.

   Ruled out in all three runs:

   • no sandboxes left at the end;
   • k3s NRestarts=0;
   • about 2.6 GB of the VM's 3.9 GB memory still available, disk 36% used;
   • agent-sandbox controller restarts happened only during environment setup, before any test ran.

   The k3s journal in the diagnostics covers only the last seconds of the run, so it doesn't show what happened to systemd/D-Bus. #5334 (time-bounded journals) would capture it, ideally with the system journal (journalctl --since <run start>, not just the k3s unit).

   Possibly relevant: #5431 runs 36 tests on this row instead of 22, so it spends longer on the VM. But the Nov 10 failure came about 1 minute in, so suite length alone doesn't explain it.
2. elezar commented on Nov 13, 2026

   

   elezar

   on Nov 13, 2026

   MemberAuthor

   More actions

   Another occurrence on the same row, with a different test: filesystem_policy::replaced_policy_path_rejects_next_launch.

   Failing run 43430611200, job 131923809738 (ubuntu-k3s, k3s-ha, after #5431 merged). The other 35 conformance tests passed. This one hit nextest's 120 s limit (TIMEOUT [120.006s] (14/36)) with no output after its startup lines.

   It is a stall, not a slow test. The same test passed on #5431's own k3s-ha run (43405949756) in 15.5 s.

   Evidence from the failing run's diagnostics (sandboxes test-7lz6u86w-{lbp,lhp,lba,lha}):

   • All four pods were scheduled and started within about 2 s of each other.
   • lbp logged FailedMount: MountVolume.SetUp failed for volume "openshell-sandbox-bootstrap": failed to sync secret cache: timed out waiting for the condition, then started about a second later than the others. This is a different error from the runc/D-Bus one in the comment above, but it is also a node-side stall.
   • The four agent containers were then killed in reverse creation order (lha, lba, lhp, lbp), about 10 to 30 s after they started. That matches the testkit cleanup hooks running last-registered-first, so the test body likely finished or stopped early and the time went into cleanup.
   • The gateway kept lbp in its store until a reconcile at 14:05:25 (Removing sandbox from store after it disappeared from the compute driver snapshot ... age_secs=357). That is about 4 minutes after its pod was killed. If the last cleanup wait_deleted was polling for the gateway to stop returning the sandbox, it would have waited on that reconcile. This is an inference; the logs do not show it directly.

   Why there was no message: testkit's own limits are all at or above nextest's 120 s kill: CLEANUP_TIMEOUT, DELETE_TIMEOUT and EXEC_TIMEOUT are 2 min each, READY_TIMEOUT is 10 min. A stall in any of those steps is killed by nextest first, so the failure shows as a bare TIMEOUT. A per-step deadline shorter than 120 s would name the step that stalled.

   Evidence gap: the diagnostics' gateway log tail starts at 14:03:44, after the test window, and gateway.log is one line (-- No entries --), so I cannot confirm what the gateway did during the stall. This is the gap #5333 describes.