Skip to content

Commit 08ac063

Browse files
committed
ci(e2e): move install gate to 32vCPU runner to end CPU oversubscription
The three sandbox QEMU guests each run with `-smp 8`, demanding 24 vCPU in total. On the 24-vCPU runner that is a 1:1 CPU commit with no host headroom for QEMU's emulation/IO threads, vhost-net, the host NAT every guest image pull traverses, containerd and the kernel. Under the peak concurrency of "Install Cozystack" the guests lose cycles to CPU steal, so in-guest readiness/liveness probes miss their deadlines and a HelmRelease intermittently flips InProgress -> Failed with dependents cascading to DependencyNotReady — on a different component each run, the signature of resource thrash rather than a product bug. Move the job to the 32-vCPU / 128 GiB runner class. Guest sizing is unchanged (24 vCPU / 72 GiB), so the install does not slow down, while the host keeps 8 vCPU and 56 GiB of headroom that absorbs the emulation and network overhead the probes were starving against. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 6c69c00 commit 08ac063

1 file changed

Lines changed: 16 additions & 2 deletions

File tree

‎.github/workflows/pull-requests.yaml‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -490,8 +490,22 @@ jobs:
490490
491491
e2e:
492492
name: "E2E Tests"
493-
runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-24cpu-96gb-x86-64' }}
494-
#runs-on: [oracle-vm-32cpu-128gb-x86-64]
493+
# The sandbox boots three QEMU guests (hack/e2e-prepare-cluster.bats), each
494+
# with `-smp 8 -m 24576`, so the guests alone demand 24 vCPU and 72 GiB.
495+
# On the 24-vCPU runner that is a 1:1 CPU commit with zero host headroom for
496+
# QEMU's own emulation/IO threads, vhost-net, the host iptables MASQUERADE
497+
# every guest image pull traverses, containerd, the kernel and the runner
498+
# agent. During "Install Cozystack" (all platform operators + image pulls
499+
# across three nodes at once) the guest vCPUs contend with those host threads
500+
# and lose cycles to CPU steal, so in-guest kubelet/apiserver readiness and
501+
# liveness probes miss their deadlines ("context deadline exceeded"). A
502+
# HelmRelease then flips InProgress -> Failed and its dependents cascade to
503+
# DependencyNotReady — on a different component each run, the signature of
504+
# resource thrash rather than a product bug. The 32-vCPU / 128 GiB class
505+
# keeps the guests' 24 vCPU / 72 GiB unchanged (no slower install) while
506+
# leaving 8 vCPU (25%) and 56 GiB of host headroom, which removes the
507+
# oversubscription that starves the probes.
508+
runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-32cpu-128gb-x86-64' }}
495509
timeout-minutes: 180
496510
permissions:
497511
contents: read

0 commit comments

Comments
 (0)