Skip to content

fix(kubernetes): parallel worker VM provisioning on tenant cluster create - #3177

Closed
IvanHunters wants to merge 1 commit into
mainfrom
fix/tenant-worker-parallel-provisioning
Closed

fix(kubernetes): parallel worker VM provisioning on tenant cluster create#3177
IvanHunters wants to merge 1 commit into
mainfrom
fix/tenant-worker-parallel-provisioning

Conversation

@IvanHunters

@IvanHunters IvanHunters commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Tenant Kubernetes clusters currently take ~13 minutes to reach 2 Ready
workers even though a single worker boots in ~5. The recurring
kubernetes-latest.bats / kubernetes-previous.bats node-join
timeouts (12m budget) are the symptom; the root cause is a 5- to
10-minute gap between "Machine №1 Ready" and "MachineSet.ScalingUp to
2" on the first tenant-cluster rollout — worker provisioning happens
serially instead of in parallel.

This PR is opened as draft and lands in two stages:

  1. Diagnostics patch (this commit) that captures controller-manager
    logs on failure, so the next CI cycle produces the reconcile
    timeline needed to pin the root cause to CAPI / CAPK / Kamaji-CP /
    CABPT / CDI / LINSTOR-CSI. Without those logs any chart-side fix
    would be guesswork.
  2. Chart-side fix + run-kubernetes.sh wait predicate fix, appended
    as separate commits once (1) points at a specific reconciler.

What (so far)

  • hack/e2e-apps/run-kubernetes.sh:352 — extend the existing
    node-join failure diagnostic block with section (d): 15-minute
    window of controller-manager logs across cozy-cluster-api,
    cozy-kamaji, cozy-kubevirt-cdi, cozy-linstor, timestamped,
    all-containers, guarded with || true. Happy path untouched.

Why

Sections (a) and (c) tell us WHERE the worker stalled (VMI/DV vs
tenant kubelet CSR). Neither shows WHY the reconcile fired late —
that lives in controller-manager reconcile events and rate-limiter
backoffs.

Snapshot from #3044 CI run 28568334647
demonstrates the pattern:

```
MachineSet.status.conditions[?type=ScalingUp]:
lastTransitionTime: 2026-07-02T06:45:17Z
message: "Scaling up MachineSet to 2 replicas (actual 1)"
```

The first Machine had been Ready since 06:40:39 — a ~5-minute idle
gap before ScalingUp fired, with no visible reconciler activity in the
current diagnostic dump.

Test plan

  • `bash -n hack/e2e-apps/run-kubernetes.sh` passes (shell syntax).
  • E2E cycle triggered by this push produces a section (d)
    controller log dump on the next `kubernetes-latest.bats`
    or `kubernetes-previous.bats` failure.
  • Root cause identified from those logs; follow-up commit applies
    the fix in `packages/apps/kubernetes/templates/cluster.yaml`
    strategy block and switches the `.status.replicas` wait to
    `.status.readyReplicas` in `hack/e2e-apps/run-kubernetes.sh:275`.
  • Verified on dev3: two `Machine` objects created within 30s of
    each other; both nodes Ready in ~7m.
  • Draft removed once `kubernetes-latest.bats` and
    `kubernetes-previous.bats` both pass on this branch.

@coderabbitai

coderabbitai Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2f69fadc-692b-418a-b017-e1574ce7adb7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/tenant-worker-parallel-provisioning

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug labels Jul 2, 2026
@IvanHunters
test(e2e): capture CAPI/CDI/LINSTOR controller logs on node-join failure
11c025b 
Sections (a) and (c) of the existing node-join failure diagnostic pin
down WHERE the worker stalled (VMI/DV provisioning versus tenant
kubelet CSR). They cannot answer WHY the second Machine was created
minutes after the first became Ready, which is the pattern observed in
recent tenant-latest / tenant-previous flakes: a 5- to 10-minute gap
between Machine №1 Ready and MachineSet.ScalingUp to 2. That gap is
only legible in the CAPI controller-manager reconcile timeline.

Add section (d) that dumps the last 15m of controller-manager output
across cozy-cluster-api (capi-manager, capk-manager,
capi-kamaji-controller-manager, capi-kubeadm-bootstrap-manager,
cabpt-manager), cozy-kamaji, cozy-kubevirt-cdi, and cozy-linstor,
with timestamps and all sidecar containers included. Guarded with
\`|| true\` end-to-end so a dead pod never masks the real \`exit 1\`.
Diagnostics-only change: the happy path is untouched and never runs
these commands.

Signed-off-by: Ivan Okhotnikov <[email protected]>
@IvanHunters
IvanHunters force-pushed the fix/tenant-worker-parallel-provisioning branch from beee9af to 11c025b Compare July 2, 2026 11:48
@myasnikovdaniil

myasnikovdaniil commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

The section (d) diagnostics largely duplicate what the e2e harness already captures on failure.

On any bats failure, hack/cozytest.sh (~L101-109) runs crust-gather collect against the host/management cluster and folds the snapshot into the uploaded cozyreport artifact (hack/cozyreport.sh ~L297-300). That snapshot already contains cluster-wide pod logs for cozy-cluster-api, cozy-kamaji, cozy-kubevirt-cdi, and cozy-linstor — i.e. the controller-manager reconcile timeline you're chasing in (d) is already in the artifact for the runs that failed.

So you can get the root cause without merging anything to main:

  1. Download the cozyreport artifact from the failing kubernetes-latest.bats / kubernetes-previous.bats run.
  2. crust-gather serve -f <snapshot-dir> and point kubectl at it (or read the collected logs directly).
  3. kubectl logs deploy/capi-controller-manager -n cozy-cluster-api (plus the Kamaji / CABPT / CDI controllers) to reconstruct the "Machine №1 Ready → ScalingUp to 2" gap.

Two things the hand-rolled loop would actually do worse than the existing snapshot:

  • kubectl logs --since=15m only reads the current container instance. If a controller was OOMKilled/restarted during the run — a very plausible cause of a multi-minute reconcile gap — those logs are in the previous instance and won't appear. crust-gather captures the fuller history.
  • The loop iterates every pod in cozy-linstor (piraeus, linstor-controller, the satellite/csi-node DaemonSets on every node) with --all-containers, which is a large dump mostly unrelated to MachineSet timing.

Suggested path: pull the artifact from a red run, pin the specific reconciler behind the gap, then open the fix PR against packages/apps/kubernetes/ directly — no need to land log scaffolding first. If it turns out crust-gather genuinely misses something you need (in-guest Talos state, or --previous on a crashed controller), that's a small standalone tooling PR worth doing on its own.

@lexfrei Aleksei Sviridkin (lexfrei) mentioned this pull request Jul 3, 2026
Closed
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

This PR has had no activity for 60 days and was marked lifecycle/stale.
It will be closed in 14 days unless commented or labelled lifecycle/frozen.

@github-actions github-actions Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown

Closed because no activity followed the lifecycle/stale warning.
Reopen if the change is still wanted.

@github-actions github-actions Bot added the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed label Sep 16, 2026
@github-actions github-actions Bot closed this Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kvaps Andrei Kvapil (kvaps) Awaiting requested review from kvaps kvaps will be requested when the pull request is marked ready for review kvaps is a code owner
@lllamnyp Timofei Larkin (lllamnyp) Awaiting requested review from lllamnyp lllamnyp will be requested when the pull request is marked ready for review lllamnyp is a code owner
@lexfrei Aleksei Sviridkin (lexfrei) Awaiting requested review from lexfrei lexfrei will be requested when the pull request is marked ready for review lexfrei is a code owner
@myasnikovdaniil myasnikovdaniil Awaiting requested review from myasnikovdaniil myasnikovdaniil will be requested when the pull request is marked ready for review myasnikovdaniil is a code owner

Assignees

No one assigned

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@IvanHunters @myasnikovdaniil