You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 013b166
Browse filesBrowse the repository at this point in the historyBrowse files
fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard (#4970)
`cozy_guard_all_helmreleases` in
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` reads the parent
`kubernetes-<test>` release and the addons selected by the
`kubernetes-<test>-` prefix. The worker pool is
`kubernetes-nodes-<test>-md0`, an Application HelmRelease the aggregated
apiserver stamps with `RetryOnFailure` like the parent, and it sits on
the node-join path through the talos-reconcile Job it renders. Its name
does not start with the addon prefix, so a failed action on it was never
read, on the passing path or in the failure-path report.
This reads it on both paths, with the `required` strategy check the
parent gets, since the apiserver stamps both. The pool name is derived
from the parent (`cozy_pool_helmrelease_name`), the way the addon prefix
is, so the two cannot drift. The failure-path report checks that the
release exists before reading it, with `get hr --ignore-not-found -o
name`: a run whose pool apply failed has no pool release (exit 0, empty
output), and the report says so instead of reporting a failed read. For
that run to reach the report at all, `run_kubernetes_test` now arms the
exit handler (`_tenant_failure_on_exit`, with
`CURRENT_TENANT_PARENT_HR`) right after the parent `Kubernetes` apply
instead of after the `KubernetesNodes` apply: before, a failing pool
apply exited under errexit with no handler installed, and the parent
went unclassified. The snapshot half still no-ops until the tenant
kubeconfig is set. A non-zero exit (a timeout, a kill, a Forbidden) is
reported as the failed read it is, through `_cozy_guard_read_failure`,
never as absence. The pool reads are gated on the report's own deadline
as well as the phase clock, like the addon listing.
Unit tests in `hack/run-kubernetes-remediation_test.bats` cover a
failing pool release on the passing path, a clean one (the composite
returns zero), an empty strategy on the pool, the report with and
without the pool release, a cut-off existence read, and a pool read past
the report budget; the ordering pin now also requires the handler to be
armed before the pool apply. The kubectl stub gained an existence read:
an `ABSENT` marker prints nothing and exits 0 under `--ignore-not-found`
(NotFound without it, as kubectl does), and the failure markers apply to
it as `<name>.exists`.
Verified with `hack/cozytest.sh` and Bats on
`hack/run-kubernetes-remediation_test.bats` (51 tests),
`hack/run-kubernetes-node-join_test.bats` and
`hack/remediation-guard.bats`.
Closes#4737
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Kubernetes release checks now include worker-pool releases alongside
the parent release and add-ons, and flag missing or invalid install
strategies and remediation failures.
* Failure reports distinguish worker-pool releases that are absent from
those whose status cannot be read. Existing releases are assessed
against the phase gate and report time budget; when the budget is
exhausted, skipped checks are identified.
* Failure handling now begins immediately after the parent resource is
applied, covering errors that occur while applying the worker pool.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
# The pool release is created by a later apply than the parent, and
5563
+
# run_kubernetes_test arms the exit handler between the two, so a run whose
5564
+
# pool apply failed reaches this with none to read. Its absence is then a fact
5565
+
# about where the run stopped, not a failed read, and is reported as such.
5566
+
# Only an exit 0 with empty output means absent: --ignore-not-found makes
5567
+
# NotFound that, so any non-zero exit is a read that failed and is named as
5568
+
# one.
5569
+
_pool=$(cozy_pool_helmrelease_name "${_parent}")
5570
+
cozy_diag_phase_has_time "HelmRelease remediation footprint of ${_pool}"||return 0
5571
+
if [ "$(date +%s)"-ge"${_deadline}" ];then
5572
+
echo"=== HelmRelease remediation footprint: stopped after ${_budget}s; ${_pool} and the addon releases of ${_parent} were not read, so nothing was observed about any of them ===">&2
5573
+
return 0
5574
+
fi
5575
+
_cozy_guard_read_bounds
5576
+
_read_rc=0
5577
+
_pool_found=$(_cozy_guard_kubectl get hr -n "$_ns""${_pool}" --ignore-not-found -o name)|| _read_rc=$?
5578
+
if [ "${_read_rc}"-ne 0 ];then
5579
+
echo"Checking whether ${_pool} exists failed - $(_cozy_guard_read_failure "${_read_rc}"); nothing was observed about it either way.">&2
5580
+
elif [ -z"${_pool_found}" ];then
5581
+
echo"» ${_pool} does not exist - the run ended before the worker pool was applied, or its HelmRelease was never created; nothing to read"
cozy_diag_phase_has_time "the addon HelmRelease listing of ${_parent}"||return 0
5545
5586
if [ "$(date +%s)"-ge"${_deadline}" ];then
5546
5587
echo"=== HelmRelease remediation footprint: stopped after ${_budget}s; the addon releases of ${_parent} were not listed, so nothing was observed about any of them ===">&2
@@ -5723,6 +5764,28 @@ ${ouroboros_addon}
5723
5764
version: "${k8s_version}"
5724
5765
EOF
5725
5766
5767
+
# Armed here, at the parent apply, rather than once the tenant answers. The
5768
+
# parent HelmRelease exists from this point on -- the aggregated apiserver
5769
+
# creates it before it answers the apply above -- and everything between here
5770
+
# and the tenant LB is a management-cluster wait on that release or on what it
5771
+
# installs: the KamajiControlPlane, the control-plane Deployments, the
5772
+
# MachineDeployment, the admin-kubeconfig Secret. That window is the one the
5773
+
# guard was written for. An install action that times out while
5774
+
# admin-kubeconfig is still being provisioned leaves the failed Snapshot this
5775
+
# reads, and every one of those waits exits under errexit with no reading at
5776
+
# all if the handler is armed after them.
5777
+
#
5778
+
# Ahead of the pool apply below as well, not after it: a pool apply that fails
5779
+
# exits under errexit too, and armed after it that run would leave the parent
5780
+
# unclassified. Armed here, it is the run the report's "pool release does not
5781
+
# exist" line describes.
5782
+
#
5783
+
# Arming this early costs the snapshot nothing: _tenant_snapshot_on_fail
5784
+
# returns on an unset CURRENT_TENANT_KC, which is assigned where the
5785
+
# kubeconfig becomes usable, so it no-ops until then.
0 commit comments