Skip to content

Commit 8b6537f

Browse files
authored
fix(e2e): collect the CDI controller log and CDI pods in cozyreport (#4414)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The e2e report now keeps the `cdi-deployment` log and the YAML of every pod with CDI's `app=containerized-data-importer` label, in any namespace and any phase. In #4374 a Completed importer pod held the worker disk PVC in `Terminating`, and the report had nothing to show why CDI did not delete it. CDI deletes the importer pod once after a successful import and does not retry. The only record of a failed delete is the controller log, which was never collected. The pod itself was out of reach too, because the broken-pod walk skips Completed pods, so there was no way to see if it ever got a `deletionTimestamp`. The controller log gets a 20000-line tail instead of the usual 2000. CDI runs at `-v=1` and logs every reconcile, and the delete can happen an hour before the report is taken. It does not solve the underlying problem from #4374. It makes the next occurrence diagnosable. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(e2e): the e2e diagnostics report now includes the CDI controller log and CDI pods in every phase. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Diagnostic reports now include up to 20,000 lines each from the current and previous Containerized Data Importer deployment logs. * Reports include YAML snapshots of all importer-associated pods across namespaces, regardless of pod phase or deployment probe results. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents ab0c9f6 + d918887 commit 8b6537f

2 files changed

Lines changed: 83 additions & 1 deletion

File tree

‎hack/cozyreport.bats‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,62 @@ fold_source() {
105105
esac
106106
}
107107

108+
@test "the cdi module keeps the controller log and every CDI-labelled pod whatever its phase" {
109+
# A DataVolume PVC stuck in Terminating can be held by an importer pod that
110+
# CDI failed to delete after a successful import. That pod is Completed, so the
111+
# broken-pod walk skips it, and the only record of why the delete did not land
112+
# is the cdi-deployment log.
113+
block=$(awk '/^# -- cdi module/,/^echo "Collecting services/' "$SCRIPT" | fold_source /dev/stdin)
114+
[ -n "$block" ] || { echo "FAIL: no cdi module in $SCRIPT"; false; }
115+
printf '%s\n' "$block" | grep -qE 'kubectl logs -n cozy-kubevirt-cdi deploy/cdi-deployment --tail=20000$' ||
116+
{ echo "FAIL: the cdi module does not read the current cdi-deployment log"; false; }
117+
printf '%s\n' "$block" | grep -qE 'kubectl logs -n cozy-kubevirt-cdi deploy/cdi-deployment --tail=20000 --previous$' ||
118+
{ echo "FAIL: the cdi module does not read the previous cdi-deployment log"; false; }
119+
printf '%s\n' "$block" | grep -qE 'kubectl get pod -A -l app=containerized-data-importer -o yaml$' ||
120+
{ echo "FAIL: the cdi module does not read CDI-labelled pods across namespaces as YAML"; false; }
121+
case "$block" in
122+
*cozyreport_pods_not_ready*) echo "FAIL: a phase filter would drop the Completed importer pod"; false ;;
123+
esac
124+
}
125+
126+
@test "the cdi pods are read even when the cdi-deployment probe fails" {
127+
# The importer pod is owned by its PVC, not by cdi-deployment, so it outlives a
128+
# Deployment that is gone or a probe that got no answer. That pod is the
129+
# evidence this module exists for, so only the log reads may sit behind the probe.
130+
tmp=$(mktemp -d)
131+
mkdir -p "$tmp/bin"
132+
cat > "$tmp/bin/kubectl" <<'STUB'
133+
#!/bin/sh
134+
echo "kubectl $*" >> "$KUBECTL_CALL_LOG"
135+
case "$*" in
136+
"get deploy "*) exit 1 ;;
137+
*"app=containerized-data-importer"*) printf 'items:\n- metadata:\n name: importer-prime-stuck\n' ;;
138+
esac
139+
exit 0
140+
STUB
141+
chmod +x "$tmp/bin/kubectl"
142+
143+
body=$(awk '/^# -- cdi module/,/^echo "Collecting services/' "$SCRIPT" | sed '$d' | fold_source /dev/stdin)
144+
[ -n "$body" ] || { echo "FAIL: could not locate the cdi module"; false; }
145+
146+
KUBECTL_CALL_LOG="$tmp/calls.txt"
147+
: > "$KUBECTL_CALL_LOG"
148+
REPORT_DIR="$tmp/report"
149+
export KUBECTL_CALL_LOG REPORT_DIR
150+
( COZYREPORT_BOUND="" PATH="$tmp/bin:$PATH" eval "$body" ) >/dev/null
151+
152+
grep -q 'importer-prime-stuck' "$REPORT_DIR/cdi/pods.yaml" 2>/dev/null || {
153+
echo "FAIL: a failed cdi-deployment probe dropped the CDI-labelled pods"
154+
cat "$KUBECTL_CALL_LOG"
155+
false
156+
}
157+
if grep -q '^kubectl logs' "$KUBECTL_CALL_LOG"; then
158+
echo "FAIL: the cdi-deployment log was read although the probe failed"
159+
false
160+
fi
161+
rm -rf "$tmp"
162+
}
163+
108164
@test "CSR capture lists and reads every request on the host cluster and each tenant kubeconfig on disk" {
109165
host=$(fold_source "$SCRIPT" | awk '/^echo "Collecting CertificateSigningRequests/,/^echo "Collecting tenant kubernetes CertificateSigningRequests/')
110166
[ -n "$host" ] || { echo "FAIL: could not locate the host CSR walk"; false; }
@@ -202,7 +258,7 @@ STUB
202258
}
203259
' "$SCRIPT" | grep -v '\$' | LC_ALL=C sort -u | tr '\n' ' ')
204260

205-
expected='cozy-cert-manager/cert-manager cozy-cert-manager/cert-manager-webhook cozy-kamaji/kamaji cozy-linstor/linstor-controller cozy-objectstorage-controller/container-object-storage-controller cozy-system/cozystack-operator '
261+
expected='cozy-cert-manager/cert-manager cozy-cert-manager/cert-manager-webhook cozy-kamaji/kamaji cozy-kubevirt-cdi/cdi-deployment cozy-linstor/linstor-controller cozy-objectstorage-controller/container-object-storage-controller cozy-system/cozystack-operator '
206262

207263
if [ "$found" != "$expected" ]; then
208264
echo "FAIL: the set of Deployments this script names changed."

‎hack/cozyreport.sh‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1902,6 +1902,32 @@ cozyreport_select_objects "virtualmachine instances" kubectl get vmi -A --no-hea
19021902
cozyreport_read_object "$DIR/describe.txt" kubectl describe vmi -n "$NAMESPACE" "$NAME"
19031903
done
19041904

1905+
# -- cdi module
1906+
#
1907+
# CDI deletes an importer pod once, right after marking its PVC Succeeded, and
1908+
# never retries that delete. If that delete fails, the Completed pod keeps
1909+
# pvc-protection on the PVC and a later DataVolume delete hangs. The broken-pod
1910+
# walk skips Completed pods, so every pod carrying CDI's own label is read here,
1911+
# whatever its phase and namespace, next to the controller log where a failed
1912+
# delete is reported. The controller runs at -v=1 and logs every reconcile, and
1913+
# the failed delete can be an hour older than the report, so this log keeps ten
1914+
# times the tail the other controller logs get.
1915+
#
1916+
# Only the log reads sit behind the probe. An importer pod is owned by its PVC,
1917+
# not by cdi-deployment, so it outlives a Deployment that is gone or a probe that
1918+
# got no answer, and it is the evidence this module exists for.
1919+
echo "Collecting CDI state..."
1920+
DIR=$REPORT_DIR/cdi
1921+
mkdir -p "$DIR"
1922+
cozyreport_read_object "$DIR/pods.yaml" \
1923+
kubectl get pod -A -l app=containerized-data-importer -o yaml
1924+
if cozyreport_probe "cdi logs" kubectl get deploy -n cozy-kubevirt-cdi cdi-deployment; then
1925+
cozyreport_read_object "$DIR/cdi-deployment.log" \
1926+
kubectl logs -n cozy-kubevirt-cdi deploy/cdi-deployment --tail=20000
1927+
cozyreport_read_object "$DIR/cdi-deployment-previous.log" \
1928+
kubectl logs -n cozy-kubevirt-cdi deploy/cdi-deployment --tail=20000 --previous
1929+
fi
1930+
19051931
echo "Collecting services..."
19061932
cozyreport_read_object "$REPORT_DIR/kubernetes/services.txt" kubectl get svc -A
19071933
COZYREPORT_OBJECTS_DEADLINE=$(( $(date +%s) + COZYREPORT_OBJECTS_BUDGET_DEFAULT ))

0 commit comments

Comments
 (0)