Skip to content

Commit d5098d9

Browse files
committed
test(e2e): capture LINSTOR satellite ErrorReports in cozyreport
The linstor-Satellite pod stdout log references its error reports only by ID: [DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to create zfsvolume [Report number 6A4A394E-E4EA0-000000] The body of that report (which carries the actual `zfs create -V …` stderr, the DRBD adjust output, the storage-driver stack trace, and the exit code) lives at `/var/log/linstor-satellite/ErrorReport-*.log` on the satellite pod. crust-gather only captures pod stdout, so the report body stays on the pod's ephemeral filesystem and is lost when the e2e sandbox is torn down at end-of-job. Consequence for post-mortem: a run that failed on a `ResourceExhausted` / `Not enough available nodes` CreateVolume retry loop shows only the driver-level symptom in cozyreport. Root cause attribution (out-of-space, dataset name conflict, ZFS module error, kernel signal) is not distinguishable without the report body, and without the sandbox is unrecoverable. Extend the linstor module in `hack/cozyreport.sh` to tar the `ErrorReport-*.log` set off every `linstor-satellite` pod into `linstor/error-reports/<node>.tgz`. Guarded so a satellite with zero reports (the healthy case) leaves nothing in the tree, and a transient exec / tar failure never fails the surrounding cozyreport run. Verified against a live dev cluster: label selector `app.kubernetes.io/component=linstor-satellite` matches all three satellite pods, and the tar bundle for one active satellite comes out at ~60 KiB. No product-side change. Signed-off-by: Ivan Okhotnikov <[email protected]>
1 parent b6ac249 commit d5098d9

1 file changed

Lines changed: 59 additions & 0 deletions

File tree

‎hack/cozyreport.sh‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,65 @@ if kubectl get deploy -n cozy-linstor linstor-controller >/dev/null 2>&1; then
268268
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color n l > $DIR/nodes.txt 2>&1
269269
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color sp l > $DIR/storage-pools.txt 2>&1
270270
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color r l > $DIR/resources.txt 2>&1
271+
# Cluster-wide ErrorReport index (IDs + timestamps + node + category)
272+
# for fast triage before diving into the per-satellite bundles below.
273+
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color error-reports list > $DIR/error-reports-index.txt 2>&1 || true
274+
275+
# Controller-side ErrorReports live on the controller pod at
276+
# /var/log/linstor-controller/ and cover autoplace decisions, RPC
277+
# errors, and controller-JVM exceptions the index above only
278+
# references by ID. Bundle them the same way as the satellite ones
279+
# below so both ends of the storage stack are recoverable offline.
280+
kubectl -n cozy-linstor exec deploy/linstor-controller --container=linstor-controller -- sh -c '
281+
cd /var/log/linstor-controller 2>/dev/null || exit 0
282+
tar -czf - ErrorReport-*.log 2>/dev/null || true
283+
' > "$DIR/controller-error-reports.tgz" 2>/dev/null || true
284+
# Drop the bundle if empty. `tar -czf - <no-match>` produces a valid
285+
# 45-byte gzipped empty archive (extracts cleanly, `tar -tzf` exits
286+
# 0), so a plain readability check keeps that stub in the tree.
287+
# Require the archive to contain at least one member to keep it.
288+
if [ ! -s "$DIR/controller-error-reports.tgz" ] || [ -z "$(tar -tzf "$DIR/controller-error-reports.tgz" 2>/dev/null | head -n 1)" ]; then
289+
rm -f "$DIR/controller-error-reports.tgz"
290+
fi
291+
292+
# LINSTOR satellite ErrorReports carry the actual storage-driver error
293+
# text (the linstor-Satellite log only references them by ID:
294+
# `ERROR ... Failed to create zfsvolume [Report number 6A4A394E-...]`).
295+
# crust-gather ships only pod stdout, so without this capture the
296+
# report body stays on the satellite ephemeral filesystem and is lost
297+
# when the sandbox is torn down. Copy them off every satellite pod so
298+
# post-mortem of a `CreateVolume ResourceExhausted` or
299+
# `Failed to create zfsvolume` retry loop has the concrete cause
300+
# (out-of-space, dataset conflict, kernel error) instead of just the
301+
# driver-level symptom.
302+
DIR=$REPORT_DIR/linstor/error-reports
303+
mkdir -p "$DIR"
304+
for pod in $(kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite -o name 2>/dev/null); do
305+
# Read the node name directly off the pod spec so the tarball key
306+
# is stable across piraeus DaemonSet regenerations. Fallback to the
307+
# pod name if .spec.nodeName is not readable for any reason (last
308+
# resort; collisions between DS pod generations are theoretically
309+
# possible but the bundle would still land in a distinct file).
310+
node=$(kubectl -n cozy-linstor get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
311+
[ -z "$node" ] && node=$(basename "$pod")
312+
# Tar the ErrorReport-*.log files into a per-satellite bundle so a
313+
# burst of retry-loop reports (dozens per incident) does not explode
314+
# the artefact tree, and a missing directory or empty set never
315+
# fails the whole cozyreport run.
316+
kubectl -n cozy-linstor exec "$pod" --container=linstor-satellite -- sh -c '
317+
cd /var/log/linstor-satellite 2>/dev/null || exit 0
318+
tar -czf - ErrorReport-*.log 2>/dev/null || true
319+
' > "$DIR/$node.tgz" 2>/dev/null || true
320+
# Drop the bundle if empty. `tar -czf - <no-match>` yields a valid
321+
# 45-byte gzipped empty archive that would otherwise slip past a
322+
# readability check. Require at least one member to keep it. A
323+
# satellite with zero ErrorReports (healthy case) leaves nothing.
324+
if [ ! -s "$DIR/$node.tgz" ] || [ -z "$(tar -tzf "$DIR/$node.tgz" 2>/dev/null | head -n 1)" ]; then
325+
rm -f "$DIR/$node.tgz"
326+
fi
327+
done
328+
# Drop the empty parent directory when no satellite had reports.
329+
rmdir "$DIR" 2>/dev/null || true
271330
fi
272331

273332
# -- sandbox-host module

0 commit comments

Comments
 (0)