Skip to content

Commit 9cc4fa1

Browse files
committed
test(e2e): capture LINSTOR satellite ErrorReports in cozyreport
The linstor-Satellite pod stdout log references its error reports only by ID: [DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to create zfsvolume [Report number 6A4A394E-E4EA0-000000] The body of that report (which carries the actual `zfs create -V …` stderr, the DRBD adjust output, the storage-driver stack trace, and the exit code) lives at `/var/log/linstor-satellite/ErrorReport-*.log` on the satellite pod. crust-gather only captures pod stdout, so the report body stays on the pod's ephemeral filesystem and is lost when the e2e sandbox is torn down at end-of-job. Consequence for post-mortem: a run that failed on a `ResourceExhausted` / `Not enough available nodes` CreateVolume retry loop shows only the driver-level symptom in cozyreport. Root cause attribution (out-of-space, dataset name conflict, ZFS module error, kernel signal) is not distinguishable without the report body, and without the sandbox is unrecoverable. Extend the linstor module in `hack/cozyreport.sh` to tar the `ErrorReport-*.log` set off every `linstor-satellite` pod into `linstor/error-reports/<node>.tgz`. Guarded so a satellite with zero reports (the healthy case) leaves nothing in the tree, and a transient exec / tar failure never fails the surrounding cozyreport run. Verified against a live dev cluster: label selector `app.kubernetes.io/component=linstor-satellite` matches all three satellite pods, and the tar bundle for one active satellite comes out at ~60 KiB. No product-side change. Signed-off-by: Ivan Okhotnikov <[email protected]>
1 parent b6ac249 commit 9cc4fa1

1 file changed

Lines changed: 54 additions & 0 deletions

File tree

‎hack/cozyreport.sh‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,60 @@ if kubectl get deploy -n cozy-linstor linstor-controller >/dev/null 2>&1; then
268268
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color n l > $DIR/nodes.txt 2>&1
269269
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color sp l > $DIR/storage-pools.txt 2>&1
270270
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color r l > $DIR/resources.txt 2>&1
271+
# Cluster-wide ErrorReport index (IDs + timestamps + node + category)
272+
# for fast triage before diving into the per-satellite bundles below.
273+
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor --no-color error-reports list > $DIR/error-reports-index.txt 2>&1 || true
274+
275+
# Controller-side ErrorReports live on the controller pod at
276+
# /var/log/linstor-controller/ and cover autoplace decisions, RPC
277+
# errors, and controller-JVM exceptions the index above only
278+
# references by ID. Bundle them the same way as the satellite ones
279+
# below so both ends of the storage stack are recoverable offline.
280+
kubectl -n cozy-linstor exec deploy/linstor-controller --container=linstor-controller -- sh -c '
281+
cd /var/log/linstor-controller 2>/dev/null || exit 0
282+
tar -czf - ErrorReport-*.log 2>/dev/null || true
283+
' > "$DIR/controller-error-reports.tgz" 2>/dev/null || true
284+
if [ ! -s "$DIR/controller-error-reports.tgz" ] || ! tar -tzf "$DIR/controller-error-reports.tgz" >/dev/null 2>&1; then
285+
rm -f "$DIR/controller-error-reports.tgz"
286+
fi
287+
288+
# LINSTOR satellite ErrorReports carry the actual storage-driver error
289+
# text (the linstor-Satellite log only references them by ID:
290+
# `ERROR ... Failed to create zfsvolume [Report number 6A4A394E-...]`).
291+
# crust-gather ships only pod stdout, so without this capture the
292+
# report body stays on the satellite ephemeral filesystem and is lost
293+
# when the sandbox is torn down. Copy them off every satellite pod so
294+
# post-mortem of a `CreateVolume ResourceExhausted` or
295+
# `Failed to create zfsvolume` retry loop has the concrete cause
296+
# (out-of-space, dataset conflict, kernel error) instead of just the
297+
# driver-level symptom.
298+
DIR=$REPORT_DIR/linstor/error-reports
299+
mkdir -p "$DIR"
300+
for pod in $(kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite -o name 2>/dev/null); do
301+
# Read the node name directly off the pod spec so the tarball key
302+
# is stable across piraeus DaemonSet regenerations. Fallback to the
303+
# pod name if .spec.nodeName is not readable for any reason (last
304+
# resort; collisions between DS pod generations are theoretically
305+
# possible but the bundle would still land in a distinct file).
306+
node=$(kubectl -n cozy-linstor get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
307+
[ -z "$node" ] && node=$(basename "$pod")
308+
# Tar the ErrorReport-*.log files into a per-satellite bundle so a
309+
# burst of retry-loop reports (dozens per incident) does not explode
310+
# the artefact tree, and a missing directory or empty set never
311+
# fails the whole cozyreport run.
312+
kubectl -n cozy-linstor exec "$pod" --container=linstor-satellite -- sh -c '
313+
cd /var/log/linstor-satellite 2>/dev/null || exit 0
314+
tar -czf - ErrorReport-*.log 2>/dev/null || true
315+
' > "$DIR/$node.tgz" 2>/dev/null || true
316+
# Drop the bundle on failure so an operator does not have to unpack
317+
# a stub. A satellite with zero ErrorReports is the healthy case and
318+
# should leave nothing in the tree.
319+
if [ ! -s "$DIR/$node.tgz" ] || ! tar -tzf "$DIR/$node.tgz" >/dev/null 2>&1; then
320+
rm -f "$DIR/$node.tgz"
321+
fi
322+
done
323+
# Drop the empty parent directory when no satellite had reports.
324+
rmdir "$DIR" 2>/dev/null || true
271325
fi
272326

273327
# -- sandbox-host module

0 commit comments

Comments
 (0)