Environment

Device and OS: Ubuntu GitHub Actions runner
App version: N/A
Kubernetes distro being used: N/A

Steps to reproduce

1. Push a commit that changed a file not on the ignore list to a PR to trigger running the E2E tests w/ the data injection test. The change should not change any behavior to ensure accurate reproduction.

Expected result

The data injection test passes

Actual Result

The data injection test fails/flakes occasionally

Visual Proof (screenshots, videos, text, etc)

https://github.com/defenseunicorns/zarf/actions/runs/7892004713/job/21537657159#step:8:5982

Additional Context

The data injection test performs a data injection 3 times back to back to ensure idempotence. Each data injection causes the kiwix pod to cycle. Immediately after the third and final data injection is ran, we attempt to fetch the pod logs. There are times when there is still a pod in a terminating state when fetching the logs. Because we are currently using the --selector=app=kiwix-serve flag when fetching logs, the terminating pod still matches against the selector query, resulting in the test failing.

https://github.com/defenseunicorns/zarf/blob/21ccaaa26d981e9c24fafce31c871c0ed2498da9/src/test/e2e/23_data_injection_test.go#L30-L36

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase

Since Kubernetes 1.27, the kubelet transitions deleted Pods, except for static Pods and force-deleted Pods without a finalizer, to a terminal phase (Failed or Succeeded depending on the exit statuses of the pod containers) before their deletion from the API server.