e2e: harbor suite flakes when rfs-harbor-test-redis Sentinel HA never reaches readiness #3445

Activity

1. Aleksei Sviridkin (lexfrei)

   added

   kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bug

   area/ciIssues or PRs related to CI workflows, GitHub Actions, automationIssues or PRs related to CI workflows, GitHub Actions, automation

   area/testingIssues or PRs related to testing (e2e, bats, unit tests)Issues or PRs related to testing (e2e, bats, unit tests)

   on Jul 23, 2026
2. Aleksei Sviridkin (lexfrei)

   closed this as completedin #3446on Jul 28, 2026
3. Aleksei Sviridkin (lexfrei)

   added a commit that references this issue on Jul 28, 2026

   fix(harbor): guarantee CPU for redis sentinels (#3446)

   ...

   This issue will close once commit bea9a2b is merged into the 'main' branch.

   bea9a2b
4. lexfrei commented on Jul 29, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Jul 29, 2026

   ContributorAuthor

   More actions

   This flaked again twice after the fix landed, on runs whose base already carried it: job 90333030340 (2026-07-28 16:34:43 UTC) and job 90385337598 (2026-07-28 21:26:45 UTC). Same step, same body, Ready: "Unknown" on the harbor-test HelmRelease.

   I pulled the duration of the wait-helmrelease-ready assert out of 20 runs. Sixteen healthy ones span 1m42s to 3m12s. The failures are 0m25s and 0m27s on one side and 10m00s and 9m59s on the other. Nothing at all lands between 3m12s and 10m.

   That says the budget was never the binding constraint. The healthy maximum of 3m12s is 53% of the old 6m and 32% of the 10m I raised it to, so the value I changed had not limited a healthy run even once. The two short failures are a separate thing: an assert whose poll was cut short by a transient API error rather than a slow release (upstream behaviour, kyverno/chainsaw#2758).

   The catch output adds one more fact: six minutes after the deadline expired, the dataplane capture reports no scheduled NotReady pods at all. The release does converge, somewhere between 10m and 16m, it just does not converge inside the window.

   What is missing to take this further is a snapshot of the RedisFailover and its sentinels taken at the moment the deadline expires. The catch collects a host snapshot, which does not detail them, so a rare slow sentinel bootstrap and a release that stalls and then recovers on its own look identical in what we keep today.
5. Aleksei Sviridkin (lexfrei)

   reopened this on Jul 29, 2026
6. lexfrei commented on Aug 7, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Aug 7, 2026

   ContributorAuthor

   More actions

   A fresh occurrence contradicts the diagnosis in this issue, so posting what the artifacts of run 31173212908 (job 92852286215, 2026-08-07) actually show before anyone goes digging in the spotahome operator.

   The sentinels are fine there. All three rfs-harbor-test-redis pods are 1/1 Running, each logs +monitor master mymaster ... quorum 2 at 12:10:07 and sees the two others by 12:10:09. Quorum formed 85 seconds after the pods were created. "All three Sentinel replicas fail their readiness probe" does not hold on this run.

   What actually eats the budget is an uninstall-remediation cycle. First install: harbor-core waits for redis, redis waits for its PVC (linstor provisioning) and the master election, and harbor-test-nginx hits the Deployment default progressDeadlineSeconds of 600s. Flux reads that kstatus Failed as fail-early (failed early due to stalled resources), and with install.remediation.retries: -1 (packages/apps/harbor/templates/harbor.yaml:92-94) it remediates by uninstall, which deletes everything including the redis PVC. The second install starts from scratch at 12:08:47, converges its sentinels by 12:10:09, and runs out of the chainsaw assert budget at 12:11:40 with harbor-core still in its startup probe.

   The arithmetic makes this a deterministic fork, not a busy-runner flake: the chainsaw wait budget is 10m and progressDeadlineSeconds is 10m, so a single remediation cycle consumes the whole test budget by construction. If the first install beats the deadline the test is green; if it does not, the test is lost no matter how fast the second attempt converges.

   This is the install-remediation-by-uninstall family, the same shape the tenant csi and cni HelmReleases had before their retry fixes. The fix probably belongs there (retry in place instead of uninstall, or ordering redis before harbor-core, or a deadline that actually fits), not in the redis operator.

   One verification limit: the July run this issue was filed from cannot be re-checked, its job log is expired and the report snapshot is taken after chainsaw wipes the namespace, so the harbor pods are already gone in it. The catch-block log is currently the only record of this mechanism, and it dies with log retention.
7. Aleksei Sviridkin (lexfrei)

   added a commit that references this issue on Aug 8, 2026

   fix(harbor): retry the nested system install instead of uninstalling …

   ...

   79ceaf1
8. lexfrei commented on Aug 10, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Aug 10, 2026

   ContributorAuthor

   More actions

   Not sentinels, and not the uninstall cycle from my last comment. Both recent harbor reds are the pod-IP collision from #3615 and #3736.

   Runs 31338950312 (job 93311099197) and 31338954173 (job 93313889744), both 2026-08-09. In each one a harbor pod got an address the node already holds on cilium_host, so kubelet's probe lands on the node's own stack and answers connection refused for as long as the pod lives.

   harbor-test-nginx-c69459d47-rnkjs on srv3, first run:

   === host netns: ip route get 10.244.0.197 ===
   local 10.244.0.197 dev lo table local src 10.244.0.197
   
   === host netns: ip -o addr show ===
   10: cilium_host    inet 10.244.0.197/32 scope global cilium_host
   

   harbor-test-db-1 on srv2, second run:

   === host netns: ip route get 10.244.1.94 ===
   local 10.244.1.94 dev lo table local src 10.244.1.94
   
   === host netns: ip -o addr show ===
   10: cilium_host    inet 10.244.1.94/32 scope global cilium_host
   

   The second run shows it is not a dead workload. kubelet on srv2 gets connection refused on https://10.244.1.94:8000/startupz while cilium conntrack on that same node has other pods talking to the address:

   TCP OUT 10.244.0.58:58282 -> 10.244.1.94:8000  RxFlagsSeen=0x1a TxFlagsSeen=0x1a [ SeenNonSyn ]
   TCP IN  10.244.0.83:51088 -> 10.244.1.94:9187  RxFlagsSeen=0x1a TxFlagsSeen=0x1a [ SeenNonSyn ]
   

   Pod to pod works, host to pod does not.

   Everything else follows from that one pod. In the first run every harbor pod is Ready at the deadline except nginx, and the release dies on failed early due to stalled resources: [Deployment/tenant-test/harbor-test-nginx status: 'Failed']. In the second the unreachable pod is postgres, with core, jobservice and exporter crash-looping behind it at 5, 6 and 5 restarts.

   Sentinels are healthy in both. All three rfs-harbor-test-redis-* pods are Ready with zero restarts at the deadline in each run, and in the first one core logs sentinel: new master="mymaster" addr="10.244.0.204:6379" 83 seconds after the RedisFailover was created. #3621 works too: harbor-test-system carries a single install in its history in the first run, First Deployed equal to Last Deployed, and nothing is torn down in either. Bucket is bucketReady: true in the first run, so not the hairpin either.

   Same night, same batch of eight concurrent runs, three of the four suite failures were this bug. Harbor twice in the quiet form above, mongodb once in the loud one with IP ipv4:10.244.1.12 is already in use. Harbor is not special, it just has the most pods and draws the unlucky address most often.

   The attribution only became readable that day. #3617 merged at 12:39 UTC and these runs started at 22:15, so they are among the first to record which interface holds the address. #3615 asked that question and the answer is cilium_host.

   I would close this one in favour of #3736.
9. Aleksei Sviridkin (lexfrei)

   added a commit that references this issue on Aug 11, 2026

   fix(e2e): stop the sandbox cluster CIDR overlapping the pod range (#3750

   ...

   4ac02f5
10. github-actions

    added

    triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

    on Aug 12, 2026
11. Aleksei Sviridkin (lexfrei)

    added

    kind/flakeCategorizes issue or PR as related to a flaky testCategorizes issue or PR as related to a flaky test

    triage/duplicateIndicates an issue is a duplicate of another issueIndicates an issue is a duplicate of another issue

    and removed

    triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

    on Sep 23, 2026
12. lexfrei commented on Sep 23, 2026

    

    Aleksei Sviridkin (lexfrei)

    on Sep 23, 2026

    ContributorAuthor

    More actions

    Duplicate of #3736. The later runs showed healthy sentinels, and harbor was failing because of a pod IP already held by cilium_host, which #3750 fixed. If harbor fails again with really unready sentinels, please open a new issue with the run artifacts.
13. Aleksei Sviridkin (lexfrei)

    closed this as not plannedon Sep 23, 2026
14. myasnikovdaniil

    added a commit that references this issue on Oct 26, 2026

    fix(harbor): retry the nested system install instead of uninstalling …

    ...

    831b670