You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bea9a2b
Browse filesBrowse the repository at this point in the historyBrowse files
fix(harbor): guarantee CPU for redis sentinels (#3446)
## What this PR does
The harbor e2e suite intermittently times out on the `harbor-test`
HelmRelease because the bundled Redis Sentinel set does not converge in
time. Sentinels are NotReady by design until the redis-operator elects a
master and rewrites their monitored-master config. Normal convergence is
2-3 minutes; on a CPU-contended e2e runner the tail stretches past the
old 6m assert.
Three changes:
- `packages/system/harbor`: the inline RedisFailover ran its three
sentinels at 50m CPU request / 100m limit. Now they run at Guaranteed
QoS (requests == limits, 250m CPU / 128Mi), so master election gets
scheduling shares and is not throttled under node contention. A helm
unit test pins the sizing. This reserves 750m CPU / 384Mi per Harbor
instance for the sentinel trio, up from 150m / 192Mi requested, paid by
every install. It also diverges from the standalone redis app on
purpose: its preset-based sentinels are Burstable with a 25m CPU
request, and matching that would lower the request and make contention
worse.
- harbor e2e suite: on failure the catch now collects the redis-operator
log, a describe of the RedisFailover, and the rfr-/rfs- pod states and
logs. None of these were collected before, so a non-converging sentinel
set was undiagnosable from the artifacts. The `harbor-test` HelmRelease
wait grows from 6m to 10m, same as the `harbor-test-system` budget in
the same suite and the documented longer-wait allowance for harbor.
- docs and hack: three copies of a wrong claim that the pinned Chainsaw
podLogs collector "exposes only container/tail/selector" are corrected.
It accepts name/namespace/selector/container/tail/timeout plus the
cluster fields; only `previous` is missing. The wrong list steered catch
authors toward imperative scripts for captures podLogs handles
declaratively.
Caveat: the failing run carried no CPU metrics, so the resource fix is
mechanism-backed (CFS shares cycles proportionally to requests), not
metric-proven. If the flake recurs, the new diagnostics show which
reconcile gate stalled.
Closes#3445
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
Trigger-map walk against the diff: `packages/system/harbor/` (template
and tests) is not in the website app lists (those cover `packages/apps/`
and `packages/extra/`) and touches no `values.schema.json`, no values
keys, no ApplicationDefinition. `hack/e2e-chainsaw/` and
`hack/e2e-capture-previous-logs.sh` appear in no trigger.
`docs/agents/e2e-testing.md` is repo-internal agent documentation. Node
prerequisites, `hack/package.mk`, and namespaces are untouched.
### Release note
```release-note
fix(harbor): the bundled Redis Sentinel pods now run with guaranteed resources (250m CPU / 128Mi, requests == limits) so the HA set converges reliably on CPU-contended nodes
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Increased Redis Sentinel CPU and memory requests/limits to improve
reliability during node contention.
* Extended Harbor readiness timing for RedisFailover to reduce false
failures during initialization.
* **Diagnostics**
* Improved Harbor e2e failure output by adding RedisFailover-specific
details and collecting relevant Redis operator/pod logs, plus
best-effort runtime log capture during readiness stalls.
* **Tests**
* Added a Redis Sentinel sizing test suite to verify the updated
request/limit values.
* **Documentation**
* Clarified guidance on capturing logs for previous container instances
in e2e runs, including what gets captured and when.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -89,7 +89,7 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
89
89
- On E2E failure, the CI step dumps `kubectl get hr -A`, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain `get` shows only NAME/AGE). The `chainsaw test` run is one step, so this dump is at run level; Chainsaw's own per-`Test` `catch` blocks add scoped events/describe/podLogs for the failing suite.
90
90
- Inside a suite, attach scoped diagnostics with a top-level `catch:` block (`events`, `describe`, `podLogs`, `get`) rather than letting a bare assert failure through — see `hack/e2e-chainsaw/bucket/chainsaw-test.yaml` and `hack/e2e-chainsaw/harbor/chainsaw-test.yaml`. This replaces the BATS `... || { echo …; kubectl get/describe …; false; }` idiom.
91
91
- `make collect-report`→ `hack/cozyreport.sh` uploads a full `cozyreport.tgz` (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (`chainsaw-report.xml`) is uploaded as its own artifact.
92
-
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 exposes only `container`/`tail`/`selector` and has no `previous` option) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the BATS tests) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster.
92
+
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 has no `previous` option — it does accept `name`/`namespace`/`selector`/`container`/`tail`/`timeout`/`cluster`/`clusters`) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the BATS tests) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster.
93
93
- **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to be killed — it reports `SCRIPT | ERROR | signal: killed` and yields nothing. That kill is strictly worse than an inner `timeout` firing: SIGKILL takes out the process group, so the collector dies mid-write **and** every later step in the script is skipped, whereas an inner `timeout` exits 124, leaves a partial capture on disk, and lets the rest of the catch run. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
94
94
- **Set `crust-gather --duration` explicitly.** It is crust-gather's own budget for the collection phase and it defaults to **60s**; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs *first* is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than `/dev/null` so "complete or truncated?" is answerable from the artifact.
95
95
- **Per-failed-test crust-gather snapshots** are the richest diagnostic — a `crust-gather serve`-able archive of the cluster *at the moment of failure*, before cleanup. The global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` captures the **host** cluster into `_out/cozyreport/snapshots/<test>/host` on any failure (the Chainsaw analog of the `hack/cozytest.sh` EXIT-trap snapshot used by the BATS tests); the kubernetes suites additionally snapshot their **tenant** cluster from `hack/e2e-chainsaw/_lib/run-kubernetes.sh` while the tenant API LB is still routable. `hack/cozyreport.sh` folds `snapshots/` into the artifact. Do not remove the global catch — without it the Chainsaw suite uploads a report with no per-test cluster state.
0 commit comments