Skip to content

Commit bea9a2b

Browse files
authored
fix(harbor): guarantee CPU for redis sentinels (#3446)
## What this PR does The harbor e2e suite intermittently times out on the `harbor-test` HelmRelease because the bundled Redis Sentinel set does not converge in time. Sentinels are NotReady by design until the redis-operator elects a master and rewrites their monitored-master config. Normal convergence is 2-3 minutes; on a CPU-contended e2e runner the tail stretches past the old 6m assert. Three changes: - `packages/system/harbor`: the inline RedisFailover ran its three sentinels at 50m CPU request / 100m limit. Now they run at Guaranteed QoS (requests == limits, 250m CPU / 128Mi), so master election gets scheduling shares and is not throttled under node contention. A helm unit test pins the sizing. This reserves 750m CPU / 384Mi per Harbor instance for the sentinel trio, up from 150m / 192Mi requested, paid by every install. It also diverges from the standalone redis app on purpose: its preset-based sentinels are Burstable with a 25m CPU request, and matching that would lower the request and make contention worse. - harbor e2e suite: on failure the catch now collects the redis-operator log, a describe of the RedisFailover, and the rfr-/rfs- pod states and logs. None of these were collected before, so a non-converging sentinel set was undiagnosable from the artifacts. The `harbor-test` HelmRelease wait grows from 6m to 10m, same as the `harbor-test-system` budget in the same suite and the documented longer-wait allowance for harbor. - docs and hack: three copies of a wrong claim that the pinned Chainsaw podLogs collector "exposes only container/tail/selector" are corrected. It accepts name/namespace/selector/container/tail/timeout plus the cluster fields; only `previous` is missing. The wrong list steered catch authors toward imperative scripts for captures podLogs handles declaratively. Caveat: the failing run carried no CPU metrics, so the resource fix is mechanism-backed (CFS shares cycles proportionally to requests), not metric-proven. If the flake recurs, the new diagnostics show which reconcile gate stalled. Closes #3445 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Trigger-map walk against the diff: `packages/system/harbor/` (template and tests) is not in the website app lists (those cover `packages/apps/` and `packages/extra/`) and touches no `values.schema.json`, no values keys, no ApplicationDefinition. `hack/e2e-chainsaw/` and `hack/e2e-capture-previous-logs.sh` appear in no trigger. `docs/agents/e2e-testing.md` is repo-internal agent documentation. Node prerequisites, `hack/package.mk`, and namespaces are untouched. ### Release note ```release-note fix(harbor): the bundled Redis Sentinel pods now run with guaranteed resources (250m CPU / 128Mi, requests == limits) so the HA set converges reliably on CPU-contended nodes ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased Redis Sentinel CPU and memory requests/limits to improve reliability during node contention. * Extended Harbor readiness timing for RedisFailover to reduce false failures during initialization. * **Diagnostics** * Improved Harbor e2e failure output by adding RedisFailover-specific details and collecting relevant Redis operator/pod logs, plus best-effort runtime log capture during readiness stalls. * **Tests** * Added a Redis Sentinel sizing test suite to verify the updated request/limit values. * **Documentation** * Clarified guidance on capturing logs for previous container instances in e2e runs, including what gets captured and when. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 8bffa34 + a7ce437 commit bea9a2b

6 files changed

Lines changed: 80 additions & 10 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
8989
- On E2E failure, the CI step dumps `kubectl get hr -A`, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain `get` shows only NAME/AGE). The `chainsaw test` run is one step, so this dump is at run level; Chainsaw's own per-`Test` `catch` blocks add scoped events/describe/podLogs for the failing suite.
9090
- Inside a suite, attach scoped diagnostics with a top-level `catch:` block (`events`, `describe`, `podLogs`, `get`) rather than letting a bare assert failure through — see `hack/e2e-chainsaw/bucket/chainsaw-test.yaml` and `hack/e2e-chainsaw/harbor/chainsaw-test.yaml`. This replaces the BATS `... || { echo …; kubectl get/describe …; false; }` idiom.
9191
- `make collect-report` → `hack/cozyreport.sh` uploads a full `cozyreport.tgz` (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (`chainsaw-report.xml`) is uploaded as its own artifact.
92-
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 exposes only `container`/`tail`/`selector` and has no `previous` option) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the BATS tests) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster.
92+
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 has no `previous` option — it does accept `name`/`namespace`/`selector`/`container`/`tail`/`timeout`/`cluster`/`clusters`) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the BATS tests) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster.
9393
- **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to be killed — it reports `SCRIPT | ERROR | signal: killed` and yields nothing. That kill is strictly worse than an inner `timeout` firing: SIGKILL takes out the process group, so the collector dies mid-write **and** every later step in the script is skipped, whereas an inner `timeout` exits 124, leaves a partial capture on disk, and lets the rest of the catch run. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
9494
- **Set `crust-gather --duration` explicitly.** It is crust-gather's own budget for the collection phase and it defaults to **60s**; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs *first* is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than `/dev/null` so "complete or truncated?" is answerable from the artifact.
9595
- **Per-failed-test crust-gather snapshots** are the richest diagnostic — a `crust-gather serve`-able archive of the cluster *at the moment of failure*, before cleanup. The global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` captures the **host** cluster into `_out/cozyreport/snapshots/<test>/host` on any failure (the Chainsaw analog of the `hack/cozytest.sh` EXIT-trap snapshot used by the BATS tests); the kubernetes suites additionally snapshot their **tenant** cluster from `hack/e2e-chainsaw/_lib/run-kubernetes.sh` while the tenant API LB is still routable. `hack/cozyreport.sh` folds `snapshots/` into the artifact. Do not remove the global catch — without it the Chainsaw suite uploads a report with no per-test cluster state.

‎hack/e2e-capture-previous-logs.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@
1616
# dead container the evidence is gone for good.
1717
#
1818
# The Chainsaw suites capture current logs through the built-in `podLogs`
19-
# collector, which has no `previous` option (chainsaw v0.2.15 PodLogs exposes
20-
# only container/tail/selector), so this closes that gap for every suite at once
19+
# collector, which has no `previous` option in chainsaw v0.2.15, so this
20+
# closes that gap for every suite at once
2121
# from the global catch rather than per-suite. The crust-gather snapshot does
2222
# collect previous.log per container, but that is a whole archive to download
2323
# and `crust-gather serve`; this puts the decisive lines directly in the failed

‎hack/e2e-chainsaw/.chainsaw.yaml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -93,9 +93,8 @@ spec:
9393
# Nth restart — a fresh replay of the same startup path — while the
9494
# evidence that explains the failure sits in the immediately preceding
9595
# instance, reachable only via `--previous`. Chainsaw v0.2.15's podLogs
96-
# collector has no `previous` option (it exposes container/tail/
97-
# selector only), so this closes the gap once for every suite here
98-
# instead of per-suite. crust-gather does archive a previous.log per
96+
# collector has no `previous` option, so this closes the gap once for
97+
# every suite here instead of per-suite. crust-gather does archive a previous.log per
9998
# container, but that is a whole archive to fetch and serve; this puts
10099
# the decisive lines straight into the failed job's log, and keeps
101100
# them even when the snapshot below is truncated.

‎hack/e2e-chainsaw/harbor/chainsaw-test.yaml‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,38 @@ spec:
3737
- podLogs:
3838
selector: app.kubernetes.io/instance=harbor-test
3939
tail: 50
40+
# harbor-test readiness is gated by the inline RedisFailover
41+
# harbor-test-redis: its three sentinels stay NotReady until the spotahome
42+
# redis-operator elects a master and rewrites their monitored-master config.
43+
# None of the collectors above see that path: the harbor-test instance
44+
# selector matches neither the operator (namespace cozy-redis-operator) nor
45+
# the rfr-/rfs- pods, and the RedisFailover CRD ships no status subresource,
46+
# so there is nothing to assert on declaratively.
47+
- describe:
48+
apiVersion: databases.spotahome.com/v1
49+
kind: RedisFailover
50+
name: harbor-test-redis
51+
- podLogs:
52+
namespace: cozy-redis-operator
53+
selector: app.kubernetes.io/name=redis-operator
54+
tail: 200
55+
- script:
56+
# A script rather than more podLogs entries: the -o wide pod tables and
57+
# the rfr-/rfs- name-prefix match (independent of the operator's
58+
# pod-label scheme) are not expressible declaratively. Every command is
59+
# best-effort (|| true) so diagnostics never mask the test's own error;
60+
# no inner timeout wrappers, so the 2m op timeout is the only budget.
61+
timeout: 2m
62+
content: |
63+
set -u
64+
echo "== redis-operator pods (cozy-redis-operator) =="
65+
kubectl -n cozy-redis-operator get pods -o wide || true
66+
echo "== rfr-/rfs- pod states ($NAMESPACE) =="
67+
kubectl -n "$NAMESPACE" get pods -o wide | grep -E 'rf[rs]-harbor-test-redis' || true
68+
for p in $(kubectl -n "$NAMESPACE" get pods -o name | grep -E 'rf[rs]-harbor-test-redis'); do
69+
echo "== logs $p =="
70+
kubectl -n "$NAMESPACE" logs "$p" --all-containers --tail=100 --prefix || true
71+
done
4072
steps:
4173
- name: create-harbor
4274
try:
@@ -45,8 +77,16 @@ spec:
4577

4678
- name: wait-helmrelease-ready
4779
try:
80+
# 10m, not the standard 5-6m: harbor-test readiness is gated by the inline
81+
# RedisFailover, whose three sentinels only go Ready after the spotahome
82+
# redis-operator elects a master and rewrites their config. Normal
83+
# convergence is 2-3m, but on a CPU-committed runner the tail runs long
84+
# (steal-induced probe misses, documented in pull-requests.yaml). Matches
85+
# the harbor-test-system budget below and the justified-longer-wait
86+
# allowance in docs/agents/e2e-testing.md. If this still times out, the
87+
# catch block's redis-operator log shows which reconcile gate stalled.
4888
- assert:
49-
timeout: 6m
89+
timeout: 10m
5090
resource:
5191
apiVersion: helm.toolkit.fluxcd.io/v2
5292
kind: HelmRelease

‎packages/system/harbor/templates/redis.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@ spec:
1818
replicas: 3
1919
resources:
2020
limits:
21-
cpu: 100m
21+
cpu: 250m
2222
memory: 128Mi
2323
requests:
24-
cpu: 50m
25-
memory: 64Mi
24+
cpu: 250m
25+
memory: 128Mi
2626
redis:
2727
replicas: {{ .Values.redis.replicas }}
2828
resources:
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
suite: redis sentinel sizing
2+
3+
release:
4+
name: harbor
5+
namespace: tenant-root
6+
7+
# templates/redis.yaml renders two documents: doc 0 is the redis-auth Secret,
8+
# doc 1 is the RedisFailover. The sentinels must run at Guaranteed QoS
9+
# (requests == limits): guaranteed CPU shares under node contention and top
10+
# eviction priority. A starved sentinel delays master election and stalls
11+
# harbor convergence.
12+
tests:
13+
- it: sentinel is sized for contended-node convergence (Guaranteed QoS)
14+
template: templates/redis.yaml
15+
documentIndex: 1
16+
set:
17+
harbor:
18+
fullnameOverride: harbor-test
19+
asserts:
20+
- equal:
21+
path: spec.sentinel.resources.requests.cpu
22+
value: 250m
23+
- equal:
24+
path: spec.sentinel.resources.limits.cpu
25+
value: 250m
26+
- equal:
27+
path: spec.sentinel.resources.requests.memory
28+
value: 128Mi
29+
- equal:
30+
path: spec.sentinel.resources.limits.memory
31+
value: 128Mi

0 commit comments

Comments
 (0)