Repository navigation
Commit 682f24f
authored
fix(opensearch): give the securityconfig update job its own resources (#4228)
## What this PR does
The chart rendered `spec.security.config` without `updateJob`, and the
operator passes that field straight through to the Job container it
builds for `securityadmin`. With nothing set, a tenant namespace applies
its own `tenant-range-limits` LimitRange default of 128Mi instead.
`securityadmin.sh` runs `java` with no `-Xmx` of its own, so the JVM
sizes its heap from the container limit and does not fit in 128Mi. The
operator builds the Job with `backoffLimit: 0`, so the single OOM kill
is final. Nothing writes the `.opendistro_security` index after that,
every node sits behind `Not yet initialized (you may need to run
securityadmin)`, and the release never reaches ready.
This is what has been failing the `opensearch` chainsaw suite in every
run that executes it since 2026-09-11. The suite waits 12 minutes for
`status.readyReplicas` on `opensearch-test-nodes`, which is the whole
748s runtime of the red test. Full write-up in #4227.
### Where the numbers come from
Measured against `opensearchproject/opensearch:2.11.1` on `linux/amd64`,
running the operator's own command against a fresh single-node cluster:
| Memory limit | Result | Peak |
| --- | --- | --- |
| 128Mi | OOMKilled mid-apply | 128Mi |
| 192Mi | OOMKilled mid-apply | 192Mi |
| 256Mi | success | 208Mi |
| 512Mi | success | 253Mi |
Having no `-Xmx`, the tool grows into whatever it is given, so the peak
tracks the limit and the headroom is what matters rather than the peak
itself. The values are constants and not a `resourcesPreset` lookup: the
tool configures the cluster once and exits, so what it needs does not
scale with the nodes it configures, and the smallest preset would hand
it back the 128Mi it dies at.
### Upgrading an already-stuck release
This does not unstick one on its own. The operator skips the Job
entirely while the existing one carries a matching securityconfig
checksum, and bumping the chart does not change that checksum, so the
failed Job stays where it is. Delete it once and the operator rebuilds
it with the resources above:
```bash
kubectl -n <namespace> delete job <release>-securityconfig-update
```
A fresh install needs nothing.
### Not fixed here
`backoffLimit: 0` is the operator's, and it is what turns one transient
kill into a permanent dead cluster. Worth raising upstream separately.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(opensearch): stop the securityconfig update job being OOMKilled at the tenant namespace memory default, which left new clusters stuck with security uninitialized and every node unready
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* OpenSearch security configuration updates now use consistent CPU and
memory allocations, regardless of the selected node sizing preset.
* Improved reliability for update jobs by ensuring they receive defined
minimum and maximum resource levels.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->2 files changed
Lines changed: 75 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
78 | 106 | | |
79 | 107 | | |
80 | 108 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
619 | 619 | | |
620 | 620 | | |
621 | 621 | | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
0 commit comments