Skip to content

Commit 682f24f

Browse files
authored
fix(opensearch): give the securityconfig update job its own resources (#4228)
## What this PR does The chart rendered `spec.security.config` without `updateJob`, and the operator passes that field straight through to the Job container it builds for `securityadmin`. With nothing set, a tenant namespace applies its own `tenant-range-limits` LimitRange default of 128Mi instead. `securityadmin.sh` runs `java` with no `-Xmx` of its own, so the JVM sizes its heap from the container limit and does not fit in 128Mi. The operator builds the Job with `backoffLimit: 0`, so the single OOM kill is final. Nothing writes the `.opendistro_security` index after that, every node sits behind `Not yet initialized (you may need to run securityadmin)`, and the release never reaches ready. This is what has been failing the `opensearch` chainsaw suite in every run that executes it since 2026-09-11. The suite waits 12 minutes for `status.readyReplicas` on `opensearch-test-nodes`, which is the whole 748s runtime of the red test. Full write-up in #4227. ### Where the numbers come from Measured against `opensearchproject/opensearch:2.11.1` on `linux/amd64`, running the operator's own command against a fresh single-node cluster: | Memory limit | Result | Peak | | --- | --- | --- | | 128Mi | OOMKilled mid-apply | 128Mi | | 192Mi | OOMKilled mid-apply | 192Mi | | 256Mi | success | 208Mi | | 512Mi | success | 253Mi | Having no `-Xmx`, the tool grows into whatever it is given, so the peak tracks the limit and the headroom is what matters rather than the peak itself. The values are constants and not a `resourcesPreset` lookup: the tool configures the cluster once and exits, so what it needs does not scale with the nodes it configures, and the smallest preset would hand it back the 128Mi it dies at. ### Upgrading an already-stuck release This does not unstick one on its own. The operator skips the Job entirely while the existing one carries a matching securityconfig checksum, and bumping the chart does not change that checksum, so the failed Job stays where it is. Delete it once and the operator rebuilds it with the resources above: ```bash kubectl -n <namespace> delete job <release>-securityconfig-update ``` A fresh install needs nothing. ### Not fixed here `backoffLimit: 0` is the operator's, and it is what turns one transient kill into a permanent dead cluster. Worth raising upstream separately. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch): stop the securityconfig update job being OOMKilled at the tenant namespace memory default, which left new clusters stuck with security uninitialized and every node unready ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * OpenSearch security configuration updates now use consistent CPU and memory allocations, regardless of the selected node sizing preset. * Improved reliability for update jobs by ensuring they receive defined minimum and maximum resource levels. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents bed172b + 4198914 commit 682f24f

2 files changed

Lines changed: 75 additions & 0 deletions

File tree

‎packages/apps/opensearch/templates/opensearch.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,34 @@ spec:
7575
name: {{ .Release.Name }}-security-config
7676
adminCredentialsSecret:
7777
name: {{ .Release.Name }}-admin-credentials
78+
{{- /*
79+
securityadmin.sh runs java with no -Xmx of its own, so the JVM sizes its
80+
heap from whatever the container is allowed. With nothing set here the
81+
tenant LimitRange supplies the namespace default instead, and the operator
82+
builds this Job with backoffLimit 0, so the single OOM kill that follows is
83+
final: the security index is never written and every node sits behind
84+
"Not yet initialized (you may need to run securityadmin)" until the release
85+
is deleted.
86+
87+
Measured against opensearchproject/opensearch:2.11.1 on linux/amd64,
88+
applying a security config to a fresh single-node cluster: OOMKilled
89+
mid-apply at 128Mi and at 192Mi, completes from 256Mi up. Having no -Xmx it
90+
grows into what it is given, peaking at 208Mi under a 256Mi limit and at
91+
253Mi under the 512Mi one below, so the headroom here is the point rather
92+
than the peak.
93+
94+
Constants rather than the preset: the tool configures the cluster once and
95+
exits, so what it needs does not grow with the nodes it configures, and the
96+
smallest preset would hand it back the 128Mi it dies at.
97+
*/}}
98+
updateJob:
99+
resources:
100+
requests:
101+
cpu: 100m
102+
memory: 256Mi
103+
limits:
104+
cpu: "1"
105+
memory: 512Mi
78106
{{- if .Values.dashboards.enabled }}
79107
dashboards:
80108
enable: true

‎packages/apps/opensearch/tests/opensearch_test.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -619,3 +619,50 @@ tests:
619619
path: spec.selector["opster.io/opensearch-cluster"]
620620
value: mydb
621621
documentIndex: 1
622+
623+
###################################
624+
# securityconfig update job #
625+
###################################
626+
627+
- it: gives the securityconfig update job its own resources
628+
release:
629+
name: test-os
630+
namespace: tenant-test
631+
set:
632+
_cluster:
633+
cluster-domain: cozy.local
634+
asserts:
635+
- equal:
636+
path: spec.security.config.updateJob.resources.requests.memory
637+
value: 256Mi
638+
documentIndex: 0
639+
- equal:
640+
path: spec.security.config.updateJob.resources.limits.memory
641+
value: 512Mi
642+
documentIndex: 0
643+
- equal:
644+
path: spec.security.config.updateJob.resources.requests.cpu
645+
value: 100m
646+
documentIndex: 0
647+
- equal:
648+
path: spec.security.config.updateJob.resources.limits.cpu
649+
value: "1"
650+
documentIndex: 0
651+
652+
# securityadmin configures the cluster once and exits, so what it needs does
653+
# not grow with the cluster. Tying it to the preset would hand it 128Mi again
654+
# on the smallest one, which is the amount it is OOMKilled at.
655+
- it: keeps the update job resources off the node sizing preset
656+
release:
657+
name: test-os
658+
namespace: tenant-test
659+
set:
660+
_cluster:
661+
cluster-domain: cozy.local
662+
resourcesPreset: "nano"
663+
resources: {}
664+
asserts:
665+
- equal:
666+
path: spec.security.config.updateJob.resources.limits.memory
667+
value: 512Mi
668+
documentIndex: 0

0 commit comments

Comments
 (0)