Repository navigation
Description
What happens
Deploying the OpenSearch application into a tenant namespace never starts — the opensearch-*-nodes StatefulSet cannot create its pod:
FailedCreate: pods "opensearch-...-nodes-0" is forbidden:
violates PodSecurity "baseline:latest":
privileged (container "init-sysctl" must not set securityContext.privileged=true)
The OpenSearch CR template hardcodes spec.general.setVMMaxMapCount: true (packages/apps/opensearch/templates/opensearch.yaml). The opster operator then injects a privileged init-sysctl initContainer to raise vm.max_map_count. Tenant namespaces enforce PodSecurity baseline by default (no pod-security.kubernetes.io/enforce label on the namespace, so the cluster-wide PSA default applies), which forbids privileged containers.
Result: zero OpenSearch master/data pods. The *-external / *-dashboards-external LoadBalancer services get IPs but their EndpointSlices stay empty, so OpenSearch and Dashboards are unreachable (a plain telnet to the LB port hangs on "Trying"). The OpenSearchCluster CR misleadingly reports Ready: True / PHASE: RUNNING because only the non-privileged securityconfig job runs.
This is an internal contradiction: Cozystack enforces baseline PSA on tenant namespaces but ships an app that requires a privileged initContainer.
Reproduction
On a clean Cozystack 1.4.2 cluster (Kubernetes 1.34), create an OpenSearch app in any tenant namespace:
apiVersion: apps.cozystack.io/v1alpha1
kind: OpenSearch
metadata:
name: repro-psa
namespace: tenant-demo
spec:
replicas: 1
size: 2Gi
resourcesPreset: c1.small
dashboards:
enabled: trueThe opensearch-repro-psa-nodes StatefulSet immediately logs the FailedCreate above and never creates a pod.
Environment
- Reproduced on Cozystack v1.4.2 and v1.4.4
- OpenSearch app (opster operator), default tenant namespace
- Nodes have
vm.max_map_count = 65530(Linux default), so the sysctl init genuinely is needed — it cannot simply be disabled without setting the sysctl elsewhere
Possible fixes
- Set
vm.max_map_count=262144at the node level (Talos sysctl via the Cozystack node config) and setsetVMMaxMapCount: falsein the package — removes the privileged init entirely. - Or run the OpenSearch app in a namespace permitted to run privileged.
Workaround (verified)
Raise the tenant namespace PSA to privileged (weakens tenant isolation, per-namespace):
kubectl label ns tenant-demo pod-security.kubernetes.io/enforce=privileged --overwriteAfter labelling, the node pod passes Init (the privileged init-sysctl runs), the PVC binds, and the cluster comes up. The label is not reconciled away by the tenant chart (it does not template any pod-security.kubernetes.io/* label).
Activity
yankawai commentedon Sep 2, 2026
Hit this on Cozystack v1.6.x (platform 54) — same failure verbatim, so the report still reproduces two and a half months on.
Confirmed all three links on a live cluster: nodes sit at
vm.max_map_count = 65530with novm.*entry in the machine config, the chart hardcodessetVMMaxMapCount: trueatpackages/apps/opensearch/templates/opensearch.yaml:16, and the tenant namespace carries nopod-security.kubernetes.io/*label, so the cluster-wideenforce: baselineapplies.Opened #4026 for the half that belongs to the chart:
setVMMaxMapCountbecomes a values knob, defaulttrue, so an operator who has already raised the sysctl at node level can stop the privileged initContainer from being injected. The default deliberately does not change — the sysctl is node-level rather than namespaced, so it cannot be set through a PodsecurityContext.sysctls, and flipping it would trade an admission failure for a bootstrap-check crash on any cluster that has not set it.The other half is the platform's:
packages/core/talosbuilds images only and ships no node sysctls, so nothing in-tree setsvm.max_map_count=262144. Happy to follow up with that if the maintainers agree it belongs in the platform rather than in operator documentation.myasnikovdaniil commentedon Nov 6, 2026
Tenant OpenSearch pods don't request the privileged sysctl init container anymore (#2682), node level sysctl is handled by the system DaemonSet from #4152.