Skip to content

OpenSearch fails to start in tenant namespaces: privileged init-sysctl violates default baseline PodSecurity #3022

Closed

Description

What happens

Deploying the OpenSearch application into a tenant namespace never starts — the opensearch-*-nodes StatefulSet cannot create its pod:

FailedCreate: pods "opensearch-...-nodes-0" is forbidden:
violates PodSecurity "baseline:latest":
privileged (container "init-sysctl" must not set securityContext.privileged=true)

The OpenSearch CR template hardcodes spec.general.setVMMaxMapCount: true (packages/apps/opensearch/templates/opensearch.yaml). The opster operator then injects a privileged init-sysctl initContainer to raise vm.max_map_count. Tenant namespaces enforce PodSecurity baseline by default (no pod-security.kubernetes.io/enforce label on the namespace, so the cluster-wide PSA default applies), which forbids privileged containers.

Result: zero OpenSearch master/data pods. The *-external / *-dashboards-external LoadBalancer services get IPs but their EndpointSlices stay empty, so OpenSearch and Dashboards are unreachable (a plain telnet to the LB port hangs on "Trying"). The OpenSearchCluster CR misleadingly reports Ready: True / PHASE: RUNNING because only the non-privileged securityconfig job runs.

This is an internal contradiction: Cozystack enforces baseline PSA on tenant namespaces but ships an app that requires a privileged initContainer.

Reproduction

On a clean Cozystack 1.4.2 cluster (Kubernetes 1.34), create an OpenSearch app in any tenant namespace:

apiVersion: apps.cozystack.io/v1alpha1
kind: OpenSearch
metadata:
  name: repro-psa
  namespace: tenant-demo
spec:
  replicas: 1
  size: 2Gi
  resourcesPreset: c1.small
  dashboards:
    enabled: true

The opensearch-repro-psa-nodes StatefulSet immediately logs the FailedCreate above and never creates a pod.

Environment

  • Reproduced on Cozystack v1.4.2 and v1.4.4
  • OpenSearch app (opster operator), default tenant namespace
  • Nodes have vm.max_map_count = 65530 (Linux default), so the sysctl init genuinely is needed — it cannot simply be disabled without setting the sysctl elsewhere

Possible fixes

  • Set vm.max_map_count=262144 at the node level (Talos sysctl via the Cozystack node config) and set setVMMaxMapCount: false in the package — removes the privileged init entirely.
  • Or run the OpenSearch app in a namespace permitted to run privileged.

Workaround (verified)

Raise the tenant namespace PSA to privileged (weakens tenant isolation, per-namespace):

kubectl label ns tenant-demo pod-security.kubernetes.io/enforce=privileged --overwrite

After labelling, the node pod passes Init (the privileged init-sysctl runs), the PVC binds, and the cluster comes up. The label is not reconciled away by the tenant chart (it does not template any pod-security.kubernetes.io/* label).

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    and removed
    securitySecurity-related issues and features
    on Jun 23, 2026
  2. yankawai commented on Sep 2, 2026

    @yankawai
    Contributor

    Hit this on Cozystack v1.6.x (platform 54) — same failure verbatim, so the report still reproduces two and a half months on.

    Confirmed all three links on a live cluster: nodes sit at vm.max_map_count = 65530 with no vm.* entry in the machine config, the chart hardcodes setVMMaxMapCount: true at packages/apps/opensearch/templates/opensearch.yaml:16, and the tenant namespace carries no pod-security.kubernetes.io/* label, so the cluster-wide enforce: baseline applies.

    Opened #4026 for the half that belongs to the chart: setVMMaxMapCount becomes a values knob, default true, so an operator who has already raised the sysctl at node level can stop the privileged initContainer from being injected. The default deliberately does not change — the sysctl is node-level rather than namespaced, so it cannot be set through a Pod securityContext.sysctls, and flipping it would trade an admission failure for a bootstrap-check crash on any cluster that has not set it.

    The other half is the platform's: packages/core/talos builds images only and ships no node sysctls, so nothing in-tree sets vm.max_map_count=262144. Happy to follow up with that if the maintainers agree it belongs in the platform rather than in operator documentation.

  3. myasnikovdaniil commented on Nov 6, 2026

    @myasnikovdaniil
    Contributor

    Tenant OpenSearch pods don't request the privileged sysctl init container anymore (#2682), node level sysctl is handled by the system DaemonSet from #4152.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions