Skip to content

Commit a3ddba4

Browse files
lexfreilexfrei
committed
fix(opensearch-operator): set vm.max_map_count from a node DaemonSet
OpenSearch refuses to start when vm.max_map_count is below 262144 and the store may use mmap. The kernel default is 65530 and nothing in the platform's node configuration raises it: the Talos profiles declare no sysctls, and neither does any documented node-preparation path. The value cannot come from the OpenSearch pod. The kernel does not namespace vm.*, so kubelet refuses it in a pod's securityContext.sysctls and in --allowed-unsafe-sysctls alike, both of which admit namespaced sysctls only. The operator's own answer, spec.general.setVMMaxMapCount, appends a privileged init container to every OpenSearch pod. Tenant namespaces carry no PodSecurity label of their own and take the cluster default, which Talos ships as baseline through the apiserver's PodSecurityConfiguration; that default rejects the container, so the StatefulSet is created and never produces a pod. Set the value once per node instead, from a DaemonSet in the operator's own namespace. The init container raises the limit to a floor rather than assigning it: the tunable is node-global, so a node already tuned higher for another mmap-heavy workload keeps its own value. The pod runs at system-node-critical: it is a precondition for every OpenSearch pod on the node, so it must not lose a scheduling race to the workload it enables, though priority is not start ordering: a pod that starts first just fails its own bootstrap check and retries. The idle container runs the sleep under a shell with a TERM handler, because PID 1 of a namespace receives only the signals it installed a handler for and a bare sleep would be SIGKILLed after the full grace period on every rollout and drain. The write goes to /proc/sys only, with no hostPath and no edit to the node's on-disk configuration; the pod restarts with the node and reapplies the value. The component is declared privileged so the platform labels that namespace enforce=privileged, which is what this DaemonSet needs and what tenant namespaces deliberately do not get. The file returns to a path bb660b5 emptied. That earlier revision bind-mounted the host's /etc to append to sysctl.conf and ran an unpinned image; this one writes no host file and pins the image by digest. This removes the reason for an OpenSearch pod to ask the operator for the sysctl, but the app chart still asks. Dropping that request is a separate change and is what stops the pods being rejected. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent d3559d4 commit a3ddba4

5 files changed

Lines changed: 231 additions & 0 deletions

File tree

‎packages/core/platform/sources/opensearch-operator.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,6 @@ spec:
1818
- name: opensearch-operator
1919
path: system/opensearch-operator
2020
install:
21+
privileged: true
2122
namespace: cozy-opensearch-operator
2223
releaseName: opensearch-operator

‎packages/core/platform/tests/sources_opensearch_operator_privileged_test.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
suite: opensearch-operator installs as a privileged component
2+
# The package ships a DaemonSet whose init container raises vm.max_map_count, and
3+
# that container has to be privileged: the kernel does not namespace vm.*, so no
4+
# pod can set the value for itself. cozystack-operator labels a namespace
5+
# enforce=privileged only when some component declares install.privileged, so
6+
# dropping this field does not fail rendering anywhere; it leaves the DaemonSet
7+
# inadmissible on any cluster whose PodSecurity default is baseline or stricter.
8+
# Pin it so the namespace cannot silently lose the level the DaemonSet needs.
9+
templates:
10+
- templates/sources.yaml
11+
release:
12+
name: cozystack
13+
namespace: cozy-system
14+
tests:
15+
- it: opensearch-operator PackageSource declares its component privileged
16+
documentSelector:
17+
path: metadata.name
18+
value: cozystack.opensearch-operator
19+
asserts:
20+
- equal:
21+
path: spec.variants[0].components[0].install.privileged
22+
value: true
23+
- equal:
24+
path: spec.variants[0].components[0].install.namespace
25+
value: cozy-opensearch-operator

‎packages/system/opensearch-operator/templates/sysctl-daemonset.yaml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
{{- /*
2+
OpenSearch refuses to start when vm.max_map_count is below 262144 and the store
3+
may use mmap; the kernel default is 65530 and nothing in the platform's node
4+
configuration raises it.
5+
6+
It cannot be raised from the OpenSearch pod itself. The kernel does not
7+
namespace vm.*, so kubelet rejects it both in a pod's securityContext.sysctls
8+
and in --allowed-unsafe-sysctls, which admits namespaced sysctls only. The
9+
operator's own answer, spec.general.setVMMaxMapCount, appends a privileged
10+
init container to every OpenSearch pod. Tenant namespaces carry no PodSecurity
11+
label of their own, so they take the cluster default, which Talos ships as
12+
baseline through the apiserver's PodSecurityConfiguration; that default rejects
13+
the container outright (cozystack/cozystack#4131).
14+
15+
This DaemonSet carries the value instead, from a namespace the platform labels
16+
enforce=privileged. It removes the reason for an OpenSearch pod to ask the
17+
operator for the sysctl. The app chart still asks, and dropping that request is
18+
a separate change.
19+
20+
system-node-critical buys scheduling priority, not start ordering: on a freshly
21+
joined node an OpenSearch pod can still start before this one has written the
22+
value, fail its own bootstrap check, and retry until the value is in.
23+
24+
The write goes to /proc/sys and nowhere else. The pod restarts with the node
25+
and reapplies the value, so persisting it into the host's own configuration
26+
would buy nothing and would mutate a node this chart does not own.
27+
*/}}
28+
apiVersion: apps/v1
29+
kind: DaemonSet
30+
metadata:
31+
name: {{ .Release.Name }}-sysctl
32+
namespace: {{ .Release.Namespace }}
33+
labels:
34+
app.kubernetes.io/name: {{ .Release.Name }}-sysctl
35+
app.kubernetes.io/component: sysctl
36+
spec:
37+
selector:
38+
matchLabels:
39+
app.kubernetes.io/name: {{ .Release.Name }}-sysctl
40+
template:
41+
metadata:
42+
labels:
43+
app.kubernetes.io/name: {{ .Release.Name }}-sysctl
44+
spec:
45+
priorityClassName: system-node-critical
46+
automountServiceAccountToken: false
47+
initContainers:
48+
- name: init-sysctl
49+
image: {{ .Values.sysctl.image | quote }}
50+
# A floor, not an assignment. vm.max_map_count is node-global, so a node
51+
# already tuned above 262144 for some other mmap-heavy workload must keep
52+
# its value; writing it unconditionally would lower the limit for
53+
# everything on that node, on every boot.
54+
command:
55+
- sh
56+
- -c
57+
- 'set -e; current=$(sysctl -n vm.max_map_count); if [ "$current" -lt 262144 ]; then sysctl -w vm.max_map_count=262144; fi'
58+
securityContext:
59+
privileged: true
60+
resources:
61+
requests:
62+
cpu: 1m
63+
memory: 4Mi
64+
limits:
65+
memory: 16Mi
66+
containers:
67+
# Carries no work: it exists so the pod stays scheduled, because a
68+
# DaemonSet pod that ran only an init container would complete and be
69+
# restarted continuously.
70+
#
71+
# Do not simplify this back to a bare sleep. The kernel delivers a signal
72+
# to PID 1 of a namespace only when PID 1 installed a handler for it, and
73+
# busybox sleep installs none, so a bare sleep ignores SIGTERM and every
74+
# pod is SIGKILLed after the full grace period on each rollout and drain.
75+
# The shell installs the handler, and wait returns when it fires.
76+
- name: idle
77+
image: {{ .Values.sysctl.image | quote }}
78+
command: ["sh", "-c", "trap 'exit 0' TERM INT; sleep infinity & wait"]
79+
securityContext:
80+
allowPrivilegeEscalation: false
81+
readOnlyRootFilesystem: true
82+
runAsNonRoot: true
83+
runAsUser: 65534
84+
capabilities:
85+
drop: ["ALL"]
86+
resources:
87+
requests:
88+
cpu: 1m
89+
memory: 4Mi
90+
limits:
91+
memory: 16Mi
92+
tolerations:
93+
- operator: Exists
94+
effect: NoSchedule
95+
- operator: Exists
96+
effect: NoExecute

‎packages/system/opensearch-operator/tests/sysctl_daemonset_test.yaml‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
suite: opensearch-operator raises vm.max_map_count per node
2+
3+
# OpenSearch will not start below vm.max_map_count=262144 and the sysctl is not
4+
# namespaced, so no pod can set it for itself. This DaemonSet is the platform's
5+
# carrier for it, which is why its shape is the contract: privileged init, that
6+
# exact sysctl, and nothing mounted from the host.
7+
release:
8+
name: opensearch-operator
9+
namespace: cozy-opensearch-operator
10+
tests:
11+
# The command is pinned whole because both halves of it are the contract: the
12+
# threshold OpenSearch needs, and the comparison that makes it a floor. Writing
13+
# the value unconditionally would lower vm.max_map_count on any node already
14+
# tuned above it, node-wide, on every boot.
15+
- it: raises the sysctl to a floor from a privileged init container
16+
template: templates/sysctl-daemonset.yaml
17+
asserts:
18+
- equal:
19+
path: kind
20+
value: DaemonSet
21+
- equal:
22+
path: spec.template.spec.initContainers[0].name
23+
value: init-sysctl
24+
- equal:
25+
path: spec.template.spec.initContainers[0].command
26+
value:
27+
- sh
28+
- -c
29+
- 'set -e; current=$(sysctl -n vm.max_map_count); if [ "$current" -lt 262144 ]; then sysctl -w vm.max_map_count=262144; fi'
30+
- equal:
31+
path: spec.template.spec.initContainers[0].securityContext.privileged
32+
value: true
33+
34+
- it: keeps the privilege in the init container only
35+
template: templates/sysctl-daemonset.yaml
36+
asserts:
37+
- equal:
38+
path: spec.template.spec.containers[0].name
39+
value: idle
40+
# The trap is the contract, not decoration. PID 1 of a namespace receives
41+
# only the signals it installed a handler for, so a bare sleep here ignores
42+
# SIGTERM and every pod is SIGKILLed after the full grace period on each
43+
# rollout and node drain.
44+
- equal:
45+
path: spec.template.spec.containers[0].command
46+
value: ["sh", "-c", "trap 'exit 0' TERM INT; sleep infinity & wait"]
47+
- notExists:
48+
path: spec.template.spec.containers[0].securityContext.privileged
49+
- equal:
50+
path: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation
51+
value: false
52+
- equal:
53+
path: spec.template.spec.containers[0].securityContext.capabilities.drop
54+
value: ["ALL"]
55+
56+
# Nothing may reach the host filesystem: the write belongs in /proc/sys and
57+
# the pod reapplies it on every node restart.
58+
- it: touches no host path
59+
template: templates/sysctl-daemonset.yaml
60+
asserts:
61+
- notExists:
62+
path: spec.template.spec.volumes
63+
- notExists:
64+
path: spec.template.spec.initContainers[0].volumeMounts
65+
- notExists:
66+
path: spec.template.spec.containers[0].volumeMounts
67+
- equal:
68+
path: spec.template.spec.automountServiceAccountToken
69+
value: false
70+
- equal:
71+
path: spec.template.spec.priorityClassName
72+
value: system-node-critical
73+
74+
- it: lands on tainted nodes and pins the image by digest
75+
template: templates/sysctl-daemonset.yaml
76+
asserts:
77+
- contains:
78+
path: spec.template.spec.tolerations
79+
content:
80+
operator: Exists
81+
effect: NoSchedule
82+
- contains:
83+
path: spec.template.spec.tolerations
84+
content:
85+
operator: Exists
86+
effect: NoExecute
87+
- matchRegex:
88+
path: spec.template.spec.initContainers[0].image
89+
pattern: '^docker\.io/library/busybox:[^@]+@sha256:[0-9a-f]{64}$'
90+
91+
- it: takes the image override for an air-gapped registry
92+
template: templates/sysctl-daemonset.yaml
93+
set:
94+
sysctl:
95+
image: registry.example.test/library/busybox:1.37.0@sha256:0000000000000000000000000000000000000000000000000000000000000000
96+
asserts:
97+
- equal:
98+
path: spec.template.spec.initContainers[0].image
99+
value: registry.example.test/library/busybox:1.37.0@sha256:0000000000000000000000000000000000000000000000000000000000000000
100+
- equal:
101+
path: spec.template.spec.containers[0].image
102+
value: registry.example.test/library/busybox:1.37.0@sha256:0000000000000000000000000000000000000000000000000000000000000000

‎packages/system/opensearch-operator/values.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,10 @@ opensearch-operator:
1212
image:
1313
# gcr.io/kubebuilder/kube-rbac-proxy was sunset; use the original upstream
1414
repository: "quay.io/brancz/kube-rbac-proxy"
15+
16+
sysctl:
17+
# busybox carries the sysctl applet and doubles as the idle main container.
18+
# Third-party pass-through pinned by digest, the same reference the kubernetes
19+
# package vendors; docs/agents/image-refs.md covers why such a ref is neither
20+
# retagged nor mirrored.
21+
image: docker.io/library/busybox:1.37.0@sha256:1487d0af5f52b4ba31c7e465126ee2123fe3f2305d638e7827681e7cf6c83d5e

0 commit comments

Comments
 (0)