Skip to content

Commit 7517843

Browse files
committed
feat(helm): configure gateway OCSF JSONL output
Previously, Helm installations could not enable the gateway OCSF JSONL destination through chart values because generated `gateway.toml` omitted the `openshell.gateway.ocsf_log` table. Now, setting `server.ocsfLog.enabled` renders the path, optional schema version, rotation, retention, and queue limits into gateway configuration. Output is disabled by default. The default path, `/tmp/gateway-ocsf.jsonl`, is writable in the gateway container with either the StatefulSet or Deployment workload, so enabling output does not require persistent storage. Invalid schema versions, rotation values, non-positive limits, or an empty path while enabled fail chart rendering. Additionally, `server.extraVolumes` and `server.extraVolumeMounts` add operator-supplied volumes to the gateway pod, so operators who want records to survive restarts can place the OCSF path on persistent storage without replacing chart-generated configuration. When `server.ocsfLog.enabled` is true and `podLifecycle.terminationGracePeriodSeconds` is left empty, the chart raises the gateway pod's termination grace period from 5 to 20 seconds. Gateway shutdown can spend up to 10 seconds on supervisor session cleanup before allowing 5 seconds to drain queued OCSF records, so the 5-second default risks a SIGKILL before the final records are written. An explicit value still takes precedence. Refs #2762 Signed-off-by: Kris Hicks <[email protected]>
1 parent cf1bbb9 commit 7517843

6 files changed

Lines changed: 240 additions & 4 deletions

File tree

‎deploy/helm/openshell/README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -355,7 +355,7 @@ discovery endpoint or its TLS CA.
355355
| pkiInitJob.timeoutSeconds | int | `120` | Maximum time in seconds for the certgen hook to poll for cert-manager certificates. When using cert-manager with BackendTLSPolicy, the hook polls for this many seconds waiting for the certificate to be issued, then creates the backend CA ConfigMap. The Job deadline is set to (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. Increase this if cert-manager takes longer than 120 seconds to issue certificates. |
356356
| podAnnotations | object | `{}` | Extra annotations to add to the gateway pod. |
357357
| podLabels | object | `{}` | Extra labels to add to the gateway pod. |
358-
| podLifecycle.terminationGracePeriodSeconds | int | `5` | Grace period, in seconds, before Kubernetes terminates the gateway pod. |
358+
| podLifecycle.terminationGracePeriodSeconds | int | 5, or 20 when server.ocsfLog.enabled is true | Grace period, in seconds, before Kubernetes terminates the gateway pod. Empty uses 20 when server.ocsfLog.enabled is true, leaving time to drain queued OCSF records after other shutdown work, and 5 otherwise. |
359359
| podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. |
360360
| probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. |
361361
| probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. |
@@ -420,12 +420,21 @@ discovery endpoint or its TLS CA.
420420
| server.enableUserNamespaces | bool | `false` | Enable Kubernetes user namespace isolation (hostUsers: false) for sandbox pods. Requires Kubernetes 1.33+ with user namespace support available (beta through 1.35, GA in 1.36+), plus a supporting container runtime and Linux 5.12+. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced. |
421421
| server.enableWebsocketTunnel | bool | `false` | Enable the WebSocket tunnel used by CLI/SDK clients behind an authenticated edge proxy. Leave disabled for direct gateway installs. |
422422
| server.externalDbSecret | string | `""` | Name of a pre-existing Opaque Secret containing a PostgreSQL connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL from this Secret instead of using dbUrl. The Secret must contain a `uri` key, e.g. postgresql://user:pass@host:5432/dbname. |
423+
| server.extraVolumeMounts | list | `[]` | Additional volume mounts for the gateway container. |
424+
| server.extraVolumes | list | `[]` | Additional volumes for the gateway pod. |
423425
| server.grpcEndpoint | string | `""` | gRPC endpoint sandboxes call back into the gateway. Leave empty to derive it from the chart fullname, release namespace, service port, and disableTls flag, for example https://openshell.openshell.svc.cluster.local:8080. Override only when sandboxes must reach the gateway via a different hostname (e.g. an external ingress or a host alias). |
424426
| server.grpcRateLimit.requests | int | `0` | Maximum gRPC requests allowed per window. Must be positive (alongside windowSeconds) to enable rate limiting; 0 (default) disables it. |
425427
| server.grpcRateLimit.windowSeconds | int | `0` | gRPC rate-limit window length in seconds. Must be positive (alongside requests) to enable rate limiting; 0 (default) disables it. |
426428
| server.hostGatewayIP | string | `""` | Host gateway IP for sandbox pod hostAliases. When set, sandbox pods get hostAliases entries mapping host.docker.internal and host.openshell.internal to this IP, allowing them to reach services running on the Docker host. Auto-detected by the cluster entrypoint script. |
427429
| server.logLevel | string | `"info"` | Gateway log level. |
428430
| server.name | string | `""` | Operator-facing gateway name. Defaults to the chart fullname so all replicas in one installation share an identity. Set explicitly when one telemetry collector receives spans from multiple namespaces or clusters. |
431+
| server.ocsfLog.enabled | bool | `false` | Write gateway OCSF events as JSONL. |
432+
| server.ocsfLog.maxFiles | int | `7` | Rotated files retained when rotation is daily. |
433+
| server.ocsfLog.path | string | `"/tmp/gateway-ocsf.jsonl"` | OCSF JSONL path. The default is writable in the gateway container but does not persist across restarts. To keep records, mount a volume with server.extraVolumes and server.extraVolumeMounts and set a path on it. |
434+
| server.ocsfLog.queueCapacity | int | `10000` | Maximum records waiting for the file writer. |
435+
| server.ocsfLog.queueMaxBytes | int | `16777216` | Maximum encoded bytes waiting for the file writer. |
436+
| server.ocsfLog.rotation | string | `"daily"` | Rotate the active file daily in UTC, or never. |
437+
| server.ocsfLog.schemaVersion | string | `""` | Optional OCSF downgrade target. Empty emits native OCSF 1.8.0. Supported values: "1.1", "1.3". |
429438
| server.oidc.adminRole | string | `""` | Role name for admin access. Leave empty (with userRole also empty) for authentication-only mode. Both must be set or both empty. |
430439
| server.oidc.audience | string | `"openshell-cli"` | Expected audience claim for the API resource server. This should match the server's --oidc-audience, NOT the CLI client ID. |
431440
| server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). |

‎deploy/helm/openshell/templates/_gateway-workload.tpl‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,14 @@ metadata:
2121
{{- toYaml . | nindent 4 }}
2222
{{- end }}
2323
spec:
24-
terminationGracePeriodSeconds: {{ .Values.podLifecycle.terminationGracePeriodSeconds }}
24+
{{- $gracePeriod := .Values.podLifecycle.terminationGracePeriodSeconds }}
25+
{{- if kindIs "invalid" $gracePeriod }}
26+
{{- $gracePeriod = 5 }}
27+
{{- if (.Values.server.ocsfLog | default dict).enabled }}
28+
{{- $gracePeriod = 20 }}
29+
{{- end }}
30+
{{- end }}
31+
terminationGracePeriodSeconds: {{ $gracePeriod }}
2532
{{- with .Values.imagePullSecrets }}
2633
imagePullSecrets:
2734
{{- toYaml . | nindent 4 }}
@@ -184,6 +191,9 @@ spec:
184191
mountPath: {{ dir .Values.server.providerTokenGrants.spiffe.workloadApiSocketPath | quote }}
185192
readOnly: true
186193
{{- end }}
194+
{{- with .Values.server.extraVolumeMounts }}
195+
{{- toYaml . | nindent 8 }}
196+
{{- end }}
187197
ports:
188198
- name: grpc
189199
containerPort: {{ .Values.service.port }}
@@ -291,6 +301,9 @@ spec:
291301
driver: csi.spiffe.io
292302
readOnly: true
293303
{{- end }}
304+
{{- with .Values.server.extraVolumes }}
305+
{{- toYaml . | nindent 4 }}
306+
{{- end }}
294307
{{- with .Values.nodeSelector }}
295308
nodeSelector:
296309
{{- toYaml . | nindent 4 }}

‎deploy/helm/openshell/templates/gateway-config.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ One value is intentionally NOT rendered here:
1414
*/}}
1515
{{- $credentialDrivers := list -}}
1616
{{- $otlp := .Values.server.otlp | default dict -}}
17+
{{- $ocsfLog := .Values.server.ocsfLog | default dict -}}
1718
{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled -}}
1819
{{- $credentialDrivers = append $credentialDrivers "kubernetes-secrets" -}}
1920
{{- end -}}
@@ -88,6 +89,41 @@ data:
8889
{{- end }}
8990
{{- end }}
9091
92+
{{- if $ocsfLog.enabled }}
93+
{{- if not $ocsfLog.path -}}
94+
{{- fail "server.ocsfLog.path must be set when server.ocsfLog.enabled is true" -}}
95+
{{- end }}
96+
97+
{{- $ocsfRotation := $ocsfLog.rotation | default "daily" -}}
98+
{{- if not (has $ocsfRotation (list "daily" "never")) -}}
99+
{{- fail "server.ocsfLog.rotation must be daily or never" -}}
100+
{{- end }}
101+
{{- $ocsfQueueCapacity := int ($ocsfLog.queueCapacity | default 10000) -}}
102+
{{- $ocsfQueueMaxBytes := int ($ocsfLog.queueMaxBytes | default 16777216) -}}
103+
{{- if or (lt $ocsfQueueCapacity 1) (lt $ocsfQueueMaxBytes 1) -}}
104+
{{- fail "server.ocsfLog.queueCapacity and queueMaxBytes must be positive" -}}
105+
{{- end }}
106+
[openshell.gateway.ocsf_log]
107+
path = {{ $ocsfLog.path | quote }}
108+
{{- $ocsfSchemaVersion := $ocsfLog.schemaVersion | default "" -}}
109+
{{- if not (has $ocsfSchemaVersion (list "" "1.1" "1.3")) -}}
110+
{{- fail "server.ocsfLog.schemaVersion must be empty, 1.1, or 1.3" -}}
111+
{{- end }}
112+
{{- if $ocsfSchemaVersion }}
113+
schema_version = {{ $ocsfSchemaVersion | quote }}
114+
{{- end }}
115+
rotation = {{ $ocsfRotation | quote }}
116+
{{- if eq $ocsfRotation "daily" }}
117+
{{- $ocsfMaxFiles := int ($ocsfLog.maxFiles | default 7) -}}
118+
{{- if lt $ocsfMaxFiles 1 -}}
119+
{{- fail "server.ocsfLog.maxFiles must be positive when rotation is daily" -}}
120+
{{- end }}
121+
max_files = {{ $ocsfMaxFiles }}
122+
{{- end }}
123+
queue_capacity = {{ $ocsfQueueCapacity }}
124+
queue_max_bytes = {{ $ocsfQueueMaxBytes }}
125+
{{- end }}
126+
91127
{{- if not .Values.server.disableTls }}
92128
93129
[openshell.gateway.tls]

‎deploy/helm/openshell/tests/gateway_config_test.yaml‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,128 @@ tests:
303303
path: data["gateway.toml"]
304304
pattern: '\[openshell\.gateway\.otlp\]'
305305

306+
- it: omits OCSF JSONL output by default
307+
template: templates/gateway-config.yaml
308+
asserts:
309+
- notMatchRegex:
310+
path: data["gateway.toml"]
311+
pattern: '\[openshell\.gateway\.ocsf_log\]'
312+
313+
- it: treats a null OCSF log map as disabled
314+
template: templates/gateway-config.yaml
315+
set:
316+
server.ocsfLog: null
317+
asserts:
318+
- notMatchRegex:
319+
path: data["gateway.toml"]
320+
pattern: '\[openshell\.gateway\.ocsf_log\]'
321+
322+
- it: omits OCSF JSONL output when a path is set without enabling it
323+
template: templates/gateway-config.yaml
324+
set:
325+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
326+
asserts:
327+
- notMatchRegex:
328+
path: data["gateway.toml"]
329+
pattern: '\[openshell\.gateway\.ocsf_log\]'
330+
331+
- it: writes OCSF JSONL to the default container path when enabled
332+
template: templates/gateway-config.yaml
333+
set:
334+
server.ocsfLog.enabled: true
335+
asserts:
336+
- matchRegex:
337+
path: data["gateway.toml"]
338+
pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?path\s*=\s*"/tmp/gateway-ocsf\.jsonl"'
339+
340+
- it: rejects enabled OCSF JSONL output without a path
341+
template: templates/statefulset.yaml
342+
set:
343+
server.ocsfLog.enabled: true
344+
server.ocsfLog.path: ""
345+
asserts:
346+
- failedTemplate:
347+
errorMessage: "server.ocsfLog.path must be set when server.ocsfLog.enabled is true"
348+
349+
- it: renders OCSF JSONL output when configured
350+
template: templates/gateway-config.yaml
351+
set:
352+
server.ocsfLog.enabled: true
353+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
354+
server.ocsfLog.schemaVersion: "1.3"
355+
server.ocsfLog.rotation: daily
356+
server.ocsfLog.maxFiles: 14
357+
server.ocsfLog.queueCapacity: 20000
358+
server.ocsfLog.queueMaxBytes: 33554432
359+
asserts:
360+
- matchRegex:
361+
path: data["gateway.toml"]
362+
pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?path\s*=\s*"/var/openshell/gateway-ocsf\.jsonl".*?schema_version\s*=\s*"1\.3".*?rotation\s*=\s*"daily".*?max_files\s*=\s*14.*?queue_capacity\s*=\s*20000.*?queue_max_bytes\s*=\s*33554432'
363+
364+
- it: omits OCSF retention when rotation is disabled
365+
template: templates/gateway-config.yaml
366+
set:
367+
server.ocsfLog.enabled: true
368+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
369+
server.ocsfLog.rotation: never
370+
asserts:
371+
- matchRegex:
372+
path: data["gateway.toml"]
373+
pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?rotation\s*=\s*"never"'
374+
- notMatchRegex:
375+
path: data["gateway.toml"]
376+
pattern: 'max_files\s*='
377+
378+
- it: rejects an unsupported OCSF rotation mode
379+
template: templates/statefulset.yaml
380+
set:
381+
server.ocsfLog.enabled: true
382+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
383+
server.ocsfLog.rotation: hourly
384+
asserts:
385+
- failedTemplate:
386+
errorMessage: "server.ocsfLog.rotation must be daily or never"
387+
388+
- it: rejects an unsupported OCSF schema version
389+
template: templates/statefulset.yaml
390+
set:
391+
server.ocsfLog.enabled: true
392+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
393+
server.ocsfLog.schemaVersion: "1.8"
394+
asserts:
395+
- failedTemplate:
396+
errorMessage: "server.ocsfLog.schemaVersion must be empty, 1.1, or 1.3"
397+
398+
- it: mounts extra gateway volumes for OCSF output in a Deployment
399+
template: templates/deployment.yaml
400+
set:
401+
workload.kind: deployment
402+
server.externalDbSecret: my-pg-secret
403+
server.ocsfLog.enabled: true
404+
server.ocsfLog.path: /var/log/openshell/gateway-ocsf.jsonl
405+
server.extraVolumes:
406+
- name: ocsf-log
407+
persistentVolumeClaim:
408+
claimName: openshell-ocsf-log
409+
server.extraVolumeMounts:
410+
- name: ocsf-log
411+
mountPath: /var/log/openshell
412+
asserts:
413+
- contains:
414+
path: spec.template.spec.volumes
415+
content:
416+
name: ocsf-log
417+
persistentVolumeClaim:
418+
claimName: openshell-ocsf-log
419+
- contains:
420+
path: spec.template.spec.containers[0].volumeMounts
421+
content:
422+
name: ocsf-log
423+
mountPath: /var/log/openshell
424+
- equal:
425+
path: spec.template.spec.terminationGracePeriodSeconds
426+
value: 20
427+
306428
- it: mounts the OIDC CA bundle when TLS is disabled
307429
template: templates/statefulset.yaml
308430
set:
@@ -1242,3 +1364,31 @@ tests:
12421364
- matchRegex:
12431365
path: data["gateway.toml"]
12441366
pattern: "(?ms)\\[openshell\\.drivers\\.kubernetes\\].*?image_pull_policy\\s*=\\s*\\\"never\\\".*?supervisor_image_pull_policy\\s*=\\s*\\\"never\\\""
1367+
1368+
- it: uses a short termination grace period without OCSF JSONL output
1369+
template: templates/statefulset.yaml
1370+
asserts:
1371+
- equal:
1372+
path: spec.template.spec.terminationGracePeriodSeconds
1373+
value: 5
1374+
1375+
- it: extends the termination grace period to drain OCSF JSONL output
1376+
template: templates/statefulset.yaml
1377+
set:
1378+
server.ocsfLog.enabled: true
1379+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
1380+
asserts:
1381+
- equal:
1382+
path: spec.template.spec.terminationGracePeriodSeconds
1383+
value: 20
1384+
1385+
- it: keeps an explicit termination grace period with OCSF JSONL output
1386+
template: templates/statefulset.yaml
1387+
set:
1388+
server.ocsfLog.enabled: true
1389+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
1390+
podLifecycle.terminationGracePeriodSeconds: 3
1391+
asserts:
1392+
- equal:
1393+
path: spec.template.spec.terminationGracePeriodSeconds
1394+
value: 3

‎deploy/helm/openshell/values.yaml‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -212,8 +212,11 @@ agentSandbox:
212212

213213
# Pod restart behavior and health probe tuning.
214214
podLifecycle:
215-
# -- Grace period, in seconds, before Kubernetes terminates the gateway pod.
216-
terminationGracePeriodSeconds: 5
215+
# -- (int) Grace period, in seconds, before Kubernetes terminates the gateway pod.
216+
# Empty uses 20 when server.ocsfLog.enabled is true, leaving time to drain
217+
# queued OCSF records after other shutdown work, and 5 otherwise.
218+
# @default -- 5, or 20 when server.ocsfLog.enabled is true
219+
terminationGracePeriodSeconds:
217220

218221
probes:
219222
startup:
@@ -268,6 +271,29 @@ server:
268271
endpoint: ""
269272
# -- Gateway OpenTelemetry service name. Empty uses openshell-gateway.
270273
serviceName: ""
274+
# Gateway-native OCSF JSONL output.
275+
ocsfLog:
276+
# -- Write gateway OCSF events as JSONL.
277+
enabled: false
278+
# -- OCSF JSONL path. The default is writable in the gateway container but
279+
# does not persist across restarts. To keep records, mount a volume with
280+
# server.extraVolumes and server.extraVolumeMounts and set a path on it.
281+
path: /tmp/gateway-ocsf.jsonl
282+
# -- Optional OCSF downgrade target. Empty emits native OCSF 1.8.0.
283+
# Supported values: "1.1", "1.3".
284+
schemaVersion: ""
285+
# -- Rotate the active file daily in UTC, or never.
286+
rotation: daily
287+
# -- Rotated files retained when rotation is daily.
288+
maxFiles: 7
289+
# -- Maximum records waiting for the file writer.
290+
queueCapacity: 10000
291+
# -- Maximum encoded bytes waiting for the file writer.
292+
queueMaxBytes: 16777216
293+
# -- Additional volumes for the gateway pod.
294+
extraVolumes: []
295+
# -- Additional volume mounts for the gateway container.
296+
extraVolumeMounts: []
271297
# -- Enable anonymous OpenShell telemetry from the gateway and the sandbox
272298
# supervisors it launches.
273299
telemetryEnabled: true

‎docs/how-it-works/gateways/configuration.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,8 @@ queue_max_bytes = 16777216
303303

304304
Unknown fields are rejected. This is one file destination, not an exporter registry. The existing `ocsf_json_enabled` sandbox setting remains independent.
305305

306+
For Helm installations, set `server.ocsfLog.enabled` to render this table and optionally set `server.ocsfLog.schemaVersion`. The default `server.ocsfLog.path`, `/tmp/gateway-ocsf.jsonl`, is writable in the gateway container with either the StatefulSet or Deployment workload but does not persist across pod restarts. To keep records, mount a volume with `server.extraVolumes` and `server.extraVolumeMounts` and set the path on it. When output is enabled and `podLifecycle.terminationGracePeriodSeconds` is empty, the chart raises the pod's termination grace period from 5 to 20 seconds so the gateway can drain queued records during shutdown.
307+
306308
Give each gateway replica its own writable file and an external shipper read access to the containing directory. New files use owner-only permissions on Unix; arrange shipper access deliberately. Rotation renames the active file to a date- and UUID-suffixed sibling. The shipper must follow rotated files and checkpoint its progress. Do not use multiple writers or external copy-truncate rotation on this path.
307309

308310
The gateway queues OCSF independently of `RUST_LOG`, excludes ordinary diagnostics, and preserves native event IDs. It writes up to 100 records per batch with a 500 ms batching interval. One additional bounded batch can be in flight. File errors do not stop sandbox execution: failed writes are not replayed, and subsequent records are discarded during reopen backoff. A restarted writer removes an incomplete trailing line before appending.

0 commit comments

Comments
 (0)