Skip to content

Commit 9b8f956

Browse files
committed
feat(helm): configure gateway OCSF JSONL output
Previously, Helm installations could not enable the gateway OCSF JSONL destination through chart values because generated `gateway.toml` omitted the `openshell.gateway.ocsf_log` table. Now, setting `server.ocsfLog.enabled` renders the path, optional schema version, rotation, retention, and queue limits into gateway configuration. Output is disabled by default. The default path, `/tmp/gateway-ocsf.jsonl`, is writable in the gateway container with either the StatefulSet or Deployment workload, so enabling output does not require persistent storage. Invalid schema versions, rotation values, non-positive limits, or an empty path while enabled fail chart rendering. Additionally, `server.extraVolumes` and `server.extraVolumeMounts` add operator-supplied volumes to the gateway pod, so operators who want records to survive restarts can place the OCSF path on persistent storage without replacing chart-generated configuration. The gateway pod's default termination grace period rises from 5 to 30 seconds. Gateway shutdown can spend up to 10 seconds on supervisor session cleanup before allowing 5 seconds to drain queued OCSF records, so the 5-second default risked a SIGKILL before the final records were written. The grace period is only an upper bound: the gateway exits as soon as its shutdown completes. Refs #2762 Signed-off-by: Kris Hicks <[email protected]>
1 parent cf1bbb9 commit 9b8f956

6 files changed

Lines changed: 216 additions & 3 deletions

File tree

‎deploy/helm/openshell/README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -355,7 +355,7 @@ discovery endpoint or its TLS CA.
355355
| pkiInitJob.timeoutSeconds | int | `120` | Maximum time in seconds for the certgen hook to poll for cert-manager certificates. When using cert-manager with BackendTLSPolicy, the hook polls for this many seconds waiting for the certificate to be issued, then creates the backend CA ConfigMap. The Job deadline is set to (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. Increase this if cert-manager takes longer than 120 seconds to issue certificates. |
356356
| podAnnotations | object | `{}` | Extra annotations to add to the gateway pod. |
357357
| podLabels | object | `{}` | Extra labels to add to the gateway pod. |
358-
| podLifecycle.terminationGracePeriodSeconds | int | `5` | Grace period, in seconds, before Kubernetes terminates the gateway pod. |
358+
| podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. |
359359
| podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. |
360360
| probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. |
361361
| probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. |
@@ -420,12 +420,21 @@ discovery endpoint or its TLS CA.
420420
| server.enableUserNamespaces | bool | `false` | Enable Kubernetes user namespace isolation (hostUsers: false) for sandbox pods. Requires Kubernetes 1.33+ with user namespace support available (beta through 1.35, GA in 1.36+), plus a supporting container runtime and Linux 5.12+. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced. |
421421
| server.enableWebsocketTunnel | bool | `false` | Enable the WebSocket tunnel used by CLI/SDK clients behind an authenticated edge proxy. Leave disabled for direct gateway installs. |
422422
| server.externalDbSecret | string | `""` | Name of a pre-existing Opaque Secret containing a PostgreSQL connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL from this Secret instead of using dbUrl. The Secret must contain a `uri` key, e.g. postgresql://user:pass@host:5432/dbname. |
423+
| server.extraVolumeMounts | list | `[]` | Additional volume mounts for the gateway container. |
424+
| server.extraVolumes | list | `[]` | Additional volumes for the gateway pod. |
423425
| server.grpcEndpoint | string | `""` | gRPC endpoint sandboxes call back into the gateway. Leave empty to derive it from the chart fullname, release namespace, service port, and disableTls flag, for example https://openshell.openshell.svc.cluster.local:8080. Override only when sandboxes must reach the gateway via a different hostname (e.g. an external ingress or a host alias). |
424426
| server.grpcRateLimit.requests | int | `0` | Maximum gRPC requests allowed per window. Must be positive (alongside windowSeconds) to enable rate limiting; 0 (default) disables it. |
425427
| server.grpcRateLimit.windowSeconds | int | `0` | gRPC rate-limit window length in seconds. Must be positive (alongside requests) to enable rate limiting; 0 (default) disables it. |
426428
| server.hostGatewayIP | string | `""` | Host gateway IP for sandbox pod hostAliases. When set, sandbox pods get hostAliases entries mapping host.docker.internal and host.openshell.internal to this IP, allowing them to reach services running on the Docker host. Auto-detected by the cluster entrypoint script. |
427429
| server.logLevel | string | `"info"` | Gateway log level. |
428430
| server.name | string | `""` | Operator-facing gateway name. Defaults to the chart fullname so all replicas in one installation share an identity. Set explicitly when one telemetry collector receives spans from multiple namespaces or clusters. |
431+
| server.ocsfLog.enabled | bool | `false` | Write gateway OCSF events as JSONL. |
432+
| server.ocsfLog.maxFiles | int | `7` | Rotated files retained when rotation is daily. |
433+
| server.ocsfLog.path | string | `"/tmp/gateway-ocsf.jsonl"` | OCSF JSONL path. The default is writable in the gateway container but does not persist across restarts. To keep records, mount a volume with server.extraVolumes and server.extraVolumeMounts and set a path on it. |
434+
| server.ocsfLog.queueCapacity | int | `10000` | Maximum records waiting for the file writer. |
435+
| server.ocsfLog.queueMaxBytes | int | `16777216` | Maximum encoded bytes waiting for the file writer. |
436+
| server.ocsfLog.rotation | string | `"daily"` | Rotate the active file daily in UTC, or never. |
437+
| server.ocsfLog.schemaVersion | string | `""` | Optional OCSF downgrade target. Empty emits native OCSF 1.8.0. Supported values: "1.1", "1.3". |
429438
| server.oidc.adminRole | string | `""` | Role name for admin access. Leave empty (with userRole also empty) for authentication-only mode. Both must be set or both empty. |
430439
| server.oidc.audience | string | `"openshell-cli"` | Expected audience claim for the API resource server. This should match the server's --oidc-audience, NOT the CLI client ID. |
431440
| server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). |

‎deploy/helm/openshell/templates/_gateway-workload.tpl‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,9 @@ spec:
184184
mountPath: {{ dir .Values.server.providerTokenGrants.spiffe.workloadApiSocketPath | quote }}
185185
readOnly: true
186186
{{- end }}
187+
{{- with .Values.server.extraVolumeMounts }}
188+
{{- toYaml . | nindent 8 }}
189+
{{- end }}
187190
ports:
188191
- name: grpc
189192
containerPort: {{ .Values.service.port }}
@@ -291,6 +294,9 @@ spec:
291294
driver: csi.spiffe.io
292295
readOnly: true
293296
{{- end }}
297+
{{- with .Values.server.extraVolumes }}
298+
{{- toYaml . | nindent 4 }}
299+
{{- end }}
294300
{{- with .Values.nodeSelector }}
295301
nodeSelector:
296302
{{- toYaml . | nindent 4 }}

‎deploy/helm/openshell/templates/gateway-config.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ One value is intentionally NOT rendered here:
1414
*/}}
1515
{{- $credentialDrivers := list -}}
1616
{{- $otlp := .Values.server.otlp | default dict -}}
17+
{{- $ocsfLog := .Values.server.ocsfLog | default dict -}}
1718
{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled -}}
1819
{{- $credentialDrivers = append $credentialDrivers "kubernetes-secrets" -}}
1920
{{- end -}}
@@ -88,6 +89,41 @@ data:
8889
{{- end }}
8990
{{- end }}
9091
92+
{{- if $ocsfLog.enabled }}
93+
{{- if not $ocsfLog.path -}}
94+
{{- fail "server.ocsfLog.path must be set when server.ocsfLog.enabled is true" -}}
95+
{{- end }}
96+
97+
{{- $ocsfRotation := $ocsfLog.rotation | default "daily" -}}
98+
{{- if not (has $ocsfRotation (list "daily" "never")) -}}
99+
{{- fail "server.ocsfLog.rotation must be daily or never" -}}
100+
{{- end }}
101+
{{- $ocsfQueueCapacity := int ($ocsfLog.queueCapacity | default 10000) -}}
102+
{{- $ocsfQueueMaxBytes := int ($ocsfLog.queueMaxBytes | default 16777216) -}}
103+
{{- if or (lt $ocsfQueueCapacity 1) (lt $ocsfQueueMaxBytes 1) -}}
104+
{{- fail "server.ocsfLog.queueCapacity and queueMaxBytes must be positive" -}}
105+
{{- end }}
106+
[openshell.gateway.ocsf_log]
107+
path = {{ $ocsfLog.path | quote }}
108+
{{- $ocsfSchemaVersion := $ocsfLog.schemaVersion | default "" -}}
109+
{{- if not (has $ocsfSchemaVersion (list "" "1.1" "1.3")) -}}
110+
{{- fail "server.ocsfLog.schemaVersion must be empty, 1.1, or 1.3" -}}
111+
{{- end }}
112+
{{- if $ocsfSchemaVersion }}
113+
schema_version = {{ $ocsfSchemaVersion | quote }}
114+
{{- end }}
115+
rotation = {{ $ocsfRotation | quote }}
116+
{{- if eq $ocsfRotation "daily" }}
117+
{{- $ocsfMaxFiles := int ($ocsfLog.maxFiles | default 7) -}}
118+
{{- if lt $ocsfMaxFiles 1 -}}
119+
{{- fail "server.ocsfLog.maxFiles must be positive when rotation is daily" -}}
120+
{{- end }}
121+
max_files = {{ $ocsfMaxFiles }}
122+
{{- end }}
123+
queue_capacity = {{ $ocsfQueueCapacity }}
124+
queue_max_bytes = {{ $ocsfQueueMaxBytes }}
125+
{{- end }}
126+
91127
{{- if not .Values.server.disableTls }}
92128
93129
[openshell.gateway.tls]

‎deploy/helm/openshell/tests/gateway_config_test.yaml‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,125 @@ tests:
303303
path: data["gateway.toml"]
304304
pattern: '\[openshell\.gateway\.otlp\]'
305305

306+
- it: omits OCSF JSONL output by default
307+
template: templates/gateway-config.yaml
308+
asserts:
309+
- notMatchRegex:
310+
path: data["gateway.toml"]
311+
pattern: '\[openshell\.gateway\.ocsf_log\]'
312+
313+
- it: treats a null OCSF log map as disabled
314+
template: templates/gateway-config.yaml
315+
set:
316+
server.ocsfLog: null
317+
asserts:
318+
- notMatchRegex:
319+
path: data["gateway.toml"]
320+
pattern: '\[openshell\.gateway\.ocsf_log\]'
321+
322+
- it: omits OCSF JSONL output when a path is set without enabling it
323+
template: templates/gateway-config.yaml
324+
set:
325+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
326+
asserts:
327+
- notMatchRegex:
328+
path: data["gateway.toml"]
329+
pattern: '\[openshell\.gateway\.ocsf_log\]'
330+
331+
- it: writes OCSF JSONL to the default container path when enabled
332+
template: templates/gateway-config.yaml
333+
set:
334+
server.ocsfLog.enabled: true
335+
asserts:
336+
- matchRegex:
337+
path: data["gateway.toml"]
338+
pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?path\s*=\s*"/tmp/gateway-ocsf\.jsonl"'
339+
340+
- it: rejects enabled OCSF JSONL output without a path
341+
template: templates/statefulset.yaml
342+
set:
343+
server.ocsfLog.enabled: true
344+
server.ocsfLog.path: ""
345+
asserts:
346+
- failedTemplate:
347+
errorMessage: "server.ocsfLog.path must be set when server.ocsfLog.enabled is true"
348+
349+
- it: renders OCSF JSONL output when configured
350+
template: templates/gateway-config.yaml
351+
set:
352+
server.ocsfLog.enabled: true
353+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
354+
server.ocsfLog.schemaVersion: "1.3"
355+
server.ocsfLog.rotation: daily
356+
server.ocsfLog.maxFiles: 14
357+
server.ocsfLog.queueCapacity: 20000
358+
server.ocsfLog.queueMaxBytes: 33554432
359+
asserts:
360+
- matchRegex:
361+
path: data["gateway.toml"]
362+
pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?path\s*=\s*"/var/openshell/gateway-ocsf\.jsonl".*?schema_version\s*=\s*"1\.3".*?rotation\s*=\s*"daily".*?max_files\s*=\s*14.*?queue_capacity\s*=\s*20000.*?queue_max_bytes\s*=\s*33554432'
363+
364+
- it: omits OCSF retention when rotation is disabled
365+
template: templates/gateway-config.yaml
366+
set:
367+
server.ocsfLog.enabled: true
368+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
369+
server.ocsfLog.rotation: never
370+
asserts:
371+
- matchRegex:
372+
path: data["gateway.toml"]
373+
pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?rotation\s*=\s*"never"'
374+
- notMatchRegex:
375+
path: data["gateway.toml"]
376+
pattern: 'max_files\s*='
377+
378+
- it: rejects an unsupported OCSF rotation mode
379+
template: templates/statefulset.yaml
380+
set:
381+
server.ocsfLog.enabled: true
382+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
383+
server.ocsfLog.rotation: hourly
384+
asserts:
385+
- failedTemplate:
386+
errorMessage: "server.ocsfLog.rotation must be daily or never"
387+
388+
- it: rejects an unsupported OCSF schema version
389+
template: templates/statefulset.yaml
390+
set:
391+
server.ocsfLog.enabled: true
392+
server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
393+
server.ocsfLog.schemaVersion: "1.8"
394+
asserts:
395+
- failedTemplate:
396+
errorMessage: "server.ocsfLog.schemaVersion must be empty, 1.1, or 1.3"
397+
398+
- it: mounts extra gateway volumes for OCSF output in a Deployment
399+
template: templates/deployment.yaml
400+
set:
401+
workload.kind: deployment
402+
server.externalDbSecret: my-pg-secret
403+
server.ocsfLog.enabled: true
404+
server.ocsfLog.path: /var/log/openshell/gateway-ocsf.jsonl
405+
server.extraVolumes:
406+
- name: ocsf-log
407+
persistentVolumeClaim:
408+
claimName: openshell-ocsf-log
409+
server.extraVolumeMounts:
410+
- name: ocsf-log
411+
mountPath: /var/log/openshell
412+
asserts:
413+
- contains:
414+
path: spec.template.spec.volumes
415+
content:
416+
name: ocsf-log
417+
persistentVolumeClaim:
418+
claimName: openshell-ocsf-log
419+
- contains:
420+
path: spec.template.spec.containers[0].volumeMounts
421+
content:
422+
name: ocsf-log
423+
mountPath: /var/log/openshell
424+
306425
- it: mounts the OIDC CA bundle when TLS is disabled
307426
template: templates/statefulset.yaml
308427
set:
@@ -1242,3 +1361,19 @@ tests:
12421361
- matchRegex:
12431362
path: data["gateway.toml"]
12441363
pattern: "(?ms)\\[openshell\\.drivers\\.kubernetes\\].*?image_pull_policy\\s*=\\s*\\\"never\\\".*?supervisor_image_pull_policy\\s*=\\s*\\\"never\\\""
1364+
1365+
- it: gives the gateway time to finish shutdown by default
1366+
template: templates/statefulset.yaml
1367+
asserts:
1368+
- equal:
1369+
path: spec.template.spec.terminationGracePeriodSeconds
1370+
value: 30
1371+
1372+
- it: keeps an explicit termination grace period
1373+
template: templates/statefulset.yaml
1374+
set:
1375+
podLifecycle.terminationGracePeriodSeconds: 3
1376+
asserts:
1377+
- equal:
1378+
path: spec.template.spec.terminationGracePeriodSeconds
1379+
value: 3

‎deploy/helm/openshell/values.yaml‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -212,8 +212,10 @@ agentSandbox:
212212

213213
# Pod restart behavior and health probe tuning.
214214
podLifecycle:
215-
# -- Grace period, in seconds, before Kubernetes terminates the gateway pod.
216-
terminationGracePeriodSeconds: 5
215+
# -- Maximum time, in seconds, Kubernetes waits for the gateway to exit before
216+
# killing it. The gateway exits as soon as shutdown completes; the limit
217+
# covers supervisor session cleanup and draining queued OCSF records.
218+
terminationGracePeriodSeconds: 30
217219

218220
probes:
219221
startup:
@@ -268,6 +270,29 @@ server:
268270
endpoint: ""
269271
# -- Gateway OpenTelemetry service name. Empty uses openshell-gateway.
270272
serviceName: ""
273+
# Gateway-native OCSF JSONL output.
274+
ocsfLog:
275+
# -- Write gateway OCSF events as JSONL.
276+
enabled: false
277+
# -- OCSF JSONL path. The default is writable in the gateway container but
278+
# does not persist across restarts. To keep records, mount a volume with
279+
# server.extraVolumes and server.extraVolumeMounts and set a path on it.
280+
path: /tmp/gateway-ocsf.jsonl
281+
# -- Optional OCSF downgrade target. Empty emits native OCSF 1.8.0.
282+
# Supported values: "1.1", "1.3".
283+
schemaVersion: ""
284+
# -- Rotate the active file daily in UTC, or never.
285+
rotation: daily
286+
# -- Rotated files retained when rotation is daily.
287+
maxFiles: 7
288+
# -- Maximum records waiting for the file writer.
289+
queueCapacity: 10000
290+
# -- Maximum encoded bytes waiting for the file writer.
291+
queueMaxBytes: 16777216
292+
# -- Additional volumes for the gateway pod.
293+
extraVolumes: []
294+
# -- Additional volume mounts for the gateway container.
295+
extraVolumeMounts: []
271296
# -- Enable anonymous OpenShell telemetry from the gateway and the sandbox
272297
# supervisors it launches.
273298
telemetryEnabled: true

‎docs/how-it-works/gateways/configuration.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,8 @@ queue_max_bytes = 16777216
303303

304304
Unknown fields are rejected. This is one file destination, not an exporter registry. The existing `ocsf_json_enabled` sandbox setting remains independent.
305305

306+
For Helm installations, set `server.ocsfLog.enabled` to render this table and optionally set `server.ocsfLog.schemaVersion`. The default `server.ocsfLog.path`, `/tmp/gateway-ocsf.jsonl`, is writable in the gateway container with either the StatefulSet or Deployment workload but does not persist across pod restarts. To keep records, mount a volume with `server.extraVolumes` and `server.extraVolumeMounts` and set the path on it.
307+
306308
Give each gateway replica its own writable file and an external shipper read access to the containing directory. New files use owner-only permissions on Unix; arrange shipper access deliberately. Rotation renames the active file to a date- and UUID-suffixed sibling. The shipper must follow rotated files and checkpoint its progress. Do not use multiple writers or external copy-truncate rotation on this path.
307309

308310
The gateway queues OCSF independently of `RUST_LOG`, excludes ordinary diagnostics, and preserves native event IDs. It writes up to 100 records per batch with a 500 ms batching interval. One additional bounded batch can be in flight. File errors do not stop sandbox execution: failed writes are not replayed, and subsequent records are discarded during reopen backoff. A restarted writer removes an incomplete trailing line before appending.

0 commit comments

Comments
 (0)