Skip to content

feat(supervisor): record each exec in the workload and its exit - #5253

Draft
krishicks wants to merge 1 commit into
feat/4268-session-process-events/krishicksfrom
feat/4268-exec-process-events/krishicks
Draft

krishicks wants to merge 1 commit into
feat/4268-session-process-events/krishicksfrom
feat/4268-exec-process-events/krishicks

Conversation

@krishicks

@krishicks krishicks commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Process Activity covered only the processes the runtime starts itself, so anything those processes ran (a shell's commands, an agent's tools) left no record. With ocsf_process_exec_enabled on, the runtime records every exec in the workload through the seccomp listener and the exit of each recorded process through a pidfd, and the supervisor emits them as Process Activity Launch and Terminate.

Third of a stack of four for #4268 (based on #5252):

  1. feat(supervisor): identify the process behind each network event #5251 Identify the process behind each network event
  2. feat(supervisor): record launch and exit of sandbox sessions #5252 Record launch and exit of sandbox sessions
  3. feat(supervisor): record each exec in the workload and its exit #5253 This PR: record each exec in the workload and its exit
  4. feat(supervisor): record the command line of each process #5254 Record the command line of each process

Related Issue

Refs #4268

Changes

  • The seccomp listener also receives execve and execveat. When exec recording is on, the runtime records each exec of an existing executable before letting it proceed: pid, parent pid, start time, the executable, and the executables of the caller before the exec and of its parent. File lookups run on a dedicated thread with a bounded queue, so a slow mount can't stall the broker. A full queue lets the exec continue and counts it as suppressed.
  • Each exec becomes a Launch with launch type Exec and an Unknown action, disposition, and status, because the kernel hasn't run it yet.
  • The runtime opens a pidfd for the caller while it is blocked in the exec. An exit-observer thread polls these and records each exit with its wait status, taken from PIDFD_GET_INFO/PIDFD_INFO_EXIT (kernel 6.15+) or from procfs while the process is still a zombie. Orphans adopted by the subreaper runtime are watched; processes the runtime waits for itself are left to the session path.
  • Terminate's actor is the process itself, with exit_code when the status was read and Unknown status otherwise.
  • Rate limit: execs beyond a burst of 500 and 200/s, and processes beyond the exit-watch cap (min(open-file soft limit / 4, 1024)), are counted. The count is reported at most once per second as a Process Activity Other record with count. Events evicted before a reading supervisor reached them are reported the same way.
  • ReadProcessEvents.executions is an optional bool. The supervisor leaves it unset until the gateway setting has been applied, so a replacement supervisor doesn't switch recording off. New capability: openshell.sandbox.process-exec-events.
  • New setting ocsf_process_exec_enabled; docs in docs/observability/ocsf-json-export.mdx.

Setting:

openshell settings set --global --key ocsf_process_exec_enabled --value true

Before (setting on, previous PR): only session Launch/Terminate; /bin/true, /bin/sleep, and an 800-exec burst don't appear.

After (Docker Desktop, kernel 7.0):

PROC:LAUNCH    [INFO] /usr/bin/bash(80)                # session (Spawn)
PROC:LAUNCH    [INFO] /usr/bin/dash(80)                # Exec: bash -> sh
PROC:LAUNCH    [INFO] /usr/bin/true(85)
PROC:TERMINATE [INFO] /usr/bin/true(85) [exit:0]
PROC:LAUNCH    [INFO] /usr/bin/sleep(86)
PROC:TERMINATE [INFO] /usr/bin/sleep(86) [exit:0]
PROC:TERMINATE [INFO] /usr/bin/false(99) [exit:1]      # orphan from setsid -f, parent 1
PROC:OTHER     [LOW]  /.openshell/runtime/openshell-sandbox(1) [count:244]
{"message":"Process exited: pid=85 exit=0","exit_code":0,"status":"Success","process":{"name":"/usr/bin/true","pid":85,"created_time":1793765997180},"uid":"<sandbox>/85/1793765997180/exit"}
{"message":"Process events suppressed: suppressed=244 not recorded","count":244,"status":"Failure"}

Totals: 575 Exec Launches and 574 Terminates, all with an exit_code. Terminate joins its Launch on pid + created_time.

Known limitations:

  • The exec is recorded at the seccomp notification, before the kernel runs it. An exec that fails for a reason other than a missing or non-executable file (ENOEXEC, E2BIG, an LSM or mount denial) still produces a Launch, with outcome Unknown.
  • The path is read from workload memory before the exec, so another thread can change it first. This is audit, not enforcement.
  • Fork without exec produces no record. Only processes with a recorded exec get exits.
  • Exit codes are reliable only on kernel 6.15+. On 6.2–6.14 they are best-effort, because the parent usually reaps the process before procfs can be read: without the ioctl, only about 24% of exits had codes.
  • A Terminate names the last requested exec. Its parent name is -.
  • The rate limit isn't configurable. Setting changes apply on the next settings poll (about 10 s).
  • An older supervisor can't decode the new event kinds, so don't downgrade the supervisor while exec recording is on.
  • Shorthand doesn't escape process names (a pre-existing issue; follow-up).

Release Notes

Set ocsf_process_exec_enabled to record every program a sandboxed workload runs, and its exit code, as OCSF Process Activity events.

Testing

  • Lockfile checks, rust:format:check, and every CI clippy invocation (--locked, including perf-harness) pass on macOS and the aarch64 Linux target for this commit
  • Unit tests added/updated (zombie and reaped status, cap suppression, managed-child skip, rate limiting, /proc/self resolution); sandbox, sandbox-backend, and supervisor suites pass in an Ubuntu 24.04 guest (kernel 6.8)
  • Manual end-to-end: gateway + Docker sandbox (kernel 7.0), before/after OCSF output above

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Process Activity covered only the processes the runtime starts itself:
the entrypoint and each exec or SSH session. Anything those processes
ran, a shell's commands or an agent's tools, left no record of starting
or finishing.

The workload's seccomp listener now also receives execve and execveat.
When the ocsf_process_exec_enabled setting is on, the runtime records each
exec of an existing executable file before letting it proceed: the
caller's pid, parent pid, and start time, the executable, and the
executables of the caller before the exec and of its parent. The file is
looked up on a dedicated thread so a slow filesystem cannot stall the
broker. The supervisor applies the setting through ReadProcessEvents,
which runtimes advertising openshell.sandbox.process-exec-events honor;
a read sent before the setting is known leaves the runtime's choice as
it is. Each exec becomes a Process Activity Launch with launch type Exec
and an Unknown outcome, because the kernel has not run it yet.

When the runtime records an exec, it opens a pidfd for the caller while
the caller is still blocked in the exec, so the descriptor names that
process even after its pid is reused. An exit observer thread polls
those descriptors and records each exit under the process's last
executable, with its wait status from PIDFD_GET_INFO on kernels that keep
it, or from procfs while the process is still a zombie. Processes the
runtime waits for itself are left to it; orphans it adopts are watched.
The supervisor emits each exit as a Process Activity Terminate whose
actor is the process itself, with exit_code when the status was read and
an Unknown status otherwise.

Execs over a rate of 500 at once and 200 per second are counted rather
than recorded, as are processes beyond the exit observer's limit of a
quarter of the runtime's open-file limit, at most 1024. The count is
reported at most once per second as a Process Activity record with its
number in count; events the runtime evicted before a supervisor that had
already been reading read them are reported the same way.

A process that forks without exec has no record. An exec that fails for
a reason other than a missing or non-executable file is still recorded,
and the record names the file the request named when it was observed.
On kernels before 6.15 the exit status is known only when the runtime
reads it before the parent reaps the process.

Refs #4268

Signed-off-by: Kris Hicks <[email protected]>
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant