Skip to content

Commit 5db9432

Browse files
Dongni-Yangclaude
andauthored
fix(sandbox): add opt-in fail-closed cap-drop enforcement (#4707)
## Summary On hosts without CAP_SETPCAP (GitHub runners, Brev shadecloud, Colossus Ubuntu 24.04, Docker Desktop, WSL), `drop_capabilities` cannot run `capsh --drop`, so dangerous bounding-set caps survive and the agent boots with a weaker posture than the security model assumes. This adds an opt-in fail-closed guard (`NEMOCLAW_REQUIRE_CAP_DROP=1`) that verifies the agent process tree's actual bounding set and refuses to start if dangerous capabilities remain. ## Related Issue Refs #3280 <!-- Refs (not Closes): this hardens the agent process tree only. The `nemoclaw connect` shell inherits the container's OCI bounding set and is not affected; tightening that needs cap_drop at sandbox-create, tracked upstream in NVIDIA/OpenShell#1452. #3280 should stay open pending that. --> ## Changes - `scripts/lib/sandbox-init.sh`: - `enforce_cap_drop_if_required` runs on **every** path out of `drop_capabilities` and verifies the **actual** `CapBnd` rather than trusting the `NEMOCLAW_CAPS_DROPPED` sentinel (closes a strict-mode bypass). - A single `DANGEROUS_CAPS` list feeds both the `capsh --drop` arguments and the verifier, so the drop-set and the verify-set (all 10 caps) cannot drift. - A non-empty but unparseable `CapBnd` is treated as unverifiable: refuse in strict mode, warn in default mode — never a raw bash arithmetic error. - `setpriv`-based privilege step-down strips the remaining load-bearing caps atomically with the setuid transition (falls back to `gosu` when `setpriv`/`CAP_SETPCAP` are unavailable). - Default stays **warn-and-continue** (zero regression — the inverse of the reverted #4266); operators set `NEMOCLAW_REQUIRE_CAP_DROP=1` to refuse-to-start. - `test/sandbox-init.test.ts`: repro + strict-mode + sentinel-bypass + malformed-`CapBnd` coverage, seam-driven (`NEMOCLAW_PROC_STATUS`) and deterministic. - `docs/security/best-practices.mdx`, `docs/deployment/sandbox-hardening.mdx`: document `NEMOCLAW_REQUIRE_CAP_DROP` and the verified scope. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - [x] `npx prek run --all-files` passes - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [x] Docs updated for user-facing behavior changes <!-- test/sandbox-init.test.ts 37/37 (vitest). pre-commit + pre-push hooks (shellcheck, shfmt, TypeScript CLI, Test CLI, commitlint, DCO, gitleaks) pass on each push; full PR CI green incl. test-e2e-sandbox / test-non-root-sandbox-smoke. --> --- Signed-off-by: Dongni Yang <[email protected]> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Opt-in strict verification via NEMOCLAW_REQUIRE_CAP_DROP=1: agent can refuse to start if dangerous Linux capabilities remain. * **Behavior** * More accurate detection and clearer diagnostics of residual bounding-set capabilities; improved sandbox step-down argument handling. * **Tests** * Added deterministic tests covering warn-and-continue, strict-mode refusal, and various verification edge cases. * **Documentation** * Updated docs explaining capability-drop behavior and the strict verification option. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Dongni Yang <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent fcacd5e commit 5db9432

4 files changed

Lines changed: 376 additions & 27 deletions

File tree

‎docs/deployment/sandbox-hardening.mdx‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@ It removes `CAP_SYS_ADMIN`, `CAP_SYS_PTRACE`, `CAP_NET_RAW`,
3939
`CAP_MKNOD`, `CAP_AUDIT_WRITE`, and `CAP_NET_BIND_SERVICE`.
4040
When `setpriv` is available, the entrypoint also removes the remaining privilege-separation capabilities during the switch from root to the `sandbox` and `gateway` users.
4141

42+
The bounding-set drop is best effort: if `capsh` or `CAP_SETPCAP` is unavailable the entrypoint logs a warning and continues with the runtime-provided capability set.
43+
If `setpriv` is unavailable, the entrypoint falls back to `gosu`.
44+
To make the drop fail-closed instead, set `NEMOCLAW_REQUIRE_CAP_DROP=1` in the entrypoint environment: the agent then refuses to start unless the agent process tree's bounding set is verified free of the dangerous capabilities.
45+
This is opt-in because hosts that cannot drop capabilities (no `CAP_SETPCAP` — many cloud VMs, Docker Desktop, WSL) are common, and the check covers the agent process tree only.
46+
4247
For defense-in-depth, also drop all Linux capabilities at the container runtime
4348
when you launch the image directly:
4449

‎docs/security/best-practices.mdx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,12 @@ During `setpriv` step-down, the child process also loses `cap_setuid`, `cap_setg
293293

294294
This behavior is best effort: if `capsh` is not available or `CAP_SETPCAP` is not in the bounding set, the entrypoint logs a warning and continues with the default capability set.
295295
If `setpriv` is unavailable, the entrypoint falls back to `gosu` and logs a warning that the remaining bounding-set capabilities were retained for the child process.
296+
297+
To make the drop fail-closed instead of best-effort, set `NEMOCLAW_REQUIRE_CAP_DROP=1` in the entrypoint environment.
298+
The agent then refuses to start unless the agent process tree's bounding set is verified free of the dangerous capabilities, so it will not boot on a host whose bounding set still holds them — typically one that cannot perform the drop (no `CAP_SETPCAP`, or `capsh` missing) and was not given a clean bounding set by the container runtime.
299+
This is opt-in because such hosts are common (many cloud VMs, Docker Desktop, WSL); leaving it unset preserves the best-effort default.
300+
The check covers the agent process tree only — a `nemoclaw connect` shell is spawned by the container runtime outside that tree and is not affected (tracked in [NVIDIA/OpenShell#1452](https://github.com/NVIDIA/OpenShell/issues/1452)).
301+
296302
For additional protection, pass `--cap-drop=ALL` with `docker run` or Compose (see [Sandbox Hardening](../manage-sandboxes/sandbox-hardening)).
297303

298304
| Aspect | Detail |

‎scripts/lib/sandbox-init.sh‎

Lines changed: 135 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -214,12 +214,44 @@ lock_config_after_write() {
214214
# cap_kill — sandbox user signals gateway-user processes via the UID
215215
# separation enforced by the entrypoint (see test 13 in
216216
# e2e-gateway-isolation.sh).
217+
# When the runtime cannot drop the bounding set (no CAP_SETPCAP, or capsh
218+
# missing), the default is to warn and continue. Set NEMOCLAW_REQUIRE_CAP_DROP=1
219+
# to make that case fail-closed instead — see enforce_cap_drop_if_required.
220+
#
217221
# Ref: https://github.com/NVIDIA/NemoClaw/issues/797
218222
# https://github.com/NVIDIA/NemoClaw/issues/3280
223+
# https://github.com/NVIDIA/OpenShell/issues/1452 (connect-shell scope)
219224
#
220225
# Usage:
221226
# drop_capabilities /usr/local/bin/nemoclaw-start "$@"
222227
#
228+
# Single source of truth for the dangerous capabilities the entrypoint drops
229+
# and (in strict mode) verifies are gone. "bit:name" pairs; bit numbers per
230+
# /usr/include/linux/capability.h. Both the capsh --drop list and
231+
# dangerous_caps_in_capbnd() derive from this array, so the drop-set and the
232+
# strict-mode verify-set cannot drift apart (issue #3280).
233+
DANGEROUS_CAPS=(
234+
"21:cap_sys_admin"
235+
"19:cap_sys_ptrace"
236+
"13:cap_net_raw"
237+
"1:cap_dac_override"
238+
"18:cap_sys_chroot"
239+
"4:cap_fsetid"
240+
"31:cap_setfcap"
241+
"27:cap_mknod"
242+
"29:cap_audit_write"
243+
"10:cap_net_bind_service"
244+
)
245+
246+
# Comma-separated capability names for `capsh --drop`, derived from DANGEROUS_CAPS.
247+
dangerous_caps_drop_list() {
248+
local entry out=""
249+
for entry in "${DANGEROUS_CAPS[@]}"; do
250+
out="${out:+$out,}${entry#*:}"
251+
done
252+
printf '%s' "$out"
253+
}
254+
223255
# The first argument is the absolute path to the entrypoint script to
224256
# re-exec via capsh. Remaining arguments are forwarded.
225257
drop_capabilities() {
@@ -232,14 +264,102 @@ drop_capabilities() {
232264
if capsh --has-p=cap_setpcap 2>/dev/null; then
233265
export NEMOCLAW_CAPS_DROPPED=1
234266
exec capsh \
235-
--drop=cap_sys_admin,cap_sys_ptrace,cap_net_raw,cap_dac_override,cap_sys_chroot,cap_fsetid,cap_setfcap,cap_mknod,cap_audit_write,cap_net_bind_service \
267+
--drop="$(dangerous_caps_drop_list)" \
236268
-- -c "exec $entrypoint \"\$@\"" -- "$@"
237-
else
238-
report_residual_capabilities
239269
fi
270+
# CAP_SETPCAP missing (or the exec above failed): the drop could not run.
271+
# Surface the residual bounding-set caps in the log.
272+
report_residual_capabilities || true
240273
elif [ "${NEMOCLAW_CAPS_DROPPED:-}" != "1" ]; then
241274
echo "[SECURITY WARNING] capsh not available — running with default capabilities" >&2
242275
fi
276+
277+
# Opt-in fail-closed gate (issue #3280). Deliberately runs on EVERY path,
278+
# including when NEMOCLAW_CAPS_DROPPED is already set: it verifies the actual
279+
# bounding set rather than trusting that sentinel, so an inherited marker
280+
# cannot mask a drop that never happened.
281+
enforce_cap_drop_if_required
282+
}
283+
284+
# Pure decode: given a CapBnd hex string, echo the comma-separated list of the
285+
# dangerous capabilities present (empty string if none). Factored out so the
286+
# residual diagnostic, the strict-mode gate, and the unit tests all share one
287+
# implementation instead of re-deriving the bit math.
288+
#
289+
# Bash arithmetic handles 64-bit ints on 64-bit platforms; CAP_LAST_CAP is ~41
290+
# today, well within range. Avoids a gawk-strtonum dependency.
291+
#
292+
# Returns nonzero with no output if the hex is empty or malformed, so callers
293+
# can treat "could not parse" the same as "could not read" instead of silently
294+
# treating an unparseable bounding set as clean (issue #3280).
295+
dangerous_caps_in_capbnd() {
296+
local cap_bnd_hex="$1" val entry bit name present=""
297+
case "$cap_bnd_hex" in
298+
"" | *[!0-9A-Fa-f]*) return 1 ;;
299+
esac
300+
val=$((16#$cap_bnd_hex))
301+
for entry in "${DANGEROUS_CAPS[@]}"; do
302+
bit="${entry%%:*}"
303+
name="${entry#*:}"
304+
if [ $(((val >> bit) & 1)) -ne 0 ]; then
305+
present="${present:+$present,}$name"
306+
fi
307+
done
308+
printf '%s' "$present"
309+
}
310+
311+
# Opt-in fail-closed enforcement (issue #3280). When NEMOCLAW_REQUIRE_CAP_DROP=1
312+
# the sandbox refuses to start unless the bounding set is provably free of the
313+
# dangerous capabilities. It verifies by reading the ACTUAL CapBnd — NOT by
314+
# trusting the NEMOCLAW_CAPS_DROPPED sentinel, which an inherited environment
315+
# could forge to bypass the gate.
316+
#
317+
# DEFAULT (unset) IS WARN-AND-CONTINUE — no host loses the ability to boot. This
318+
# is the lesson of #4266/#4341: a default-fail-closed drop broke EVERY host that
319+
# does not grant CAP_SETPCAP (GitHub runners, Brev shadecloud, Colossus Ubuntu
320+
# 24.04, Docker Desktop, WSL) and was reverted within hours. Inverting the
321+
# default to opt-in keeps that regression off by default.
322+
#
323+
# Scope: the AGENT process tree only. A `nemoclaw connect` shell is spawned by
324+
# the container runtime outside that tree and inherits the container's OCI
325+
# bounding set; tightening that requires cap_drop at sandbox create, tracked
326+
# upstream in NVIDIA/OpenShell#1452.
327+
#
328+
# Test seam: NEMOCLAW_PROC_STATUS overrides the status source so unit tests can
329+
# feed a known CapBnd fixture without a real /proc.
330+
enforce_cap_drop_if_required() {
331+
[ "${NEMOCLAW_REQUIRE_CAP_DROP:-}" = "1" ] || return 0
332+
333+
local status_path="${NEMOCLAW_PROC_STATUS:-/proc/self/status}"
334+
local cap_bnd_hex present reason=""
335+
cap_bnd_hex=$(awk '/^CapBnd:/{print $2}' "$status_path" 2>/dev/null || true)
336+
if [ -z "$cap_bnd_hex" ]; then
337+
# Cannot verify → in strict mode, refuse rather than assume safety.
338+
reason="could not read bounding set from ${status_path} — cannot verify drop"
339+
elif ! present="$(dangerous_caps_in_capbnd "$cap_bnd_hex")"; then
340+
# Non-empty but unparseable CapBnd is equally unverifiable → refuse.
341+
reason="could not parse bounding set (CapBnd=${cap_bnd_hex}) — cannot verify drop"
342+
elif [ -n "$present" ]; then
343+
reason="dangerous caps remain in bounding set (CapBnd=${cap_bnd_hex}): ${present}"
344+
fi
345+
[ -n "$reason" ] || return 0
346+
347+
cat >&2 <<'EOF'
348+
349+
┌─ [SECURITY] Refusing to start sandbox: bounding-set capability drop failed ──
350+
│
351+
│ NEMOCLAW_REQUIRE_CAP_DROP=1 is set, so NemoClaw refuses to start a sandbox
352+
│ that still holds dangerous bounding-set capabilities. The runtime could not
353+
│ drop them (capsh or CAP_SETPCAP unavailable on this host), so they remain.
354+
│
355+
│ To run anyway with the weaker (warn-only) posture, unset the variable:
356+
│ unset NEMOCLAW_REQUIRE_CAP_DROP
357+
│
358+
│ Tracking: https://github.com/NVIDIA/NemoClaw/issues/3280
359+
└──────────────────────────────────────────────────────────────────────────────
360+
EOF
361+
echo "[SECURITY] ${reason}" >&2
362+
exit 1
243363
}
244364

245365
# Emit a loud diagnostic when capsh-based dropping is unavailable so that
@@ -249,31 +369,19 @@ drop_capabilities() {
249369
report_residual_capabilities() {
250370
echo "[SECURITY] CAP_SETPCAP not available — cannot drop bounding-set caps via capsh" >&2
251371

252-
local cap_bnd_hex val name bit present_caps=""
253-
if ! cap_bnd_hex=$(awk '/^CapBnd:/{print $2}' /proc/self/status 2>/dev/null) \
372+
local status_path="${NEMOCLAW_PROC_STATUS:-/proc/self/status}"
373+
local cap_bnd_hex present
374+
if ! cap_bnd_hex=$(awk '/^CapBnd:/{print $2}' "$status_path" 2>/dev/null) \
254375
|| [ -z "$cap_bnd_hex" ]; then
255-
echo "[SECURITY] Could not read /proc/self/status — residual caps unknown" >&2
376+
echo "[SECURITY] Could not read ${status_path} — residual caps unknown" >&2
256377
return 0
257378
fi
258379
echo "[SECURITY] Residual CapBnd=${cap_bnd_hex}" >&2
259380

260-
# Bash arithmetic handles 64-bit ints on 64-bit platforms; CAP_LAST_CAP
261-
# is ~41 today, well within range. Avoids a gawk-strtonum dependency.
262-
val=$((16#$cap_bnd_hex))
263-
for entry in \
264-
"21:cap_sys_admin" \
265-
"19:cap_sys_ptrace" \
266-
"13:cap_net_raw" \
267-
"1:cap_dac_override" \
268-
"10:cap_net_bind_service"; do
269-
bit="${entry%%:*}"
270-
name="${entry#*:}"
271-
if [ $(((val >> bit) & 1)) -ne 0 ]; then
272-
present_caps="${present_caps:+$present_caps,}$name"
273-
fi
274-
done
275-
if [ -n "$present_caps" ]; then
276-
echo "[SECURITY] Dangerous caps remain in bounding set: ${present_caps}" >&2
381+
if ! present="$(dangerous_caps_in_capbnd "$cap_bnd_hex")"; then
382+
echo "[SECURITY] Could not parse CapBnd=${cap_bnd_hex} — residual caps unknown" >&2
383+
elif [ -n "$present" ]; then
384+
echo "[SECURITY] Dangerous caps remain in bounding set: ${present}" >&2
277385
fi
278386
}
279387

@@ -331,13 +439,13 @@ init_step_down_prefixes() {
331439
local drop="-setuid,-setgid,-fowner,-chown,-kill"
332440
# shellcheck disable=SC2034 # consumed by entrypoint scripts (cross-file)
333441
STEP_DOWN_PREFIX_SANDBOX=(
334-
setpriv --reuid=sandbox --regid=sandbox --init-groups
335-
--bounding-set="$drop" --
442+
setpriv "--reuid=sandbox" "--regid=sandbox" --init-groups
443+
"--bounding-set=$drop" --
336444
)
337445
# shellcheck disable=SC2034 # consumed by entrypoint scripts (cross-file)
338446
STEP_DOWN_PREFIX_GATEWAY=(
339-
setpriv --reuid=gateway --regid=gateway --init-groups
340-
--bounding-set="$drop" --
447+
setpriv "--reuid=gateway" "--regid=gateway" --init-groups
448+
"--bounding-set=$drop" --
341449
)
342450
else
343451
echo "[SECURITY WARNING] setpriv or CAP_SETPCAP unavailable — falling back to gosu (bounding set will retain cap_setuid/setgid/fowner/chown/kill — issue #3280)" >&2

0 commit comments

Comments
 (0)