Skip to content

Commit ce74e7f

Browse files
TonyLuo-NVclaude
andauthored
fix(sandbox): cap sandbox open file descriptors (nofile) at entrypoint (#4944)
## Summary Caps the sandbox open-file-descriptor limit (`nofile`) at the entrypoint so sandboxes no longer inherit the Docker daemon default `RLIMIT_NOFILE` (~1048576), which can exceed the host runtime limit and enable fd-exhaustion DoS. Restores the invariant that the sandbox is no more privileged than the launching process, hardening `nofile` the same way `nproc` is already hardened. ## Related Issue Closes #4527 <!-- Note: content-overlaps the older, broader #2173 (nproc + nofile); maintainers may cross-link/close as dup. --> ## Changes - Add a shared `harden_resource_limits()` helper in `scripts/lib/sandbox-init.sh` that caps `nproc` (512, #809) and `nofile` (65536, #4527), soft-before-hard, best-effort (warns under `set -euo pipefail`, never aborts startup). - Replace the duplicated inline `nproc`-only blocks in both entrypoints (`scripts/nemoclaw-start.sh` for OpenClaw, `agents/hermes/start.sh` for Hermes) with a single call to the helper, kept at root PID 1 **before** the capsh capability drop and setpriv step-down so the caps are inherited by every descendant and are unraisable by the unprivileged agent (raising a hard RLIMIT needs `CAP_SYS_RESOURCE`, which it never holds). - Tests (`test/sandbox-init.test.ts`): ordered `ulimit`-shim assertion (`-Su 512`/`-Hu 512`/`-Sn 65536`/`-Hn 65536`), best-effort exit-0 + `[SECURITY]` warning assertions, entrypoint grep checks, and a `harden`-before-`drop_capabilities` ordering invariant. - Docs: parallel `nofile` coverage in `docs/deployment/sandbox-hardening.mdx` and `docs/security/best-practices.mdx`, including the runtime `--ulimit nofile=N:N` override and the honest `openshell sandbox connect` residual gap (NVIDIA/OpenShell#1452). ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - [ ] `npx prek run --all-files` passes <!-- CLI test gate hits pre-existing env failures (missing nemoclaw/node_modules/json5 #3118 tests, flaky snapshot timeout) unrelated to this diff; shellcheck/shfmt/gitleaks/docs-to-skills passed --> - [ ] `npm test` passes <!-- same pre-existing env failures; targeted test/sandbox-init.test.ts is 33/33 green --> - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [x] Docs updated for user-facing behavior changes - [ ] `make docs` builds without warnings (doc changes only) <!-- fern check: 0 errors, 2 pre-existing warnings --> - [x] Doc pages follow the style guide (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) <!-- no new pages --> --- Signed-off-by: Tony Luo <[email protected]> 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Feature** * Entrypoints now apply process and open-file descriptor limits earlier during container startup. * **Documentation** * Added "Open File Descriptor Limits" and best-practices guidance, plus runtime examples for enforcing/overriding limits. * **Tests** * Expanded coverage to validate resource-limit hardening and startup ordering of the hardening step. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Tony Luo <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 396eb34 commit ce74e7f

6 files changed

Lines changed: 145 additions & 17 deletions

File tree

‎agents/hermes/start.sh‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,8 @@ fi
2929
# shellcheck source=scripts/lib/sandbox-init.sh
3030
source "$_SANDBOX_INIT"
3131

32-
# Harden: limit process count to prevent fork bombs
33-
if ! ulimit -Su 512 2>/dev/null; then
34-
echo "[SECURITY] Could not set soft nproc limit (container runtime may restrict ulimit)" >&2
35-
fi
36-
if ! ulimit -Hu 512 2>/dev/null; then
37-
echo "[SECURITY] Could not set hard nproc limit (container runtime may restrict ulimit)" >&2
38-
fi
32+
# Harden RLIMITs (nproc #809 + nofile #4527) as root PID 1, before any step-down.
33+
harden_resource_limits
3934

4035
# SECURITY: Lock down PATH
4136
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

‎docs/deployment/sandbox-hardening.mdx‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,23 @@ The startup script (`nemoclaw-start.sh`) applies the same limit.
3131

3232
Adjust the value with the `--ulimit nproc=512:512` flag if you launch with `docker run` directly.
3333

34+
## Open File Descriptor Limits
35+
36+
The same ENTRYPOINT also sets `ulimit -n 65536` to cap the number of open file
37+
descriptors a sandbox user can hold. Without this cap the sandbox inherits the
38+
Docker daemon default (`nofile` ~1048576), which can exceed the host runtime
39+
limit and lets a runaway process exhaust file descriptors. The startup script
40+
(`nemoclaw-start.sh`) applies the same limit.
41+
42+
Adjust the value via the `--ulimit nofile=65536:65536` flag if launching with
43+
`docker run` directly.
44+
45+
Like the process limit, this is applied to the PID 1 entrypoint process tree
46+
(gateway + agent). `openshell sandbox connect` shells are spawned outside that
47+
tree and still inherit the runtime default (tracked upstream in
48+
NVIDIA/OpenShell#1452), so enforce both limits at the container runtime when
49+
that residual matters to you.
50+
3451
## Dropping Linux Capabilities
3552

3653
The NemoClaw entrypoint drops dangerous capabilities from the process bounding set before it starts agent services.
@@ -51,6 +68,7 @@ when you launch the image directly:
5168
docker run --rm \
5269
--cap-drop=ALL \
5370
--ulimit nproc=512:512 \
71+
--ulimit nofile=65536:65536 \
5472
nemoclaw-sandbox
5573
```
5674

@@ -68,6 +86,9 @@ services:
6886
nproc:
6987
soft: 512
7088
hard: 512
89+
nofile:
90+
soft: 65536
91+
hard: 65536
7192
security_opt:
7293
- no-new-privileges:true
7394
read_only: true
@@ -132,4 +153,5 @@ The `test/e2e/e2e-cloud-experimental/checks/04-landlock-readonly.sh` script vali
132153
- [#807](https://github.com/NVIDIA/NemoClaw/issues/807): gcc in sandbox image
133154
- [#808](https://github.com/NVIDIA/NemoClaw/issues/808): netcat in sandbox image
134155
- [#809](https://github.com/NVIDIA/NemoClaw/issues/809): No process limit
156+
- [#4527](https://github.com/NVIDIA/NemoClaw/issues/4527): Cap open file descriptors (nofile)
135157
- [#797](https://github.com/NVIDIA/NemoClaw/issues/797): Drop Linux capabilities

‎docs/security/best-practices.mdx‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -346,6 +346,21 @@ This behavior is best effort: if the container runtime restricts `ulimit` modifi
346346
| Risk if relaxed | Removing or raising the limit makes the sandbox vulnerable to fork-bomb attacks, where a runaway process spawns children until the host runs out of resources. If the entrypoint cannot set the limit (logs `[SECURITY] Could not set soft/hard nproc limit`), the container runs without process limits. |
347347
| Recommendation | Keep the default at 512. If the agent runs workloads that spawn many child processes (such as parallel test runners), increase to 1024 and monitor host resource usage. If the entrypoint logs a warning about ulimit restrictions, set the limit through the container runtime instead. |
348348

349+
### Open File Descriptor Limit
350+
351+
An open file descriptor limit caps the number of files, sockets, and pipes the
352+
sandbox user can hold open at once. The entrypoint sets both soft and hard
353+
limits using `ulimit -n 65536`. This is best-effort: if the container runtime
354+
restricts `ulimit` modification, the entrypoint logs a security warning and
355+
continues without the limit.
356+
357+
| Aspect | Detail |
358+
|---|---|
359+
| Default | 65536 open files, soft and hard (`ulimit -n 65536`), best-effort. |
360+
| What you can change | Increase or decrease the limit with `--ulimit nofile=N:N` in `docker run` or the `ulimits` section in Compose. The runtime-level ulimit takes precedence over the entrypoint's setting. |
361+
| Risk if relaxed | Without this cap the sandbox inherits the Docker daemon default (`nofile` ~1048576). A runaway or hostile process can then open file descriptors until it exhausts them — a denial-of-service that can starve the gateway, the agent, or the host of file handles. If the entrypoint cannot set the limit (logs `[SECURITY] Could not set soft/hard nofile limit`), the container runs without a file-descriptor cap. Ref [#4527](https://github.com/NVIDIA/NemoClaw/issues/4527). |
362+
| Recommendation | Keep the default at 65536. If the agent legitimately keeps many connections or files open, raise it deliberately and monitor host file-descriptor usage. If the entrypoint logs a warning about ulimit restrictions, set the limit through the container runtime instead. |
363+
349364
### Non-Root User
350365

351366
The sandbox runs agent processes as a dedicated `sandbox` user and group.

‎scripts/lib/sandbox-init.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -220,6 +220,34 @@ lock_config_after_write() {
220220
done
221221
}
222222

223+
# ── Resource limits (RLIMITs) ────────────────────────────────────
224+
# Harden RLIMITs at PID 1 (root) so the caps are inherited by every descendant
225+
# and cannot be raised after the privilege step-down. nproc (#809) prevents
226+
# fork bombs; nofile (#4527) caps open file descriptors so the sandbox no
227+
# longer inherits the Docker daemon default (~1048576), which can exceed the
228+
# host runtime limit. Best-effort: some container runtimes (e.g. brev, Docker
229+
# Desktop, WSL) reject ulimit changes with "Invalid argument"; warn but do not
230+
# block startup. Set the soft limit before the hard limit (ordering matters;
231+
# see #951). Hard==soft makes the cap unraisable: raising the hard RLIMIT
232+
# requires CAP_SYS_RESOURCE, which the unprivileged stepped-down agent never
233+
# holds. NOTE: this only covers the PID 1 entrypoint process tree (gateway +
234+
# agent); 'openshell sandbox connect' shells are spawned outside this tree and
235+
# still inherit the runtime default (NVIDIA/OpenShell#1452).
236+
harden_resource_limits() {
237+
if ! ulimit -Su 512 2>/dev/null; then
238+
echo "[SECURITY] Could not set soft nproc limit (container runtime may restrict ulimit)" >&2
239+
fi
240+
if ! ulimit -Hu 512 2>/dev/null; then
241+
echo "[SECURITY] Could not set hard nproc limit (container runtime may restrict ulimit)" >&2
242+
fi
243+
if ! ulimit -Sn 65536 2>/dev/null; then
244+
echo "[SECURITY] Could not set soft nofile limit (container runtime may restrict ulimit)" >&2
245+
fi
246+
if ! ulimit -Hn 65536 2>/dev/null; then
247+
echo "[SECURITY] Could not set hard nofile limit (container runtime may restrict ulimit)" >&2
248+
fi
249+
}
250+
223251
# ── Capability dropping ──────────────────────────────────────────
224252
# CIS Docker Benchmark 5.3: containers should not run with default caps.
225253
# OpenShell manages the container runtime so we cannot pass --cap-drop=ALL

‎scripts/nemoclaw-start.sh‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -92,15 +92,9 @@ fi
9292
# shellcheck source=scripts/lib/sandbox-init.sh
9393
source "$_SANDBOX_INIT"
9494

95-
# Harden: limit process count to prevent fork bombs (ref: #809)
96-
# Best-effort: some container runtimes (e.g., brev) restrict ulimit
97-
# modification, returning "Invalid argument". Warn but don't block startup.
98-
if ! ulimit -Su 512 2>/dev/null; then
99-
echo "[SECURITY] Could not set soft nproc limit (container runtime may restrict ulimit)" >&2
100-
fi
101-
if ! ulimit -Hu 512 2>/dev/null; then
102-
echo "[SECURITY] Could not set hard nproc limit (container runtime may restrict ulimit)" >&2
103-
fi
95+
# Harden RLIMITs (nproc #809 + nofile #4527) as root PID 1, before the capsh
96+
# drop and the setpriv step-down, so the caps are inherited and unraisable.
97+
harden_resource_limits
10498

10599
# PATH was already locked down at the top of this script (before the
106100
# early stderr capture). This comment marks the original location.

‎test/sandbox-init.test.ts‎

Lines changed: 75 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -667,6 +667,77 @@ EOF
667667
});
668668
});
669669

670+
describe("harden_resource_limits", () => {
671+
// Shim `ulimit` (a bash builtin) by overriding it with a function inside the
672+
// sourced body. The function records each invocation so we can assert both
673+
// the nproc (#809) and nofile (#4527) caps are applied, soft-before-hard.
674+
it("applies nproc and nofile soft+hard limits in order", () => {
675+
const { stdout } = runWithLib(
676+
[
677+
// Override the ulimit builtin to record args and succeed.
678+
"ulimit() { printf 'ulimit %s\\n' \"$*\"; return 0; }",
679+
"harden_resource_limits",
680+
].join("\n"),
681+
);
682+
const calls = stdout.split("\n").filter((line) => line.startsWith("ulimit "));
683+
expect(calls).toEqual([
684+
"ulimit -Su 512",
685+
"ulimit -Hu 512",
686+
"ulimit -Sn 65536",
687+
"ulimit -Hn 65536",
688+
]);
689+
});
690+
691+
it("is best-effort: exits 0 and warns when ulimit fails", () => {
692+
// Shim ulimit to always fail. The function must not abort (best-effort)
693+
// and must emit a [SECURITY] warning for each of the four limits.
694+
const { stdout } = runWithLib(
695+
[
696+
"ulimit() { return 1; }",
697+
"harden_resource_limits 2>&1",
698+
'echo "HARDEN_OK"',
699+
].join("\n"),
700+
);
701+
expect(stdout).toContain("HARDEN_OK");
702+
expect(stdout).toContain("Could not set soft nproc limit");
703+
expect(stdout).toContain("Could not set hard nproc limit");
704+
expect(stdout).toContain("Could not set soft nofile limit");
705+
expect(stdout).toContain("Could not set hard nofile limit");
706+
});
707+
});
708+
709+
describe("entrypoints call harden_resource_limits", () => {
710+
// Both entrypoints must delegate RLIMIT hardening to the shared helper and
711+
// must no longer carry the pre-#4527 raw inline `ulimit -Su 512` block.
712+
for (const rel of ["../scripts/nemoclaw-start.sh", "../agents/hermes/start.sh"]) {
713+
it(`${rel} calls harden_resource_limits and has no raw inline nproc block`, () => {
714+
const src = readFileSync(join(import.meta.dirname, rel), "utf-8");
715+
expect(src).toContain("harden_resource_limits");
716+
expect(src).not.toContain("ulimit -Su 512");
717+
expect(src).not.toContain("ulimit -Hu 512");
718+
});
719+
}
720+
721+
// SECURITY (#4527): the RLIMIT caps are only unraisable if they are set
722+
// while still root PID 1, BEFORE drop_capabilities (capsh) and the
723+
// setpriv/gosu step-down. A refactor that moved the harden call after the
724+
// privilege drop would turn it into dead code (cap set as the unprivileged
725+
// agent, hard limit no longer lowered) while every other test stayed green.
726+
// Pin the ordering so that regression is caught.
727+
for (const rel of ["../scripts/nemoclaw-start.sh", "../agents/hermes/start.sh"]) {
728+
it(`${rel} calls harden_resource_limits before drop_capabilities`, () => {
729+
const src = readFileSync(join(import.meta.dirname, rel), "utf-8");
730+
// Anchor to executable command lines, not free-text, so a comment
731+
// mentioning either name cannot satisfy (or break) the ordering check.
732+
const hardenIdx = src.match(/^\s*harden_resource_limits\s*$/m)?.index ?? -1;
733+
const dropIdx = src.match(/^\s*drop_capabilities\b.*$/m)?.index ?? -1;
734+
expect(hardenIdx).toBeGreaterThanOrEqual(0);
735+
expect(dropIdx).toBeGreaterThanOrEqual(0);
736+
expect(hardenIdx).toBeLessThan(dropIdx);
737+
});
738+
}
739+
});
740+
670741
describe("init_step_down_prefixes", () => {
671742
it("falls back to gosu when setpriv is unavailable", () => {
672743
// Source-time init runs before our test body, so re-run it with a
@@ -832,7 +903,10 @@ EOF
832903
it("nemoclaw-start.sh sources sandbox-init.sh", () => {
833904
const src = readFileSync(join(import.meta.dirname, "../scripts/nemoclaw-start.sh"), "utf-8");
834905
const start = src.indexOf("_SANDBOX_INIT=");
835-
const end = src.indexOf("# Harden: limit process count", start);
906+
// Bound the source block at the harden_resource_limits call line itself
907+
// (executable, stable) rather than a free-text comment that may be reworded.
908+
const hardenCallFromStart = src.slice(start).match(/^\s*harden_resource_limits\s*$/m);
909+
const end = hardenCallFromStart ? start + (hardenCallFromStart.index ?? 0) : -1;
836910
if (start === -1 || end === -1 || end <= start) {
837911
throw new Error("Expected sandbox-init source block in scripts/nemoclaw-start.sh");
838912
}

0 commit comments

Comments
 (0)