Skip to content

refactor(jailer): copy only shim + libkrunfw to box bin/ directory#337

Merged
DorianZheng merged 3 commits intomainfrom
refactor/remove-bundled-lib-copy-from-jailer
Mar 4, 2026
Merged

refactor(jailer): copy only shim + libkrunfw to box bin/ directory#337
DorianZheng merged 3 commits intomainfrom
refactor/remove-bundled-lib-copy-from-jailer

Conversation

@DorianZheng
Copy link
Copy Markdown
Member

Summary

  • Stop copying libkrun, libgvproxy, and other bundled libraries to box_dir/bin/ during box creation
  • Keep copying only the shim binary (for memory isolation, Firecracker pattern) and libkrunfw (dlopen'd by libkrun at runtime via the shim's rpath)
  • Add original binary's parent directory to the sandbox's read-only path list so the dynamic linker can find remaining libraries from their original location
  • Remove dead LD_LIBRARY_PATH setting from Jailer::command() — it was overwritten by configure_library_env in spawn.rs
  • Fix pre-existing compile error in clone_export_import.rs (ParallelRuntime → standard test setup pattern)

Why libkrunfw must be copied

On macOS, sandbox-exec is a SIP-protected system binary that strips DYLD_* environment variables. Since libkrun loads libkrunfw via dlopen("libkrunfw.5.dylib"), the library must be physically present in a directory on the shim's rpath (i.e., bin/). Without it, the VM fails to start with Couldn't find or load libkrunfw.5.dylib.

Test plan

  • cargo clippy -p boxlite --tests -- -D warnings passes (zero warnings)
  • cargo fmt --all --check clean
  • Integration tests pass (107/111 — 4 snapshot test failures are pre-existing on main)
  • Verify box_dir/bin/ contains only boxlite-shim + libkrunfw.* (no libkrun, libgvproxy)

The jailer previously copied libkrun, libkrunfw, and libgvproxy alongside
the shim binary into each box's bin/ directory. This was unnecessary since
configure_library_env in spawn.rs already handles library discovery via
dladdr, and the LD_LIBRARY_PATH set in command() was immediately
overwritten.

Changes:
- Remove BUNDLED_LIB_PATTERNS constants and copy_bundled_libraries()
- Simplify copy_shim_to_box() to only copy the shim binary
- Grant sandbox read access to original binary's parent directory so the
  dynamic linker can load libraries from their original location
- Remove dead LD_LIBRARY_PATH setting from Jailer::command()
- Update comments to reflect shim-only copying
- Fix pre-existing ParallelRuntime compile error in clone_export_import tests
- Fix pre-existing cargo fmt issues in lib.rs and go-sdk lib.rs

This reduces per-box disk usage by ~20MB+ and eliminates unnecessary I/O
during box creation.
On macOS, sandbox-exec (SIP-protected) strips DYLD_* env vars, so
dlopen("libkrunfw.5.dylib") inside libkrun can only find the library
via the shim binary's rpath. Create symlinks (not copies) in bin/ so
rpath resolution works without duplicating multi-MB dylibs per box.
Replace the symlink approach with copying only libkrunfw alongside the
shim binary. libkrun dlopen's libkrunfw at runtime and the shim's rpath
resolves to bin/ — on macOS, DYLD_* env vars are stripped by SIP when
going through sandbox-exec, so the library must be physically present.

Other bundled libraries (libkrun, libgvproxy) are no longer copied:
libkrun is statically linked on macOS, and libgvproxy is a separate
process found via binary discovery.
@DorianZheng DorianZheng merged commit 578d366 into main Mar 4, 2026
17 of 19 checks passed
@DorianZheng DorianZheng deleted the refactor/remove-bundled-lib-copy-from-jailer branch March 4, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant