Skip to content

Commit f763611

Browse files
committed
refactor(docker): use native rust builds for split gateway/supervisor images
Drop the in-Docker BUILD_FROM_SOURCE path so both images consume only prebuilt binaries staged natively via tasks/scripts/stage-prebuilt-binaries.sh. This mirrors what CI does and reuses the host's cargo target cache and sccache across rebuilds. - Dockerfile.gateway: nvcr.io/nvidia/distroless/cc:v4.0.4 base (the 4.0.0 tag does not exist on nvcr.io; the registry uses a v prefix). GNU-linked binary copied to /usr/local/bin. - Dockerfile.supervisor: scratch base, static musl binary. Static linkage lets the image stay scratch while still being executable as a Kubernetes init container. - skaffold.yaml: each artifact invokes tasks/scripts/docker-build-image.sh, which stages the binary natively (cargo / cargo-zigbuild) and then builds the image. Drops the cross-build.sh dependency from the supervisor build. - seccomp.rs: add a local SYS_kexec_file_load constant for musl/aarch64. libc 0.2.185 omits the symbol from its musl/aarch64 bindings, so the supervisor's seccomp filter previously failed to compile for that target. - architecture/build.md: describe the native-first pipeline and per-image runtime choices. Local validation: gateway image 101MB (was 194MB), supervisor image 21.7MB. helm:skaffold:run deploys cleanly; the static musl supervisor binary runs correctly in a non-glibc agent container.
1 parent 2387d04 commit f763611

5 files changed

Lines changed: 82 additions & 118 deletions

File tree

‎architecture/build.md‎

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,14 +22,29 @@ Sandbox community images are built outside this repository.
2222

2323
## Container Builds
2424

25-
The Docker image pipeline stages prebuilt Rust binaries, then builds the gateway
26-
image from `deploy/docker/Dockerfile.gateway` and the supervisor image from
27-
`deploy/docker/Dockerfile.supervisor`. CI builds native artifacts on the target
28-
architecture, stages them under `deploy/docker/.build/`, and then uses Buildx to
29-
publish per-architecture images and multi-architecture tags. The gateway image
30-
uses the NVIDIA distroless C/C++ runtime. The supervisor image remains
31-
`scratch`, so the staged `openshell-sandbox` image binary is built as a static
32-
musl binary.
25+
The Docker image pipeline is a two-step flow: build the Rust binary natively
26+
for the target architecture, then assemble the container image from the
27+
prebuilt binary. The gateway image is built from `deploy/docker/Dockerfile.gateway`
28+
and the supervisor image from `deploy/docker/Dockerfile.supervisor`. Neither
29+
Dockerfile compiles Rust — both copy a staged binary out of
30+
`deploy/docker/.build/prebuilt-binaries/<arch>/` into the final image.
31+
32+
Binary staging is driven by `tasks/scripts/stage-prebuilt-binaries.sh`, which
33+
runs `cargo build` natively on a matching host or `cargo zigbuild` when
34+
cross-compiling. CI invokes the same staging step via the
35+
`rust-native-build.yml` workflow (per-architecture, per-component) and uploads
36+
the result as an artifact that the image build job downloads back into the
37+
staging directory before running Buildx.
38+
39+
Runtime layout:
40+
41+
- **Gateway**: `nvcr.io/nvidia/distroless/cc` base, GNU-linked binary at
42+
`/usr/local/bin/openshell-gateway`, runs as UID/GID `65532:65532`.
43+
- **Supervisor**: `scratch` base, static musl binary at `/openshell-sandbox`.
44+
Static linkage is required because the image is mounted/extracted into
45+
sandbox environments (Docker extraction, Podman image volumes, Kubernetes
46+
init-container copy-self) and cannot rely on a dynamic loader.
47+
3348
Gateway image builds bake the corresponding supervisor image tag into the
3449
gateway binary so Docker sandboxes do not depend on `:latest` by default.
3550
Package formulas also pin Docker supervisor extraction to the matching release

‎crates/openshell-sandbox/src/sandbox/linux/seccomp.rs‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,16 @@ use tracing::debug;
2525
/// Value of `SECCOMP_SET_MODE_FILTER` (linux/seccomp.h).
2626
const SECCOMP_SET_MODE_FILTER: u64 = 1;
2727

28+
// libc 0.2.185 omits `SYS_kexec_file_load` from the musl/aarch64 bindings even
29+
// though the kernel exposes syscall 294. Fall back to the literal so the
30+
// supervisor's seccomp filter still blocks fileless kernel-image loads when
31+
// built statically against musl on aarch64.
32+
#[cfg(all(target_arch = "aarch64", target_env = "musl"))]
33+
#[allow(non_upper_case_globals)]
34+
const SYS_kexec_file_load: libc::c_long = 294;
35+
#[cfg(not(all(target_arch = "aarch64", target_env = "musl")))]
36+
use libc::SYS_kexec_file_load;
37+
2838
/// Apply the supervisor seccomp filter across the running process.
2939
///
3040
/// This runs after privileged startup helpers complete and synchronizes the
@@ -81,7 +91,7 @@ fn build_supervisor_prelude_rules() -> BTreeMap<i64, Vec<SeccompRule>> {
8191
libc::SYS_finit_module,
8292
libc::SYS_delete_module,
8393
libc::SYS_kexec_load,
84-
libc::SYS_kexec_file_load,
94+
SYS_kexec_file_load,
8595
] {
8696
rules.entry(syscall).or_default();
8797
}
@@ -423,7 +433,7 @@ mod tests {
423433
libc::SYS_finit_module,
424434
libc::SYS_delete_module,
425435
libc::SYS_kexec_load,
426-
libc::SYS_kexec_file_load,
436+
SYS_kexec_file_load,
427437
] {
428438
assert!(
429439
filter_rules.contains_key(&syscall),

‎deploy/docker/Dockerfile.gateway‎

Lines changed: 16 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -5,54 +5,30 @@
55

66
# Gateway image build.
77
#
8-
# Rust binaries are normally built natively before the image build and staged at:
8+
# The Rust binary is built natively before this image build runs and staged at:
99
# deploy/docker/.build/prebuilt-binaries/<arch>/openshell-gateway
1010
#
11-
# Local dev callers such as Skaffold can pass BUILD_FROM_SOURCE=1 to compile the
12-
# gateway inside Docker instead.
13-
14-
ARG BUILD_FROM_SOURCE=0
15-
ARG GATEWAY_BASE_IMAGE=nvcr.io/nvidia/distroless/cc:4.0.0
16-
17-
FROM rust:1.95.0-slim-bookworm AS rust-builder
18-
19-
RUN apt-get update && apt-get install -y --no-install-recommends \
20-
build-essential \
21-
ca-certificates \
22-
cmake \
23-
libssl-dev \
24-
pkg-config \
25-
&& rm -rf /var/lib/apt/lists/*
26-
27-
WORKDIR /build
28-
29-
COPY Cargo.toml Cargo.lock ./
30-
COPY crates/ crates/
31-
COPY proto/ proto/
32-
COPY providers/ providers/
33-
34-
RUN --mount=type=cache,target=/usr/local/cargo/registry \
35-
--mount=type=cache,target=/build/target \
36-
cargo build --release \
37-
--features "openshell-core/dev-settings" \
38-
-p openshell-server \
39-
--bin openshell-gateway \
40-
&& mkdir -p /build/out \
41-
&& install -m 0755 target/release/openshell-gateway /build/out/openshell-gateway
42-
43-
FROM scratch AS gateway-binary-0
44-
ARG TARGETARCH
45-
COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-gateway /build/out/openshell-gateway
46-
47-
FROM rust-builder AS gateway-binary-1
11+
# Use tasks/scripts/docker-build-image.sh gateway (or `mise run build:docker:gateway`)
12+
# to stage the binary and build the image in one step. CI builds the binary
13+
# per-architecture via the `rust-native-build.yml` workflow and uploads it as
14+
# an artifact, which is downloaded into the same staging directory before the
15+
# image build job runs.
16+
#
17+
# The runtime is `nvcr.io/nvidia/distroless/cc:4.0.0`, which provides glibc and
18+
# the dynamic loader needed by the GNU-linked gateway binary while keeping the
19+
# attack surface small.
4820

49-
FROM gateway-binary-${BUILD_FROM_SOURCE} AS gateway-binary
21+
ARG GATEWAY_BASE_IMAGE=nvcr.io/nvidia/distroless/cc:v4.0.4
5022

5123
FROM ${GATEWAY_BASE_IMAGE} AS gateway
5224

25+
ARG TARGETARCH
26+
5327
WORKDIR /app
5428

55-
COPY --from=gateway-binary /build/out/openshell-gateway /usr/local/bin/openshell-gateway
29+
# --chmod=755 preserves the executable bit through actions/upload-artifact +
30+
# download-artifact, which strip exec perms during the roundtrip.
31+
COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-gateway /usr/local/bin/openshell-gateway
5632

5733
USER 65532:65532
5834
EXPOSE 8080

‎deploy/docker/Dockerfile.supervisor‎

Lines changed: 14 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -5,60 +5,26 @@
55

66
# Supervisor image build.
77
#
8-
# The final image intentionally stays `scratch`: it only carries the static
9-
# openshell-sandbox binary used by Docker extraction, Podman image volumes, and
10-
# the Kubernetes init container copy-self path.
8+
# The final image is `scratch`: it only carries the static `openshell-sandbox`
9+
# binary used by Docker extraction, Podman image volumes, and the Kubernetes
10+
# init container copy-self path. A static musl binary lets the image stay
11+
# `scratch` while still being executable as an init container.
1112
#
12-
# Rust binaries are normally built natively before the image build and staged at:
13+
# The Rust binary is built natively before this image build runs and staged at:
1314
# deploy/docker/.build/prebuilt-binaries/<arch>/openshell-sandbox
1415
#
15-
# Local dev callers such as Skaffold can pass BUILD_FROM_SOURCE=1 to compile the
16-
# supervisor inside Docker instead.
16+
# Use tasks/scripts/docker-build-image.sh supervisor (or `mise run build:docker:supervisor`)
17+
# to stage the binary and build the image in one step. CI builds the binary
18+
# per-architecture via the `rust-native-build.yml` workflow (with the musl
19+
# target) and uploads it as an artifact, which is downloaded into the same
20+
# staging directory before the image build job runs.
1721

18-
ARG BUILD_FROM_SOURCE=0
19-
20-
FROM rust:1.95.0-slim-bookworm AS rust-builder
21-
ARG TARGETARCH
22-
ARG BUILDARCH
23-
24-
RUN apt-get update && apt-get install -y --no-install-recommends \
25-
build-essential \
26-
ca-certificates \
27-
pkg-config \
28-
&& rm -rf /var/lib/apt/lists/*
29-
30-
COPY deploy/docker/cross-build.sh /usr/local/bin/cross-build.sh
31-
RUN . /usr/local/bin/cross-build.sh \
32-
&& install_musl_toolchain \
33-
&& add_musl_target
34-
35-
WORKDIR /build
36-
37-
COPY Cargo.toml Cargo.lock ./
38-
COPY crates/ crates/
39-
COPY proto/ proto/
40-
COPY providers/ providers/
41-
42-
RUN --mount=type=cache,target=/usr/local/cargo/registry \
43-
--mount=type=cache,target=/build/target \
44-
. /usr/local/bin/cross-build.sh \
45-
&& cargo_musl_build --release \
46-
--features "openshell-core/dev-settings" \
47-
-p openshell-sandbox \
48-
--bin openshell-sandbox \
49-
&& mkdir -p /build/out \
50-
&& install -m 0755 "$(musl_output_dir release)/openshell-sandbox" /build/out/openshell-sandbox
22+
FROM scratch AS supervisor
5123

52-
FROM scratch AS supervisor-binary-0
5324
ARG TARGETARCH
54-
COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /build/out/openshell-sandbox
55-
56-
FROM rust-builder AS supervisor-binary-1
57-
58-
FROM supervisor-binary-${BUILD_FROM_SOURCE} AS supervisor-binary
59-
60-
FROM scratch AS supervisor
6125

62-
COPY --from=supervisor-binary /build/out/openshell-sandbox /openshell-sandbox
26+
# --chmod=755 preserves the executable bit through actions/upload-artifact +
27+
# download-artifact, which strip exec perms during the roundtrip.
28+
COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /openshell-sandbox
6329

6430
ENTRYPOINT ["/openshell-sandbox"]

‎deploy/helm/openshell/skaffold.yaml‎

Lines changed: 17 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
11
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
# SPDX-License-Identifier: Apache-2.0
33

4-
# Local dev: builds gateway + supervisor images using split Dockerfiles with
5-
# BUILD_FROM_SOURCE=1, which compiles Rust binaries inside Docker without
6-
# requiring pre-staged artifacts.
4+
# Local dev: builds gateway + supervisor images via tasks/scripts/docker-build-image.sh,
5+
# which first stages Rust binaries natively on the host (using cargo / cargo-zigbuild
6+
# when cross-compiling) and then builds the image from the prebuilt binary. This
7+
# mirrors CI and is faster than compiling inside Docker on every rebuild because
8+
# the host's cargo target cache and sccache are reused across iterations.
79
#
810
# Run from repo root:
9-
# skaffold dev -f deploy/helm/openshell/skaffold.yaml
11+
# mise run helm:skaffold:dev
12+
# mise run helm:skaffold:run
1013
#
1114
# See https://skaffold.dev/docs/deployers/helm/ (setValueTemplates, IMAGE_* fields).
1215
apiVersion: skaffold/v4beta14
@@ -23,40 +26,34 @@ build:
2326
context: ../../..
2427
custom:
2528
buildCommand: |
26-
docker buildx build \
27-
--build-arg BUILD_FROM_SOURCE=1 \
28-
--target gateway \
29-
--tag "$IMAGE" \
30-
--load \
31-
--file deploy/docker/Dockerfile.gateway \
32-
.
29+
IMAGE_NAME="${IMAGE%:*}" \
30+
IMAGE_TAG="${IMAGE##*:}" \
31+
tasks/scripts/docker-build-image.sh gateway
3332
dependencies:
3433
paths:
3534
- Cargo.toml
3635
- Cargo.lock
3736
- crates/**
3837
- proto/**
3938
- deploy/docker/Dockerfile.gateway
40-
- crates/openshell-server/migrations/**
39+
- tasks/scripts/docker-build-image.sh
40+
- tasks/scripts/stage-prebuilt-binaries.sh
4141
- image: openshell/supervisor
4242
context: ../../..
4343
custom:
4444
buildCommand: |
45-
docker buildx build \
46-
--build-arg BUILD_FROM_SOURCE=1 \
47-
--target supervisor \
48-
--tag "$IMAGE" \
49-
--load \
50-
--file deploy/docker/Dockerfile.supervisor \
51-
.
45+
IMAGE_NAME="${IMAGE%:*}" \
46+
IMAGE_TAG="${IMAGE##*:}" \
47+
tasks/scripts/docker-build-image.sh supervisor
5248
dependencies:
5349
paths:
5450
- Cargo.toml
5551
- Cargo.lock
5652
- crates/**
5753
- proto/**
5854
- deploy/docker/Dockerfile.supervisor
59-
- deploy/docker/cross-build.sh
55+
- tasks/scripts/docker-build-image.sh
56+
- tasks/scripts/stage-prebuilt-binaries.sh
6057
deploy:
6158
helm:
6259
releases:

0 commit comments

Comments
 (0)