feat(deploy): harden image for rootless podman with read-only rootfs#577
Conversation
34cd042 to
99af4c0
Compare
The frontend no longer bakes the backend URL into the bundle at build
time and mutates it with sed on every container start. Instead:
* web/lib/api.ts: apiUrl and wsUrl become one-line pass-throughs, so
the browser fetches relative paths through the frontend
(:3782/api/...). All existing call sites keep working unchanged.
* web/proxy.ts: catches /api/* and /ws/* and rewrites them to
DEEPTUTOR_API_BASE_URL at request time, reading
DEEPTUTOR_AUTH_ENABLED for the auth-redirect check.
* web/lib/auth.ts: switches from NEXT_PUBLIC_AUTH_ENABLED (build-time)
to DEEPTUTOR_AUTH_ENABLED (set by the entrypoint on every start).
* Dockerfile: drops the .env.local placeholder block, the sed -i in
start-frontend.sh, and the runtime API-base fallback chain. Adds
gosu, bakes a non-root 'deeptutor' user (UID 1000), and has the
entrypoint chown /app/data and export DEEPTUTOR_API_BASE_URL and
DEEPTUTOR_AUTH_ENABLED from data/user/settings/*.json on every
start. supervisord is now invoked under 'gosu deeptutor'.
* web/tests/api-resolve-base.test.ts: deleted (its 8 tests reference
resolveBase / isApiBasePlaceholder, both gone; apiUrl/wsUrl are
now pass-throughs so the rest is meaningless). The auth-redirect
test is updated to prime DEEPTUTOR_AUTH_ENABLED instead.
The upstream ghcr.io/muchobien/pocketbase image's entrypoint uses absolute paths (no /pb/ prefix): --dir=/pb_data --publicDir=/pb_public --hooksDir=/pb_hooks. The previous example mounted at /pb/pb_data and crashed on first start with 'mkdir /pb_data: read-only file system'. Split the single mount into three matching the upstream entrypoint: /pb_data, /pb_public, /pb_hooks.
Tested rootless-podman deployment shape:
* read_only: true on every service
* userns_mode: keep-id (host UID 1000 -> container UID 1000)
* tmpfs mounts for /tmp, /run, /var/run, /var/log, /root, /home
* bind mounts on ./data (no named volumes: podman auto-creates
them with the userns-mapped root UID 100000, which 755's
incorrectly against the host)
* loopback-only port bindings (drop the 127.0.0.1: prefix to
expose on all interfaces)
* PocketBase sidecar uses the corrected /pb_data, /pb_public,
/pb_hooks mount paths from the docker-compose fix.
The entrypoint chowns /app/data and gosu-drops to the unprivileged
deeptutor user (UID 1000) before starting supervisord, so under
userns_mode: keep-id the running process is your host user.
.env.example carries only the host-side loopback bindings and TZ. URL
knowledge lives in data/user/settings/system.json (read by the
entrypoint on every start -> DEEPTUTOR_API_BASE_URL -> proxy.ts). No
compose env var for the API base.
CONTAINERIZATION.md (7 sections):
* Overview -- the architectural change (URL knowledge moves from
build-time sed to request-time proxy rewriter) and the three
deployment shapes that follow.
* Docker (default) -- docker run with -p 3782/8001 and a
deeptutor-data volume; remote / reverse-proxy deployment via
system.json; host LLM providers (Ollama / LM Studio / etc.)
via host.docker.internal.
* Podman / rootless / read-only rootfs -- the compose.yaml walk-
through, why each tmpfs/keep-id/read-only line is there, and a
portable podman run equivalent. Notes the known
/var/run/supervisord.pid follow-up.
* Runtime configuration -- the data/user/settings/*.json layout
and the no-compose-env-vars rule.
* PocketBase -- the corrected mount path, single-user caveat.
* Troubleshooting -- the four most likely first-boot issues.
* Security notes -- non-root user, RO rootfs, keep-id, sandbox
runner trade-off, CORS, auth gating.
README.md gets a one-line pointer in the 'Option 3 -- Docker'
section, right after the GHCR image list.
99af4c0 to
cd26210
Compare
|
@pancacake Please take a look. The current version is not containerization friendly. |
Thanks for inviting. I've actually took a look days before, I think im gonna do some improvement on your pr before merging it. Thanks for your contribution! |
The PR's relative-path + proxy.ts rewrite design depends on two env vars (DEEPTUTOR_API_BASE_URL, DEEPTUTOR_AUTH_ENABLED) read server-side by the Next.js middleware. As submitted they were only set by the Docker entrypoint, and the auth one read a non-existent auth.json key — so auth enforcement was silently disabled and the PyPI `deeptutor start` path was unwired. - Export both vars from render_environment() (the single JSON-backed source of truth), so the Docker entrypoint AND the `deeptutor start` launcher get them consistently. DEEPTUTOR_API_BASE_URL falls back to localhost:<backend_port>. - launcher.py: set both in common_env using the resolved backend_url, so they track port-conflict reassignment. - Dockerfile: drop the duplicated, buggy per-var heredocs (one read the wrong key `auth_enabled` instead of `enabled`, leaving the proxy auth gate always off); the export_runtime_settings_to_env eval now provides both. Add them to the env-unset list for hygiene. - Update the two placeholder tests to assert the new proxy-based design instead of the removed build-time sed mechanism; add DEEPTUTOR_* coverage in test_runtime_settings. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…ploads Two gaps in the PR's frontend changes: 1. Client auth awareness was permanently false. api.ts/auth.ts read process.env.DEEPTUTOR_AUTH_ENABLED in browser code, but Next.js only inlines NEXT_PUBLIC_ vars into the bundle — so the Sign-out / Admin affordances never rendered and apiFetch's 401 -> /login redirect never fired when auth was on. Resolve auth state at runtime from the backend instead: fetchAuthStatus() records it via setRuntimeAuthEnabled(), and components observe it through the new useAuthStatus() hook. Works on Docker (read-only rootfs), PyPI, and dev with no build-time baking. (The server-side gate in proxy.ts is unaffected.) 2. KB document uploads >10MB were truncated. All /api/* now flows through the proxy, whose body buffer defaults to 10MB while the backend accepts 100MB (DocumentValidator.MAX_FILE_SIZE). Raise experimental.proxyClientMaxBodySize to 110MB so uploads aren't silently cut off. Adds a regression test that a 401 does NOT redirect when auth is disabled. Co-Authored-By: Claude Opus 4.8 <[email protected]>
CONTAINERIZATION.md described the old browser-talks-to-:8001-directly model, which contradicts the PR's own server-side proxy design. Clarify that the browser only ever talks to the frontend origin (:3782); /api/* and /ws/* are forwarded to the backend inside the container, so publishing :8001 is optional. Rewrite the remote/reverse-proxy section (single-container needs no API base; next_public_api_base is only for split deployments) and the troubleshooting note. Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
Thanks for the exceptionally well-documented issue and PR, @enihcam — the rootless-podman + read-only-rootfs hardening, the PocketBase mount fix, and the move from build-time We dug into the request-routing change end-to-end (including reading the installed Next 16 source) and confirmed the core design works:
Full gates green: -- From my claude code :) |
Self-service /profile page (account info, icon/image avatar, sign out) implementing the profile half of #553. Contributed by @IliaAvdeev. Integration performed during merge onto current dev: - Migrate ProfileLink and the profile page off the removed AUTH_ENABLED build-time export to runtime auth status (auth.ts was refactored on dev by #577). ProfileLink reads the full status via fetchAuthStatus (it needs username/avatar, which the useAuthStatus hook does not expose); the page gates on status.enabled. - Resolve en/zh locale conflicts as a union of dev and PR keys. - Honor EXIF orientation in createImageBitmap so the primary decode path matches the <img> fallback. - Drop an unreachable AUTH_ENABLED render guard in the profile page. Verified locally: tsc, eslint, ruff, web node tests (142), pytest tests/multi_user (81 passed). Co-Authored-By: Claude Opus 4.8 <[email protected]>
#577 moved /api and /ws behind a request-time Next.js proxy (web/proxy.ts), so the browser only talks to :3782 and the frontend forwards to the backend server-side. Update the Docker quickstart accordingly: - drop the :8001 host mapping from the default `docker run`; publishing it is now optional (direct API access only) - replace the "map both ports / no in-container proxy" warning with the single-port reality - rewrite the reverse-proxy guidance: single-container needs no API base (point the proxy at :3782); only split deployments set next_public_api_base, which is read server-side and never sent to the browser Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
fix: run supervisord as root so services start under rootful Docker PR #577 dropped supervisord itself to UID 1000 via gosu; under a rootful daemon (Docker Desktop / Docker Engine) the root-owned /dev/fd/1,2 and pidfile become unopenable, so backend+frontend never spawn (EACCES). Run supervisord as root and drop each child to deeptutor via per-program user= instead — app processes stay non-root in both rootful Docker and rootless podman. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ss keep-id Removes `user=root` from the [supervisord] section in both production and development supervisord configs, and rewrites the entrypoint comment to explain why. Background: under rootless podman + userns_mode: keep-id, container PID 1 runs as the host user (UID 1000) instead of root. The previous design explicitly forced supervisord to drop privileges to root (`user=root`), which works under rootful daemons (PID 1 is root, has CAP_SETUID) but fails under rootless + keep-id — supervisord sees `user=root` in its config, tries to setuid(0), lacks CAP_SETUID, and exits with "Error: Can't drop privilege as nonroot user" (per supervisord options.py: refuses to drop privileges when not running as root). Fix: omit `user=` from the [supervisord] section so supervisord inherits PID 1's UID (root in rootful; UID 1000 in rootless + keep-id). The backend and frontend programs still drop to the unprivileged deeptutor user via per-program `user=deeptutor` — setuid(1000) works because either PID 1 is root (rootful) or PID 1 is already UID 1000 (rootless keep-id, where the setuid is a no-op). /dev/fd/1,2 ownership matches PID 1's UID in both runtimes, so supervisord's stdout/stderr writes succeed without an explicit setuid dance. Verified locally: clean heredoc indentation; supervisord config parses; both [supervisord] sections confirmed without `user=root`. Closes the rootless-podman + read-only-rootfs follow-up to PR #577 (reported by the user after #577 + #586 merged).
…e proxy Harden the published image for rootless podman with a read-only rootfs, and move URL/auth knowledge from build-time sed to a request-time proxy (web/proxy.ts). Includes follow-up fixes: JSON-backed DEEPTUTOR_* env wiring for both Docker and the PyPI launcher (correct auth.json key), runtime client auth resolution, 100MB-safe upload proxy buffer, updated tests, and corrected docs. Closes HKUDS#576. Co-authored with @enihcam.
Self-service /profile page (account info, icon/image avatar, sign out) implementing the profile half of HKUDS#553. Contributed by @IliaAvdeev. Integration performed during merge onto current dev: - Migrate ProfileLink and the profile page off the removed AUTH_ENABLED build-time export to runtime auth status (auth.ts was refactored on dev by HKUDS#577). ProfileLink reads the full status via fetchAuthStatus (it needs username/avatar, which the useAuthStatus hook does not expose); the page gates on status.enabled. - Resolve en/zh locale conflicts as a union of dev and PR keys. - Honor EXIF orientation in createImageBitmap so the primary decode path matches the <img> fallback. - Drop an unreachable AUTH_ENABLED render guard in the profile page. Verified locally: tsc, eslint, ruff, web node tests (142), pytest tests/multi_user (81 passed). Co-Authored-By: Claude Opus 4.8 <[email protected]>
HKUDS#577 moved /api and /ws behind a request-time Next.js proxy (web/proxy.ts), so the browser only talks to :3782 and the frontend forwards to the backend server-side. Update the Docker quickstart accordingly: - drop the :8001 host mapping from the default `docker run`; publishing it is now optional (direct API access only) - replace the "map both ports / no in-container proxy" warning with the single-port reality - rewrite the reverse-proxy guidance: single-container needs no API base (point the proxy at :3782); only split deployments set next_public_api_base, which is read server-side and never sent to the browser Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…visord fix: run supervisord as root so services start under rootful Docker PR HKUDS#577 dropped supervisord itself to UID 1000 via gosu; under a rootful daemon (Docker Desktop / Docker Engine) the root-owned /dev/fd/1,2 and pidfile become unopenable, so backend+frontend never spawn (EACCES). Run supervisord as root and drop each child to deeptutor via per-program user= instead — app processes stay non-root in both rootful Docker and rootless podman. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ss keep-id Removes `user=root` from the [supervisord] section in both production and development supervisord configs, and rewrites the entrypoint comment to explain why. Background: under rootless podman + userns_mode: keep-id, container PID 1 runs as the host user (UID 1000) instead of root. The previous design explicitly forced supervisord to drop privileges to root (`user=root`), which works under rootful daemons (PID 1 is root, has CAP_SETUID) but fails under rootless + keep-id — supervisord sees `user=root` in its config, tries to setuid(0), lacks CAP_SETUID, and exits with "Error: Can't drop privilege as nonroot user" (per supervisord options.py: refuses to drop privileges when not running as root). Fix: omit `user=` from the [supervisord] section so supervisord inherits PID 1's UID (root in rootful; UID 1000 in rootless + keep-id). The backend and frontend programs still drop to the unprivileged deeptutor user via per-program `user=deeptutor` — setuid(1000) works because either PID 1 is root (rootful) or PID 1 is already UID 1000 (rootless keep-id, where the setuid is a no-op). /dev/fd/1,2 ownership matches PID 1's UID in both runtimes, so supervisord's stdout/stderr writes succeed without an explicit setuid dance. Verified locally: clean heredoc indentation; supervisord config parses; both [supervisord] sections confirmed without `user=root`. Closes the rootless-podman + read-only-rootfs follow-up to PR HKUDS#577 (reported by the user after HKUDS#577 + HKUDS#586 merged).
Closes #576.
This PR hardens the published
ghcr.io/hkuds/deeptutorimage for rootless podman with read-only rootfs, and ships a testedcompose.yamlfor that deployment shape. The change set is intentionally small: the architectural core (URL knowledge moves from build-time sed to request-time proxy rewriter) plus the minimum plumbing to make it work.What's actually changing
The image:
deeptutoruser (UID 1000); the entrypoint drops privileges viagosuafter chowning/app/data.start-frontend.shno longer mutates the bundle withsed -i(which broke under RO rootfs). URL knowledge moves toweb/proxy.ts, which rewrites/api/*and/ws/*toDEEPTUTOR_API_BASE_URLat request time.entrypoint.shexportsDEEPTUTOR_API_BASE_URLandDEEPTUTOR_AUTH_ENABLEDfromdata/user/settings/*.jsonon every start.The frontend:
web/lib/api.ts:apiUrlandwsUrlbecome(p) => ppass-throughs (the browser fetches relative paths through the frontend; the proxy does the cross-origin work). All existing call sites stay as-is.web/proxy.ts: added the rewrite block + readsDEEPTUTOR_AUTH_ENABLEDfor the auth-redirect check.Deployment:
docker-compose.yml: pocketbase mount path corrected (/pb_data,/pb_public,/pb_hooks— no/pb/prefix, matching the upstream image's entrypoint).compose.yaml(new): tested rootless-podman config withread_only: true,userns_mode: keep-id, tmpfs, and bind mounts..env.example(new): companion env file forcompose.yaml.Docs:
CONTAINERIZATION.md(new): full per-installation guide.README.md: one-line pointer.Commits
Verification
npm run lint,tsc --noEmit,npm run test:nodeall clean (119 node tests pass; the deletedapi-resolve-base.test.tsis replaced by 3apiFetchredirect tests on the newDEEPTUTOR_AUTH_ENABLEDenv var).pre-commit run --all-filesclean.podman build -t deeptutor:pr-test -f Dockerfile --target production .succeeds.__NEXT_PUBLIC_API_BASE_PLACEHOLDER__token (the.env.localblock is gone), andapiUrl/wsUrlresolve to identity functions.proxy.tsreadsDEEPTUTOR_API_BASE_URL(set by the entrypoint) and rewrites/api/*and/ws/*to it.Known minor follow-up
supervisordemitsCRIT could not write pidfile /var/run/supervisord.pidon startup because/var/runtmpfs ismode=0755(owned by root) and the now-unprivileged PID 1 can't write there. Children still spawn fine and reach RUNNING state. The fix ismode=1777on/var/run(matching/tmp); deferred to a follow-up to keep this PR focused.gosu capabilities under rootless podman
The image's
gosubinary is grantedcap_setuid,cap_setgid+epviasetcapin the production stage. Without this,gosu deeptutor supervisordfails withoperation not permittedinside the rootless podman user namespace (userns_mode: keep-id); with it, supervisord drops cleanly to UID 1000. Requireslibcap2-binin the image (one extra apt package, ~80 KB).