Skip to content

helm: atelet mounts /var/lib/kubelet/plugins with HostToContainer propagation - #39

Open
teemow wants to merge 22 commits into
kagent-dev:mainfrom
giantswarm:upstream/atelet-plugins-mount-propagation
Open

teemow wants to merge 22 commits into
kagent-dev:mainfrom
giantswarm:upstream/atelet-plugins-mount-propagation

Conversation

@teemow

@teemow teemow commented Sep 15, 2026

Copy link
Copy Markdown

Problem

atelet hostPath-mounts /var/lib/kubelet/plugins to reach the CSI driver sockets. With the default (private) propagation, every atelet start copies the node's CSI globalmounts into the container's mount namespace, and kubelet's later unmount of a volume never reaches that copy. The volume's filesystem — and the LUKS mapper of an encrypted volume — stays open, so the volume cannot be unstaged while atelet runs.

We hit this with Longhorn on a cluster running Substrate: after a pod moved to another node, NodeUnstageVolume failed forever with cryptsetup luksClose: Device is still in use, and the pod could not attach its volume on the new node until atelet was restarted.

Fix

mountPropagation: HostToContainer (rslave) on the kubelet-plugins volume mount — what CSI node plugins use for the kubelet directories. Mounts and unmounts made by kubelet propagate into the container; nothing propagates back.

Rendered manifests are otherwise unchanged. The change has been running in our Substrate deployment since 2026-09-11.

Prepared with AI assistance, reviewed and tested by the author.

EItanya and others added 19 commits September 23, 2026 18:09
Build and publish versioned binaries, container images, and Helm charts from release tags.

Signed-off-by: Eitan Yarmush <[email protected]>
Resolve atelet discovery and identity from the pod namespace, centralize install defaults, and allow explicitly selected local clusters to run without Pod Certificates. Keep authenticated transport as the default.

Signed-off-by: Eitan Yarmush <[email protected]>
Add a configurable end-to-end workflow deadline and propagate it through lease acquisition. Apply released worker assignments to the cache immediately so subsequent scheduling sees the completed pause.

Signed-off-by: Eitan Yarmush <[email protected]>
Parse PKCS1 RSA and SEC1 EC keys alongside PKCS8 keys, including regression coverage for RSA bundles.

Signed-off-by: Eitan Yarmush <[email protected]>
Require the agentgateway E2E lane, reuse the installed control plane for microVM demos, wait for asset storage initialization, and accommodate runtime startup and counter persistence behavior in E2E checks.

Signed-off-by: Eitan Yarmush <[email protected]>
Package the control plane, workers, PostgreSQL, RustFS, and CRDs as Helm charts. Keep manifests and generated RBAC aligned, add Helm E2E checks, and include current scheduling, sandbox permissions, and agentgateway configuration.

Co-authored-by: Jet Chiang <[email protected]>
Co-authored-by: Keith Mattix II <[email protected]>
Signed-off-by: Jet Chiang <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Allow an external PostgreSQL instance and a configurable schema, validate connection settings, and pass the schema to the API server.

Signed-off-by: Eitan Yarmush <[email protected]>
Wire the API server snapshot backend and S3 settings to the chart storage configuration.

Signed-off-by: Eitan Yarmush <[email protected]>
Configure trace, metric, and log endpoints independently, expose trace sampling, and route agentgateway access logs through the collector logs pipeline.

Signed-off-by: Eitan Yarmush <[email protected]>
Isolate sandbox asset download tests from pause image pulls, explicitly advance the CA file timestamp, and disable VCS stamping for license checks in temporary verification worktrees.

Signed-off-by: Eitan Yarmush <[email protected]>
Delete application containers before the pause container so their shared sandbox remains available throughout teardown. Cover the deletion order with a regression test.

Signed-off-by: Eitan Yarmush <[email protected]>
Rebuild the fork on upstream while preserving features, require agentgateway runtime validation, and use a guarded push. Delete task-owned clusters and disposable assets before finishing while preserving shared resources and recovery data.

Signed-off-by: Eitan Yarmush <[email protected]>
Select agentgateway expectations in the Helm test job, align the chart sandbox assets with the canonical manifest, and enable the CONNECT tunnel logging used by egress validation. This retains upstream gVisor checkpoint and restore fixes and closes configuration gaps between Helm and manifest installations.

Signed-off-by: Eitan Yarmush <[email protected]>
Expose ateApi.extraArgs so installations can configure API flags such as the template resync interval without editing the deployment template.

Signed-off-by: Eitan Yarmush <[email protected]>
A layer pull can share a singleflight call with retirement and return without unpacking the removed layer. Distinguish pull results from retirement results and retry after retirement completes. Cover the interleaving with a deterministic concurrency test.

Signed-off-by: Eitan Yarmush <[email protected]>
The egress ext_proc server listens on loopback. Probe the metrics readiness endpoint so Kubernetes can observe readiness through the pod IP, matching the upstream manifests.

Signed-off-by: Eitan Yarmush <[email protected]>
global.imageRegistry redirects every image at once for air-gapped
mirrors. Component images now resolve through the same
registry/repository split every kagent-family chart uses:
image.registry was one string carrying its path
(ghcr.io/kagent-dev/substrate) and is now the registry host only,
joined onto image.repository, so one global value (which overrides
image.registry) redirects the whole family. This is a breaking change
for values files that put a full prefix in image.registry: rendered
silently they would produce a doubled prefix failing only at pod start,
so the render fails instead, naming the split. A default render is
byte-identical to main.

Single-string images.* references (postgres, rustfs, aws-cli,
agentgateway) have their registry segment replaced by the containerd
rule (first path segment with a dot or colon), preserving repository
paths either way. global.imagePullSecrets merges (union) into every pod
spec, which previously had no pull-secret surface at all.
global.imagePullPolicy replaces the hardcoded IfNotPresent values as a
fallback, via substrate.imagePullPolicy.

Verified: a default render is byte-identical to main; the mirror knob
redirects all 9 images with paths preserved; the old-shape registry
fails loudly at template time; pull secrets land on all 9 pod specs;
the pullPolicy fallback fires.

Signed-off-by: Jonathan Jamroga <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Retain upstream secret URI syntax and client CA rotation while adding Helm deployment, configurable injector identity, strict default-deny authorization, and credential injection coverage.

Signed-off-by: Eitan Yarmush <[email protected]>
Build the nested provider package with the release images and publish it under the kubernetes-secrets basename expected by the Helm chart.

Signed-off-by: Eitan Yarmush <[email protected]>
krisztianfekete and others added 2 commits September 23, 2026 18:09
Pass the resolved atelet and router ServiceAccounts to the control plane, expose the controller pod namespace, and configure router Service discovery. Cover canonical and custom Helm releases in manifest tests.

Signed-off-by: Eitan Yarmush <[email protected]>
…pagation

atelet hostPath-mounts /var/lib/kubelet/plugins to reach the CSI driver sockets. With the
default (private) propagation every atelet start copies the node's CSI globalmounts into
the container's mount namespace, and kubelet's later unmount of a volume does not reach
that copy: the volume's filesystem stays mounted there, the block device (and the LUKS
mapper of an encrypted volume) stays open, and the volume can never be unstaged while
atelet runs. Observed with Longhorn: NodeUnstageVolume failed forever with
"cryptsetup luksClose: Device is still in use" after a pod moved to another node, and the
pod could not attach the volume on the new node until atelet was restarted.

HostToContainer (rslave) is what CSI node plugins use for the kubelet directories: mounts
and unmounts made by kubelet propagate into the container, nothing propagates back.

Signed-off-by: Timo Derstappen <[email protected]>
@QuentinBisson
QuentinBisson force-pushed the upstream/atelet-plugins-mount-propagation branch from ac9a51b to 374ccd9 Compare September 25, 2026 13:06
@EItanya
EItanya force-pushed the main branch 7 times, most recently from 156e3da to 4d0e339 Compare October 1, 2026 15:52
@EItanya
EItanya force-pushed the main branch 4 times, most recently from 16274c0 to 9f9b552 Compare October 6, 2026 22:08
@EItanya
EItanya force-pushed the main branch 2 times, most recently from 446feee to 783f8bf Compare October 29, 2026 04:06
alexnovak pushed a commit to alexnovak/kagent-substrate that referenced this pull request Nov 4, 2026
…gent-substrate#3435)

`ate.dev/llms.txt` linked to full Docsy HTML pages instead of standalone
Markdown files, omitted page status markers and schema links, and still
used the old `ateom` component name. Meanwhile, `raw-markdown.html`
wrote into an HTML template buffer and expanded `include-file` with
`replaceRE`, which HTML-escaped code blocks and blockquotes across
`ate.dev/llms-full.txt` and "Copy as Markdown" (`&kagent-dev#39;`, `&kagent-dev#34;`,
`&gt;`), stripped `$VAR` shell references as regexp capture groups, left
relative links unrewritten, and kept decorative `<figure>` HTML blocks.

- Add the `MARKDOWN` output format for doc pages
(`site/layouts/docs/single.md`) so each page publishes a clean
`index.md` alongside `index.html` and exposes `<link rel="alternate"
type="text/markdown">` in `<head>`.
- Return a plain string from `raw-markdown.html`, expand `include-file`
with literal replacement, drop decorative diagram HTML, and rewrite
relative links to absolute URLs across `llms-full.txt`, per-page
`index.md`, and "Copy as Markdown".
- Update `index.llms.txt` to link to each page's `index.md` build,
include `[Status: ...]` labels from front matter, link the
`ateapi.proto`, `ate.dev/v1` CRD, and OpenTelemetry Weaver metric
schemas, and summarize the sandbox model and the split between
`ate.dev/v1` Kubernetes CRDs and the `ate-api-server` gRPC API.
- Replace leftover `ateom`, `ate-api`, and `herder` references in docs
with `ateworker`, `atenet-router`, and `ate-api-server`, and add
template tests in `hack/test-site-templates.sh`.

AI use: AI-assisted, I reviewed and tested it.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants