atelet prepares each local worker's sandbox assets and versions file by zoez7 · Pull Request #2521 · agent-substrate/substrate

Zoe Zhao (zoez7) · 2026-10-12T18:02:18Z

Part of #587. This is the first part of the worker sandbox versions design, after #2393.

#2393 made each WorkerPool record the SandboxConfig it uses in status.sandboxClasses[].configRef. This PR is the atelet side of getting that config to the workers. For every worker pod on its node, atelet computes the runtime hash of every enabled version of the pool's config, creates the worker's directory, and writes a versions file in it. atelet keeps the mapping from each hash back to the SandboxConfig version.

The worker side is not in this PR. No ateom reads the file yet. Reading it at startup and registering again when the file changes come in the next PR.

How the syncer works

Each worker pod on the node is a key in one workqueue, drained by a single worker. The sections below explain the parts of the diagram that need a reason.

flowchart TD
    subgraph Events["Informer events"]
        P["Pod Add / Delete"]
        WP["WorkerPool Add<br/>Update: configRef changed"]
        SC["SandboxConfig Add<br/>Update: Generation changed"]
        D["Startup scan of ateoms/ directories"]
    end

    P -->|"Add(uid)"| Q[("workqueue<br/>key = pod UID")]
    WP -->|"Add immediately / Update random delay 0-30s<br/>the pool's local pods"| Q
    SC -->|"random delay 0-30s<br/>local pods of pools using the config"| Q
    D -->|"Add, or AddAfter(until 1 minute old)"| Q

    W["run: wait for Pod, WorkerPool<br/>and SandboxConfig caches to sync"] --> D
    W --> G
    Q --> G["run: queue.Get()<br/>single worker"]
    G --> L{"pod found<br/>in cache?"}
    L -->|"no"| R["removeDir(uid)"]
    L -->|"DeletionTimestamp set"| N["skip<br/>wait for Delete event"]
    L -->|"yes"| S["syncPod"]

    S --> S1["resolve + plan<br/>pool -> configRef -> versions for this arch"]
    S1 --> S2["build<br/>hash each version's asset and pause image digests"]
    S2 --> S3["write<br/>atomically write sandbox-versions.json if changed"]

    R --> F{"finish(err)"}
    N --> F
    S3 --> F
    S1 -. "error" .-> F
    S2 -. "default version error" .-> F

    F -->|"success / errPermanent"| FG["Forget"]
    F -->|"fewer than 8 retries"| RL["AddRateLimited<br/>doubling from 1s, max 5 min"]
    F -->|"8 retries reached"| FG
    RL --> Q

A worker pod is scheduled to a node

atelet watches the worker pods on its own node, the WorkerPools, and the SandboxConfigs. When a worker pod is scheduled to the node, atelet:

Finds the pod's WorkerPool and the config in its configRef. It never uses pendingConfigRef, which the atecontroller has not confirmed yet.
Picks the config's enabled versions that have assets for the node's architecture.
Computes each version's sandbox_runtime_hash: a sha256 over one sorted key=value line per asset digest and the pause image digest. Version names and URLs are left out, so the same binaries under another name or in another bucket are the same runtime. The hashes come from the SandboxConfig spec alone, so this downloads nothing.
Creates the worker directory /var/lib/ate/ateoms/<pod-uid>/ and writes sandbox-versions.json in it.

For a gVisor pool on an amd64 node using the bundled gvisor-default:

cat /var/lib/ate/ateoms/<pod-uid>/sandbox-versions.json

{
  "defaultRuntime": {
    "sandbox_runtime_hash": "sha256:bd071f1eff6bc2b6b6598aaec03540a7c0f86163586282da2e4400a6de072b5d"
  }
}

defaultRuntime comes from the config's default version. It is what new and cold-booted actors start under.
restorableRuntimes lists the other Enabled versions in spec order. The worker can only restore snapshots under them. It is omitted when there are none.
RegisterWorkerRequest reports these two separately, which is why the file keeps them apart instead of a flat list of hashes.
Each runtime is listed once. A version whose hash equals an earlier one's is left out.
A non-default version with no assets for the node's architecture, or with a pause image not pinned by digest, is left out. If the default version has either problem, no file is written, since no worker of the pool could start actors on this node.

The file does not promise that a runtime's assets are on the node. The existing prewarmer still downloads each config's default version, and Run/Restore still fetch any version on demand. Fetching through cmd/atelet/internal/filecache comes in a follow-up PR (see the TODOs in the code).

The SandboxConfig changes, or the pool switches config

When an operator edits the SandboxConfig's spec, or the pool's configRef moves to another config, atelet rewrites the file of every affected worker on the node.

Every node sees the change within about a second, and each rewrite will make the worker register again. So that every worker in the fleet does not hit ateapi at once, each atelet waits a random delay of up to 30 seconds before it rewrites. A newly scheduled worker pod is enqueued without that delay, since only its own node acts on it and the worker is waiting for the file.

Changes that do not affect the file, such as a SandboxConfig update that leaves its spec unchanged, a pendingConfigRef, or a replica count change, do not trigger a rewrite.

A sync fails

If the pool has no resolved config yet, or the pool or config is not in the cache, atelet keeps the existing file and retries. The 8 retries take about 4 minutes. After that it gives up until the pool's config changes or atelet restarts.
errPermanent covers a pod with no worker pool label, and a config whose default version has no assets for the node or has a pause image not pinned by digest. Retrying cannot fix them, so they are not retried.

atelet starts, the worker pod goes away, or atelet restarts

The syncer never holds up atelet startup. It waits for the Pod, WorkerPool and SandboxConfig caches to sync before it scans the worker directories or drains the queue. Before then, every pod looks gone and no pod's config resolves. While a list/watch fails (e.g. Forbidden while an RBAC rollout lags the binary), files wait and existing ones stay as they are.
When atelet restarts, the informer's initial list replays every pod as an Add, so every worker's file is rewritten from current state.
The startup scan removes directories left by pods that went away while atelet was down. It waits until a directory is a minute old, because its ateom may have created it before the pod reached atelet's cache.
The ateom no longer removes its directory on shutdown, only its own socket. The directory and the file now survive a container restart.

Other changes

atelet's ClusterRole can now list and watch WorkerPools.
atelet watches only the worker pods on its own node (spec.nodeName field selector), so the list/watch cost scales with the pods on the node, not the cluster.
hardware.AttrSandboxRuntimeHash (sandbox_runtime_hash) replaces the gvisor_asset_hash TODO.

Not in this PR

The ateom reading the file when it starts, and waiting for it if atelet has not written it yet.
The ateom watching the file and registering again when it changes.
Fetching assets through cmd/atelet/internal/filecache and removing the prewarmer.
Tests pass
Appropriate changes to documentation are included in the PR

A WorkerPool records the SandboxConfig it uses in status.sandboxClasses[].configRef, but nothing on the node acted on it. The sandbox asset prewarmer downloaded the default version of every SandboxConfig in the cluster, including configs no local worker uses, and skipped every other enabled version. This change does the atelet half of delivering assets to workers. The worker half, where the ateom reads the file, watches it, and registers again when it changes, comes in a later change. When a worker pod is scheduled to a node, atelet on that node: - finds the pod's WorkerPool and the config in its configRef. It never uses pendingConfigRef, which the atecontroller has not confirmed yet. - downloads the assets and pulls the pause image of every enabled version of that config for the node's architecture, since a worker can restore snapshots taken on any of them. The versions are fetched in parallel. A version counts as fetched once both its assets and its pause image are on the node, so the first Run/Restore under it finds them in the cache. - creates the worker directory /var/lib/ate/ateoms/<pod-uid>/ and writes sandbox-versions.json in it: the versions, default first, each with its local asset paths and digests. The file is written atomically, and only when its content changes. When an operator edits the SandboxConfig's spec, or the pool switches to another config, atelet fetches the new versions and rewrites the file of every affected worker on the node. Every node sees the change at about the same time, so each atelet waits a random delay of up to 30 seconds first, so the nodes do not all download the same bucket objects at once. A newly scheduled pod is enqueued without that delay, but one worker drains the queue, so it still waits behind syncs already queued. If the default version fails to fetch, or the pool has no resolved config yet, atelet keeps the existing file. If another version fails, atelet writes the file without it. Either way it retries with backoff. After 8 retries, about 4 minutes, it gives up until the pool's config changes or atelet restarts. One pod's fetches time out after 5 minutes, so a stalled download cannot hold up the other workers on the node. When the worker pod is deleted, atelet removes its directory. When atelet restarts, it rewrites every worker's file from current state and removes directories left by pods that went away while it was down. The ateom no longer removes its directory on shutdown, only its own socket, so the file survives a container restart. The syncer replaces the prewarmer. Its micro-VM gate goes with it, since a micro-VM pool's pods only schedule to nodes with /dev/kvm. atelet needs get/list/watch on workerpools. The syncer lives in cmd/atelet/worker_sandbox_syncer.go, which decides when each worker is synced, and cmd/atelet/worker_sandbox_versions.go, which builds one worker's file. internal/sandboxversions reads and writes the file.

Dmitry Berkovich (dberkov) · 2026-10-14T01:45:40Z

The /var/lib/ate/ateoms/<pod-uid>/sandbox-versions.json file contains information that is irrelevant to ateom. The only thing ateom needs to know are the hash the {pauseImage and assets}. ateom will use these hashes at the time of registration in the ate-apiserver and when it creates a snapshot. atelet is the component that is supposed to know the mapping between the hash and the actual sandbox config.

My proposal is, instead of generating /var/lib/ate/ateoms/<pod-uid>/sandbox-versions.json to be:

{
  "sandboxClass": "gvisor",
  "sandboxConfig": "gvisor-default",
  "versions": [
    {
      "name": "v1",
      "isDefault": true,
      "pauseImage": "registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4",
      "assetPaths": {
        "gvisor": "/var/lib/ate/static-files/gvisor-d547d81401461fd1c679c5c4fa0a6c2b8ef7dc3c22ce23c9e25dcc4c69cfd06f/runsc"
      },
      "assetDigests": {
        "gvisor": "d547d81401461fd1c679c5c4fa0a6c2b8ef7dc3c22ce23c9e25dcc4c69cfd06f"
      }
    }
  ]
}

Let's simplify it to:

[ "hash1", "hash2", "hash3" ]

atelet will locally keep the mapping between hash1 and the original struct from SandboxConfig. When it schedules an actorResume, it will download the required files and pass them to ateom.

In a follow-up PR, we can integrate this code with cmd/atelet/internal/filecache, which will automatically manage local copies of assets.

I would even keep the current logic that pre-downloads default assets as-is, and remove it later when the integration with filecache is completed. This way, this PR serves as the first step of the integration, without actual real-world usage yet.

I even think, we dont need to download all configurations all the time, just download the latest config only. With filecache, unused configurations for more than X min will be deleted in any case.

Thoughts?

The file listed each version's name, pause image, asset paths and digests, and the SandboxConfig name. The ateom needs none of that: it only registers which runtimes it can start and restore actors under. The file now holds the compatibility attributes of the default runtime and of each restorable runtime, matching RegisterWorkerRequest's default_runtime and restorable_runtimes. Each runtime has one attribute, sandbox_runtime_hash: a sha256 over the version's asset digests and its pause image digest. Version names, URLs and the SandboxConfig stay with atelet, so the same binaries under another version name are the same runtime, and a version with the same hash as an earlier one is left out.

The syncer downloaded the assets and pulled the pause image of every enabled version before writing a worker's file. Fetching will move to cmd/atelet/internal/filecache, which evicts unused assets, so until then the syncer only writes the file and the prewarmer keeps doing what it did before. The runtime hash needs no download: asset digests are in the SandboxConfig spec, and the CRD requires the pause image to be pinned by digest, so the syncer reads the pause digest from the ref. A default version whose pause image is not pinned fails permanently, and another such version is left out of the file. A runtime the file lists may not be on the node yet. The prewarmer fetches each config's default version, and Run/Restore fetch any other version on demand. The syncer keeps its jitter, since every rewrite makes a worker register again, and keeps retrying while the pool is unresolved or the caches have not synced.

Zoe Zhao (zoez7)

Thanks, agreed on the direction. Here is where I'd land on each point.

File contents: hashes only. In 623045a I updated file content to:

{
  "defaultRuntime": {"sandbox_runtime_hash": "sha256:..."},
  "restorableRuntimes": [
    {"sandbox_runtime_hash": "sha256:..."}
  ]
}

Kept one level more than a flat ["hash1", "hash2", ...], because RegisterWorkerRequest needs to report default_runtime (what new and cold-booted actors start under) separately from restorable_runtimes (what the worker can only restore from).

Prewarming and filecache. Agreed. In 97b036c I restored the existing prewarmer as is, and now the syncer only computes the hashes and writes the file. It downloads nothing and leaves TODOs pointing at the filecache integration.

Dmitry Berkovich (dberkov) · 2026-10-14T04:31:02Z

+// node. Apart from a pod in no pool, every failure can succeed once an
+// informer catches up or the atecontroller resolves the pool.
+func (r *workerSandboxResolver) resolve(pod *corev1.Pod) (v1alpha1.SandboxClass, *v1alpha1.SandboxConfig, error) {
+	if !r.synced() {


🤖 nit 🟢 – Until the WorkerPool and SandboxConfig caches sync, every local pod fails here, logs a warning per attempt, and spends retries from its budget of 8. run already waits for the pod cache before the directory scan. Waiting there for all three caches would let you drop synced and the warning burst at startup. The trade-off is that orphan removal would also wait on the WorkerPool watch.

Done in 7804400. run now waits for the Pod, WorkerPool and SandboxConfig caches before it scans the worker directories or drains the queue, so I dropped synced and the check in resolve. I'm fine with the trade-off: orphan removal now waits on the WorkerPool and SandboxConfig watches too, and syncing pods would have failed until then anyway.

Dmitry Berkovich (dberkov) · 2026-10-14T04:31:02Z

+}
+
+// IsTemp reports whether name is one of Write's temporary files.
+func IsTemp(name string) bool {


🤖 nit 🟢 – IsTemp has no caller outside its test. If it is for the next PR's file watcher, add it there with the watcher.

Removed in 7804400. The Write test now checks for leftover temp files with tempPrefix directly. I'll add a helper back with the watcher in the next PR if it needs one.

The syncer drained its queue as soon as it started and only gated the startup directory scan on the pod cache. Until the WorkerPool and SandboxConfig caches synced, every local pod failed to resolve, logged a warning, and spent one of its retries. run now waits for all three caches before it scans the worker directories or drains the queue, so the resolver no longer needs its own sync check. Orphan removal now also waits on the WorkerPool and SandboxConfig watches. Drop sandboxversions.IsTemp, which had no caller outside its test.

… for dynamic worker registration (agent-substrate#2660) Part of agent-substrate#587. Follows agent-substrate#2521. agent-substrate#2521 added the initial `atelet` syncer that wrote `sandbox_runtime_hash` per enabled `SandboxConfig` version to each local worker pod's directory. This PR completes the runtime and hardware compatibility pipeline from `atelet` through the worker into `WorkerService.RegisterWorker`: 1. **Renames `sandbox-versions.json` to `sandbox-runtimes.json` (`internal/sandboxruntimes`)**: `atelet` now writes the complete set of `VersionedSandboxCompat` attributes for `defaultRuntime` and `restorableRuntimes` (`architecture`, `sandbox_runtime_hash`, `cpu_features` for `gvisor`, and `cpu_vendor` + `cpu_model` for `microvm`). 2. **Probes CPU identity and gVisor `cpu_features` in `atelet`**: `atelet` owns all hardware and runtime probing on the node — reading `/proc/cpuinfo` for microVM CPU vendor/model and running `<runsc> cpu-features` (memoized by asset SHA-256) for each enabled gVisor version, encoded as a canonical lowercase hexadecimal bitmask whose bit numbers are gVisor's own `Feature` constants (`internal/hardware`). 3. **Watches `sandbox-runtimes.json` in the worker**: `ateworker-gvisor` and `ateworker-microvm` watch `/var/lib/ate/workers/<pod-uid>/sandbox-runtimes.json` via `register.WatchFile` and keep `default_runtime` and `restorable_runtimes` registered through `WorkerSupport.RegisterWorker` (`internal/ateworker/register`). ## End-to-End Flow ```mermaid flowchart LR subgraph Node["Kubernetes Node"] direction TB SC["SandboxConfig CRD"] --> Atelet["atelet (workerSandboxSyncer)"] CPU["Host /proc/cpuinfo\n(microVM vendor & model)"] --> Atelet Runsc["Cached runsc binary\n(<runsc> cpu-features)"] --> Atelet Atelet -->|"Atomic rename"| File["/var/lib/ate/workers/<pod-uid>/\nsandbox-runtimes.json"] File -->|"register.WatchFile (fsnotify)"| Worker["ateworker-gvisor / ateworker-microvm"] Worker -->|"WorkerSupport.RegisterWorker"| Atelet end Atelet -->|"WorkerService.RegisterWorker"| AteAPI["ateapi (Control Plane)"] ``` ## How it works ```mermaid sequenceDiagram participant Atelet as atelet participant FS as /var/lib/ate/workers/<pod-uid>/ participant Watcher as register.WatchFile (fsnotify / poll fallback) participant Reconciler as reconciler.run / sync participant Support as WorkerSupport.RegisterWorker Note over Watcher,FS: Watches parent dir for Create/Write on sandbox-runtimes.json Reconciler->>FS: Initial sync(): Read sandbox-runtimes.json alt File not written yet FS-->>Reconciler: ErrNotExist (wait for watch event or 5m resync) end Atelet->>FS: Probe hardware/runsc & atomic rename -> sandbox-runtimes.json FS-->>Watcher: fsnotify Create/Write event Watcher->>Reconciler: signal on changes channel Reconciler->>FS: Read sandbox-runtimes.json Reconciler->>Reconciler: proto.Equal(req, registered)? alt Changed (or first registration) Reconciler->>Support: RegisterWorker(capacity, default_runtime, restorable_runtimes) alt RPC succeeds Support-->>Reconciler: OK -> record registered = req else InvalidArgument (malformed runtime attributes) Support-->>Reconciler: Log Error, do not retry until file changes else Transient RPC error (e.g. Worker record not in ateapi yet) Support-->>Reconciler: Retry with exponential backoff (500ms..30s) end end ``` - **`atelet` owns all probing**: - For **`gvisor`** workers, `atelet` fetches the version's `gvisor` asset via `ensureSandboxAssets`, executes `<runsc> cpu-features` (memoized in-memory by the asset's SHA-256 digest), and encodes the reported names via `hardware.EncodeCPUFeatures` into a compact lowercase hex bitmask alongside `architecture` and `sandbox_runtime_hash`. - For **`microvm`** workers, `atelet` reads `/proc/cpuinfo` once (memoized after the first successful read) to record `cpu_vendor` and `cpu_model` alongside `architecture` and `sandbox_runtime_hash`. - **`cpu_features` bit layout**: - A feature's bit is the value of its gVisor `Feature` constant: on amd64, `32*block + bit` in that block's CPUID register (`X86FeatureAVX2 = 2*32 + 5 = 69`); on arm64, the value gVisor declares. Intel/AMD never reassign a CPUID bit, so a bit never moves when gVisor adds, reorders, or renames a feature, and a bit with no entry is one the hardware reserves. - `internal/hardware/cpufeatures_table.go` is a committed table, produced once from gVisor's `pkg/cpuid` sources at `gvisor.dev/[email protected]`: for every `Feature` constant with an entry in `allFeatures`, the constant's value keyed to the display name `runsc cpu-features` prints. Each entry's comment names the constant and, on amd64, its block and bit. The file header records this and how to add features from a newer gVisor by hand; because a bit is the hardware's, existing entries never change. - **This mapping is temporary.** Once `runsc` can print its feature set as this bitmask directly, `atelet` records that value as printed and the table, `EncodeCPUFeatures`, and `DecodeCPUFeatures` go away (TODO in the table header). The mask format and subset matching stay. No gVisor sources or generator are vendored for it. - **Subset matching**: `hardware.MatchesCompat` decodes `cpu_features` into a stack-allocated `[6]uint64` mask (384 bits; amd64 bits reach 261 today, with room for gVisor blocks through 11) and checks bitwise subset inclusion ($S_{\text{snapshot}} \subseteq S_{\text{worker}}$) while requiring exact equality on all other attributes. `DecodeCPUFeatures` rejects a mask that sets a bit with no name. - **Resilient watching & registration in the worker**: - `register.WatchFile` watches the parent directory for `Create`/`Write` events on the target file (reused by both `register.Report` and `ateworker-gvisor`'s `flagStore`), and falls back to periodic polling (`5s`) with a warning if `fsnotify` initialization fails (e.g. when `fs.inotify.max_user_instances` is exhausted on dense nodes). - `reconciler` runs a `5m` periodic resync tick alongside watch events, deduplicates consecutive registrations via `proto.Equal`, retries transient read/RPC failures with capped exponential backoff (`500ms..30s`), and drops permanent `InvalidArgument` rejections until the file changes. ## Compatibility `cpu_features` values recorded before this PR used a different bit numbering and a narrower mask. They do not compare meaningfully against workers built from this PR: any dev-cluster snapshot carrying one should be retaken rather than restored. No compatibility is guaranteed before v1.0.0, so there is no migration. - [x] Tests pass (`go test -race ./internal/hardware/... ./internal/sandboxruntimes/... ./internal/ateworker/register/... ./cmd/atelet/... ./cmd/ateworker-gvisor/... ./cmd/ateworker-microvm/...`) - [x] Appropriate changes to documentation are included in the PR

Zoe Zhao (zoez7) force-pushed the worker-sandbox-versions-file branch 2 times, most recently from 958adf3 to abf3a02 Compare October 12, 2026 20:49

Zoe Zhao (zoez7) marked this pull request as ready for review October 12, 2026 21:06

Zoe Zhao (zoez7) requested a review from Dmitry Berkovich (dberkov) October 12, 2026 21:06

Zoe Zhao (zoez7) force-pushed the worker-sandbox-versions-file branch 2 times, most recently from 46df703 to e68df13 Compare October 13, 2026 19:04

Zoe Zhao (zoez7) requested a review from Shrutiya Mohan (shrutiyam-glitch) October 13, 2026 23:02

Zoe Zhao (zoez7) force-pushed the worker-sandbox-versions-file branch from e68df13 to d8f08d8 Compare October 13, 2026 23:59

Zoe Zhao (zoez7) added 2 commits October 13, 2026 21:05

Zoe Zhao (zoez7) commented Oct 14, 2026

View reviewed changes

Dmitry Berkovich (dberkov) reviewed Oct 14, 2026

View reviewed changes

Dmitry Berkovich (dberkov) approved these changes Oct 14, 2026

View reviewed changes

Dmitry Berkovich (dberkov) enabled auto-merge October 14, 2026 04:57

Dmitry Berkovich (dberkov) added this pull request to the merge queue Oct 14, 2026

Merged via the queue into agent-substrate:main with commit b98f041 Oct 14, 2026
14 checks passed

Shrutiya Mohan (shrutiyam-glitch) mentioned this pull request Oct 14, 2026

atelet, ateworker: probe CPU features and watch sandbox-runtimes.json for dynamic worker registration #2660

Merged

2 tasks done

Chenyi Wang (chw120) mentioned this pull request Oct 20, 2026

benchmarking: spread actors over several worker pools #1760

Closed

1 task done

Shrutiya Mohan (shrutiyam-glitch) mentioned this pull request Oct 20, 2026

api: report worker runtimes per sandbox class (WorkerSandboxClass, WorkerSandboxRuntime, ActorSandboxRuntime) #3008

Closed

Navigation Menu

Repository navigation

atelet prepares each local worker's sandbox assets and versions file - #2521

atelet prepares each local worker's sandbox assets and versions file#2521
Dmitry Berkovich (dberkov) merged 4 commits into
agent-substrate:mainfrom
zoez7:worker-sandbox-versions-file

Zoe Zhao (zoez7) commented Oct 12, 2026 •

edited

Loading

Uh oh!

Dmitry Berkovich (dberkov) commented Oct 14, 2026 •

edited

Loading

Uh oh!

Zoe Zhao (zoez7) left a comment

Uh oh!

Dmitry Berkovich (dberkov) Oct 14, 2026

Uh oh!

Zoe Zhao (zoez7) Oct 14, 2026

Uh oh!

Dmitry Berkovich (dberkov) Oct 14, 2026

Uh oh!

Zoe Zhao (zoez7) Oct 14, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Zoe Zhao (zoez7) commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How the syncer works

A worker pod is scheduled to a node

The SandboxConfig changes, or the pool switches config

A sync fails

atelet starts, the worker pod goes away, or atelet restarts

Other changes

Not in this PR

Uh oh!

Dmitry Berkovich (dberkov) commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zoe Zhao (zoez7) left a comment

Choose a reason for hiding this comment

Uh oh!

Dmitry Berkovich (dberkov) Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Zoe Zhao (zoez7) Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Dmitry Berkovich (dberkov) Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Zoe Zhao (zoez7) Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Zoe Zhao (zoez7) commented Oct 12, 2026 •

edited

Loading

Dmitry Berkovich (dberkov) commented Oct 14, 2026 •

edited

Loading