Repository navigation
a readiness gate on a user-sized operation holds one of the shard's five helm-controller slots for its whole run #4145
Description
A HelmRelease whose readiness gate waits on a long operation holds one of the shard's five concurrency slots for the whole wait.
The mechanics. packages/system/flux-shard-operator/values.yaml:19 sets concurrent: 5, which internal/fluxshardoperator/provisioner.go:292 turns into --concurrent=5 for each shard's helm-controller. Application HelmReleases carry sharding.fluxcd.io/key, and helm-controller runs the Helm action synchronously inside Reconcile, so a release that waits is a slot that is occupied rather than a goroutine that is parked.
The exposure is not new. http-cache, tcp-balancer, vpn and postgres render plain Deployments, set no disable-wait, and already hold a slot for their whole Helm wait on the 10m default. What varies between kinds is what the wait is bounded by: for those it is deployment readiness, which the platform sizes, and for a kind gated on an image import it is the size of the data the user asked for.
kubernetes-rd and tenant-rd also wait, and their windows are longer still, but they arrive once with tenant setup rather than with every object a tenant creates. So the axis that matters is not whether a kind waits, it is the product of how long the wait can run and how many of that kind a tenant makes.
Two consequences worth pricing together. A gate that never resolves, as in #2908 where a cross-StorageClass clone rests in an in-flight phase indefinitely, produces a permanent duty cycle rather than one long occupancy: take a slot, hold it for the window, time out, release for the retry interval, take it again. At a 30m window and a 30s retry interval that is roughly 98% occupancy for as long as the object exists. And shrinking the window does not help, since occupancy runs at window/(window+retry) and a shorter window only raises the retry frequency.
So the lever is the shard's concurrent, or keeping kinds whose gate can wait on user-sized work off the shared shard. What would settle it is a number nobody here has: how many such objects a busy tenant creates at once, and how often one of them is permanently stuck.
Surfaced while adding a DataVolume readiness gate to vm-disk-rd, which widens that kind's window from the default to 30m.
Activity
lexfrei commentedon Sep 7, 2026
Sharper than the body, from the case that prompted this.
A disk whose clone never completes does not reach a terminal phase. The cross-StorageClass failure in #2908 leaves the DataVolume resting in an in-flight phase indefinitely, so the gate never resolves either way. What that produces is not one long occupancy but a permanent duty cycle: the release takes a slot, holds it for the whole install window, times out, frees the slot for the retry interval, and takes it again. With a 30m window and a 30s retry interval that is roughly 98% occupancy, for as long as the disk exists and nobody deletes it.
So the shape is worse than "a big import is slow". A handful of permanently broken disks, which nobody is watching precisely because the failure used to be silent, can hold the shard's slots more or less continuously and starve every other release on it. The disks do not have to be created simultaneously; they only have to exist.
That also means shrinking the install window does not help, and would make it slightly worse: occupancy runs at window/(window+retry), so a shorter window raises the retry frequency while barely moving the fraction. The lever really is the shard's
concurrent, or keeping kinds whose gate can wait on user-sized work off the shared shard.Nothing here changes the decision this issue is for; it changes the number that decision should be made against.
lexfrei commentedon Sep 22, 2026
I traced the wait in source, and I'd move readiness for kinds that tenants create per workload out of the Helm wait and into WorkloadMonitor. Cozystack paths below are at origin/main a759190, Flux paths at the tag named in each paragraph.
The wait holds a reconcile worker for its whole length. Cozystack pins helm-controller v1.5.0 (
internal/fluxinstall/manifests/fluxcd.yaml:8097). At that tag the Helm action runs fromAtomicRelease.Reconcile(internal/controller/helmrelease_controller.go:418). The waiter then callsWaitForSetWithContext(internal/action/wait.go:191). In fluxcd/pkg ssa/v0.67.1 that function blocks on<-done(ssa/manager_wait.go:152) until everything is Current, something is Failed, or the timeout fires. The number of workers is--concurrent(main.go:292at helm-controller v1.5.0).A CEL error does not end the wait early. In fluxcd/pkg runtime/v0.100.2 the evaluator returns the error (
runtime/cel/status_evaluator.go:120-122), and the poller records it as Unknown. The terminal check and FailFast only look at Failed (ssa/manager_wait.go:133-144at ssa/v0.67.1). So a DataVolume without astatuskey waits out the full window. The same goes for one where no expression is true, which reads as InProgress (status_evaluator.go:129).Flux has no non-blocking mode up to v1.6.4.
disableWaitswitches to the hook-only strategy (internal/action/wait.go:51at v1.5.0). In that mode CEL is not evaluated at all (api/v2/helmrelease_types.go:193-199).disableWaitForJobsonly drops the wait on Jobs, and bothwaitStrategyvalues go through the same waiter.CancelHealthCheckOnNewRevisioninterrupts a wait only when a new request for the same HelmRelease arrives, so it frees nothing for other releases.internal/action/wait.goat v1.6.4 is byte-identical to v1.5.0, so a Flux upgrade does not change this.The duty cycle in the issue body holds in source. A failed attempt requeues after
RetryInterval(internal/controller/helmrelease_controller.go:436at v1.5.0). Cozystack sets it to 30s for every generated release (pkg/cmd/server/start.go:90). There is no per-kind override for it.Neither lever in the issue body fixes this. Raising
concurrent(packages/system/flux-shard-operator/values.yaml:19) only changes how many stuck releases it takes to fill a shard. Keeping a kind off the shared shard is not possible today, because the admission webhook overwrites any shard key with the tenant's shard (internal/fluxshardoperator/webhook.go:92-101). Making placement per kind would give disks a shard shared across tenants, where one tenant's stuck disks block the others.WorkloadMonitor can already carry this signal. The Application's
WorkloadsReadycondition is computed fromWorkloadMonitor.Status.Operational(pkg/registry/apps/application/rest.go:1440-1495). vm-disk already renders a monitor whose selector matches its DataVolume. Right nowOperationalonly counts pods (internal/controller/workloadmonitor_controller.go:727-729), and vm-disk setsminReplicas: 0, so it is always true. If the monitor controller reads DataVolume phases, the signal is continuous, holds no Helm slot, and also covers disks created before the change. vm-instance already setsminReplicas: 1and virt-launcher carries the instance label, so itsWorkloadsReadyshould already be False for a VM that is not running.A Helm gate is fine on the system controller.
vm-default-imagesis installed by a PackageSource intocozy-system, gets no shard key, and is reconciled by the flux-aio helm-controller (internal/fluxinstall/manifests/fluxcd.yaml:8076). It is a single release, so the worst case is one slot there.A small correction to #4146: postgres-rd already sets
healthCheckExprs(packages/system/postgres-rd/cozyrds/postgres.yaml:57), so vm-disk-rd would not be the first.Open questions that block the choice:
WorkloadsReady? And should ApplicationReadyever include workload state? Today it deliberately does not (rest.go:1497-1506).runStrategy: Halted) count inWorkloadsReady? It has no virt-launcher pod, so it would read as not operational.fix(vm-default-images,kubernetes-worker-image,vm-disk): wait for gold…
fix(vm-disk): report DataVolume readiness through WorkloadsReady (#4380)
kvaps commentedon Oct 26, 2026
The vm-disk wait is already removed in f4b1f08: DataVolume health check is
current: "true"and disk readiness comes fromWorkloadsReadysince #4380. With the backoff from #4155 this covers the case here.One thing left: existing vm-disk HelmReleases pick this up only on the next Application update, because the definition sync copies only chart ref,
valuesFromand labels. I'd make the sync carryhealthCheckExprstoo, then we can close this.myasnikovdaniil commentedon Oct 29, 2026
#5087 does that: the definition reconciler now carries
healthCheckExprsandwaitStrategyto existing releases (internal/controller/applicationdefinition_helmreconciler.go:259-282at v1.7.0-rc.3, backport #5126). The other slot holder on upgrade was the tenantgatewayrelease, #5086 makes it wait only until the TenantGateway controller has observed the spec.I checked the vm-disk part on upgrade from v1.7.0-rc.2 to rc.3, with a disk whose HTTP source returns 404 and whose HelmRelease had both fields removed, same as a release built on v1.6.5. One gap is left on that first upgrade. The vm-disk chart artifact changed 12m38s before the new cozystack-controller could sync, because cozystack-controller is upgraded only after cilium, kube-ovn and application-definition-crd, so two attempts ran on the old spec and timed out on the DataVolume, and the release held one shard worker for 20m33s. After the sync a forced upgrade took 0.34 s with the import still failing, so next upgrades don't wait on it.
The v1.7.0 release notes carry it as a pre-upgrade step (finish running imports first), see https://github.com/cozystack/cozystack/releases/tag/v1.7.0. I think this issue can be closed.