Skip to content

Commit 41ff94e

Browse files
lexfreilexfrei
committed
fix(flux): tune source-watcher concurrency and CPU under load
source-watcher reconciles one ArtifactGenerator per PackageSource and builds the resulting ExternalArtifacts. On the management cluster that is ~100 artifact builds during install, and the platform HelmRelease blocks until all of them are Ready. It was the only Flux controller left untuned. In the bootstrap all-in-one pod it ran without an explicit --concurrent (binary default 10) and with the shared 100m CPU request; in the production FluxInstance it was missing from the --concurrent / CPU patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. Raise source-watcher to --concurrent=20 in both deployments. In the bootstrap all-in-one pod also give it a 500m CPU request; the pod has no CPU limits, so the request is what drives its CPU share under contention. In the production FluxInstance add it to the shared reconciler patch, so it now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the other tuned controllers already have. The flux-aio module exposes a resources override for the watcher (set via CUE) but no args override, so the bootstrap --concurrent bump is a post-render patch in the Makefile. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 9c18260 commit 41ff94e

4 files changed

Lines changed: 22 additions & 2 deletions

File tree

‎internal/fluxinstall/manifests/fluxcd.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8209,6 +8209,7 @@ spec:
82098209
- --storage-path=/data
82108210
- --storage-adv-addr=flux.$(RUNTIME_NAMESPACE).svc
82118211
- --events-addr=http://localhost:9690
8212+
- --concurrent=20
82128213
env:
82138214
- name: SOURCE_CONTROLLER_LOCALHOST
82148215
value: localhost:9790
@@ -8251,7 +8252,7 @@ spec:
82518252
limits:
82528253
memory: 1Gi
82538254
requests:
8254-
cpu: 100m
8255+
cpu: 500m
82558256
memory: 64Mi
82568257
securityContext:
82578258
allowPrivilegeEscalation: false

‎packages/core/flux-aio/Makefile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,11 @@ MANIFESTS_DIR=../../../internal/fluxinstall/manifests
1717
update:
1818
timoni bundle build -f flux-aio.cue > $(MANIFESTS_DIR)/fluxcd.yaml
1919
yq eval '(select(.kind == "Namespace") | .metadata.labels."pod-security.kubernetes.io/enforce") = "privileged"' -i $(MANIFESTS_DIR)/fluxcd.yaml
20+
# The flux-aio module does not expose a --concurrent flag for source-watcher; inject it here.
21+
# source-watcher builds one ExternalArtifact per PackageSource component, so raise its
22+
# concurrency above the binary default of 10 to match the other controllers under load.
23+
# The append is guarded so a future module that emits its own --concurrent is not duplicated.
24+
yq eval '(select(.kind == "Deployment" and .metadata.name == "flux").spec.template.spec.containers[] | select(.name == "source-watcher") | select(.args | contains(["--concurrent=20"]) | not).args) += ["--concurrent=20"]' -i $(MANIFESTS_DIR)/fluxcd.yaml
2025
sed -i $(MANIFESTS_DIR)/fluxcd.yaml \
2126
-e '/timoni/d' \
2227
-e 's|\.cluster\.local\.,||g' -e 's|\.cluster\.local\,||g' -e 's|\.cluster\.local\.||g' \

‎packages/core/flux-aio/flux-aio.cue‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,20 @@ bundle: {
1010
namespace: "cozy-fluxcd"
1111
values: {
1212
securityProfile: "privileged"
13+
// source-watcher builds one ExternalArtifact per PackageSource
14+
// component; on the management cluster that is ~100 concurrent
15+
// builds during install. Give it CPU headroom so it is not
16+
// starved by its sibling controllers in the all-in-one pod.
17+
// The module has no per-controller args override for the
18+
// watcher, so its --concurrent bump is applied as a post-render
19+
// patch in the Makefile.
20+
controllers: watcher: resources: {
21+
requests: {
22+
cpu: "500m"
23+
memory: "64Mi"
24+
}
25+
limits: memory: "1Gi"
26+
}
1327
tolerations: [{
1428
operator: "Exists"
1529
key: "node.kubernetes.io/not-ready"

‎packages/system/fluxcd/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ flux-instance:
1919
patches:
2020
- target:
2121
kind: Deployment
22-
name: "(kustomize-controller|helm-controller|source-controller)"
22+
name: "(kustomize-controller|helm-controller|source-controller|source-watcher)"
2323
patch: |
2424
- op: add
2525
path: /spec/template/spec/containers/0/args/-

0 commit comments

Comments
 (0)