perf(webhooks): remove eager json.MarshalIndent from admission validation hot paths#7670
Conversation
|
Thank you for your contribution! 🙏 Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected. While you are waiting, make sure to:
Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient. Learn more about our contribution guide. |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
bd5eed4 to
cdbc34e
Compare
…admission cost verifyScaledObjects performs two duplicate-conflict checks on every ScaledObject admission: one for duplicate scaleTargetRef and one for duplicate HPA name. Both checks listed ALL ScaledObjects in the namespace via kc.List, which — because controller-runtime's cached client DeepCopies every returned item — allocated O(N) memory per admission. Measured impact with a heap profile during 10k SO creation burst: verifyScaledObjects consumed 71 % of inuse_space (106 MB at peak). At 60k SOs the list allocates ~900 MB per admission; at 30/s creation rate that is ~27 GB/s of allocation, which outpaces MADV_FREE and causes the webhook OOMKill loop seen in production-scale tests. The fix registers two controller-runtime field indexes in SetupWebhookWithManager: spec.scaleTargetRef.name → so.Spec.ScaleTargetRef.Name spec.hpaName → getHpaName(*so) (computed default or explicit override) verifyScaledObjects now issues two narrow indexed List calls (each returning 0–1 items in the common case) instead of one full-namespace scan. Per-admission allocation drops from O(N * object_size) to O(1 * object_size) regardless of cluster scale. Pairs with kedacore#7670 (remove eager MarshalIndent from the same loop) for a complete webhook memory fix at scale. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Greg Garber <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR optimizes admission webhook validation paths by removing unconditional JSON serialization and replacing spec comparisons that previously relied on json.MarshalIndent + string compare.
Changes:
- Replace
json.MarshalIndent-based debug logging with structured logr key/value logging in multiple webhook validators. - Replace
isRemovingFinalizer-style spec comparisons from JSON string equality toreflect.DeepEqual. - Update changelog entry to document the performance/oom mitigation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| apis/keda/v1alpha1/triggerauthentication_webhook.go | Switch validation logs to structured kv logging; change finalizer-removal spec compare to reflect.DeepEqual. |
| apis/keda/v1alpha1/scaledobject_webhook.go | Switch several validation logs to structured kv logging; change finalizer-removal spec compare to reflect.DeepEqual. |
| apis/keda/v1alpha1/scaledjob_webhook.go | Switch validation logs to structured kv logging; change finalizer-removal spec compare to reflect.DeepEqual. |
| apis/eventing/v1alpha1/cloudeventsource_webhook.go | Switch validation logs to structured kv logging; change finalizer-removal spec compare to reflect.DeepEqual. |
| CHANGELOG.md | Document the webhook hot-path logging/spec-compare performance improvement. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
JorTurFer
left a comment
There was a problem hiding this comment.
nice improvement! I guess that the copilot's suggestion about using k8s.io/apimachinery/pkg/api/equality.Semantic.DeepEqual makes sense, other of its suggestions can be ignored
|
LGTM! |
|
/run-e2e internals |
dbe8af7 to
debee6f
Compare
looks to be a flaky failure? Error related to replica scaling behavior @rickbrouwer |
…tion hot paths
Every admission call on ScaledObject, ScaledJob, TriggerAuthentication,
ClusterTriggerAuthentication, CloudEventSource, and ClusterCloudEventSource
was unconditionally calling json.MarshalIndent on the full object for a
.V(1).Info debug log — which is off by default, so the marshaled bytes
are discarded. At high admission throughput this generates significant
transient garbage; Go's runtime lazily returns memory to the kernel via
MADV_FREE, so under sustained burst the container RSS grows faster than
it shrinks and the webhook OOMs.
Additionally, four near-identical isRemovingFinalizer-style helpers were
using json.MarshalIndent on both specs and comparing the resulting
strings — an O(N) allocating/walking operation where reflect.DeepEqual
is both correct and cheap.
Changes:
- Replace the eager marshal+Sprintf pattern with structured logr kv-args
(`log.V(1).Info("msg", "key", value)`). Values are passed as interface{}
and only serialized by the sink if the verbosity level is enabled.
- Fix two pre-existing copy-paste bugs where the wrong logger was used
(scaledobjectlog in scaledjob_webhook.go and triggerauthentication_webhook.go).
- Replace four isRemovingFinalizer-style JSON-string-compare implementations
with reflect.DeepEqual(spec, oldSpec). Same semantics (both treat nil
slice and empty slice as different), much less work per call.
- Remove encoding/json import from files that no longer need it.
Observed impact at Netflix: during a 60k-ScaledObject KWOK+Karpenter load
test, the admission webhook was OOMKilled in a crash loop at 4 GiB during
the creation burst (~2000 admissions/sec sustained). Heap profile
captured between restarts showed only a few MiB of inuse_space — the
memory pressure was on the runtime, not live Go heap, confirming the
MarshalIndent-generated garbage was the driver rather than cache growth.
This change is net -35 lines and has no behavior change other than
slightly different log keys at V(1).
Signed-off-by: Greg Garber <[email protected]>
…helpers Replace reflect.DeepEqual with k8s.io/apimachinery/pkg/api/equality.Semantic.DeepEqual in the four isRemovingFinalizer helpers. Semantic.DeepEqual handles Kubernetes types (Quantity, nil vs empty slices) correctly, matching the behavior users expect. Also fix copy-paste variable name oldTa -> oldSj in scaledjob_webhook.go. Signed-off-by: Greg Garber <[email protected]>
debee6f to
c975fc4
Compare
|
/run-e2e internals |
@rickbrouwer looks like we need to run it one more time |
|
/run-e2e internals |
…eda (2.19.0 ➔ 2.20.0) (#779) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/home-operations/charts-mirror/keda](https://github.com/kedacore/keda) | minor | `2.19.0` → `2.20.0` | --- ### Release Notes <details> <summary>kedacore/keda (ghcr.io/home-operations/charts-mirror/keda)</summary> ### [`v2.20.0`](https://github.com/kedacore/keda/blob/HEAD/CHANGELOG.md#v2200) [Compare Source](kedacore/keda@v2.19.0...v2.20.0) ##### New - **General**: Add `scalingModifiers` fallback behavior ([#​7366](kedacore/keda#7366)) - **General**: Introduce Elastic Forecast Scaler ([#​7494](kedacore/keda#7494)) - **General**: Introduce new OpenSearch Scaler ([#​7456](kedacore/keda#7456)) ##### Improvements - **General**: Add cooldownPeriod and pollingInterval checks for ScaledObject ([#​7271](kedacore/keda#7271)) - **General**: Add CRD-level validation markers (Minimum, MinLength, MinItems, Enum) for ScaledObject, ScaledJob, ScaleTriggers, and TriggerAuthentication API types ([#​7533](kedacore/keda#7533)) - **General**: Add `--leader-election-id` flag to allow configuring the leader election Lease name ([#​7564](kedacore/keda#7564)) - **General**: Add scaler HTTP request metrics (`keda_scaler_http_requests_total`, `keda_scaler_http_request_duration_seconds`) for outbound HTTP requests made during scaler metric collection ([#​6600](kedacore/keda#6600)) - **General**: Allow more control of TLS versions & ciphers via `KEDA_HTTP_TLS_CIPHER_LIST`, `KEDA_SERVICE_TLS_CIPHER_LIST` and `KEDA_SERVICE_MIN_TLS_VERSION` env vars ([#​7617](kedacore/keda#7617)) - **General**: Cap each scalers-cache reader at a per-reader budget derived from `globalHTTPTimeout` so `ScalersCache.Close` cannot block indefinitely ([#​7574](kedacore/keda#7574)) - **General**: Make APIService cert injections optional ([#​7559](kedacore/keda#7559)) - **General**: Remove unconditional `json.MarshalIndent` calls from admission webhook validation hot paths; replace spec-comparison `MarshalIndent`-and-string-compare in `isRemovingFinalizer` variants with `reflect.DeepEqual`. Prevents webhook OOM under sustained admission load at large scale (observed at \~60k ScaledObjects) ([#​7670](kedacore/keda#7670)) - **AWS Scalers**: Add support for AWS External ID in TriggerAuthentication podIdentity for all AWS scalers (SQS, Kinesis, DynamoDB, CloudWatch, etc.) to enable cross-account access scenarios ([#​6921](kedacore/keda#6921)) - **Elasticsearch Scaler**: Add HTTP status check for Elasticsearch errors ([#​7480](kedacore/keda#7480)) - **Github Runner Scaler**: Handle rate limit errors by respecting X-RateLimit-Reset and Retry-After headers and returning cached queue length ([#​7683](kedacore/keda#7683)) - **Kubernetes Workload Scaler**: Add `groupByNode` parameter ([#​7628](kedacore/keda#7628)) - **Metrics API Scaler**: Add custom HTTP client timeout ([#​7549](kedacore/keda#7549)) - **MSSQL Scaler**: Add Azure Workload Identity support for Azure SQL authentication ([#​6104](kedacore/keda#6104)) - **Prometheus Scaler**: Emit metric tracking empty responses from Prometheus ([#​7062](kedacore/keda#7062)) - **RabbitMQ Scaler**: Add support for OAuth2 authentication for RabbitMQ over HTTP ([#​7379](kedacore/keda#7379)) - **Temporal Scaler**: Add support for scaling based on Worker Deployment Version backlog via new `workerDeploymentName` and `workerDeploymentBuildId` fields. Deprecate `buildId`, `selectAllActive`, and `selectUnversioned` because those parameters are used for Rules-Based Worker Versioning, which was a short-lived experimental feature that has been deprecated in the Temporal server since December 2024 and will stop being supported soon. Users of Rules-Based Worker Versioning should use Worker Deployments instead. ([#​7672](kedacore/keda#7672)) ##### Fixes - **General**: Check updated status for Fallback condition instead of ScaledObject ([#​7488](kedacore/keda#7488)) - **General**: Fail fast in `GetMetrics` when the gRPC connection is in Shutdown state instead of waiting for context timeout ([#​7251](kedacore/keda#7251)) - **General**: Fix int64 overflow in milli-quantity conversion for very large metric values ([#​7441](kedacore/keda#7441)) - **General**: Fix `keda_scaler_active` not being emitted for CPU and memory triggers ([#​4945](kedacore/keda#4945)) - **General**: Fix misleading namespace in error log when secret access is restricted ([#​7739](kedacore/keda#7739)) - **General**: Fix race in scalers cache rebuild that caused transient scaler errors ([#​7574](kedacore/keda#7574)) - **General**: Fix ScaledJob emitting wrong CloudEvent type (`ScaledObjectReadyType` instead of `ScaledJobReadyType`) when transitioning to ready state ([#​7792](kedacore/keda#7792)) - **General**: Fix ScaledObject admission webhook to return validation error from `verifyReplicaCount`, preventing invalid ScaledObjects from being created ([#​5954](kedacore/keda#5954)) - **General**: Fix ScaledObject Ready condition not reflecting HPA status ([#​7649](kedacore/keda#7649)) - **General**: Handle paused scaling directly in reconciler ([#​7663](kedacore/keda#7663)) - **General**: Honor `stderrthreshold` when `logtostderr` is enabled by updating klog to v2.140.0 ([#​7568](kedacore/keda#7568)) - **General**: Limit projected service account token reads during Vault authentication ([#​7783](kedacore/keda#7783)) - **General**: Reject ScaledObject creation and update when the name exceeds 63 characters ([#​6998](kedacore/keda#6998)) - **AWS Scalers**: Fix TCP connection leak by closing HTTP idle connections on scaler `Close()` for SQS, Kinesis, DynamoDB, DynamoDB Streams, and CloudWatch scalers ([#​7756](kedacore/keda#7756)) - **Azure Data Explorer Scaler**: Remove clientSecretFromEnv support ([#​7554](kedacore/keda#7554)) - **Azure Event Hub Scaler**: Reject non-positive `unprocessedEventThreshold` to prevent integer division by zero when computing lag ([#​7732](kedacore/keda#7732)) - **Azure Pipelines Scaler**: Exclude already-assigned jobs from queue length ([#​7747](kedacore/keda#7747)) - **Cron Scaler**: Fix metric name generation so cron expressions with comma-separated values no longer produce invalid metric names ([#​7448](kedacore/keda#7448)) - **External Scaler**: gRPC Pool uses TLS context in the key ([#​7687](kedacore/keda#7687)) - **Forgejo Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **Forgejo Scaler**: Return correct activity to enable scale-to-zero ([#​7527](kedacore/keda#7527)) - **GCP Cloud Tasks Scaler**: Implement escapeFilterValue for metric filtering ([#​7482](kedacore/keda#7482)) - **GCP Scaler**: Validate Pub/Sub resource name in BuildMQLQuery ([#​7468](kedacore/keda#7468)) - **GCP Storage Scaler**: Metadata is not printed in the log ([#​7688](kedacore/keda#7688)) - **Github Runner Scaler**: Bound etag and per-repo caches to prevent unbounded memory growth when `enableEtags` is on ([#​7685](kedacore/keda#7685)) - **Github Runner Scaler**: Improve URL construction and error handling ([#​7495](kedacore/keda#7495)) - **Github Runner Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **InfluxDB Scaler**: Make `authToken` optional to support unauthenticated InfluxDB instances ([#​7616](kedacore/keda#7616)) - **Loki Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **Loki Scaler**: `serverAddress` now appends `/loki/api/v1/query` to the end of existing path instead of overriding ([#​7648](kedacore/keda#7648)) - **Metrics API Scaler**: Fix `aggregateFromKubeServiceEndpoints` using empty label selector that matched all EndpointSlices in the namespace instead of only the target service's ([#​7641](kedacore/keda#7641)) - **Metrics API Scaler**: Fix division by zero in average aggregation when all kube service endpoints fail ([#​7742](kedacore/keda#7742)) - **Metrics API Scaler**: Prevent response value reflection in scaler errors ([#​7693](kedacore/keda#7693)) - **NATS JetStream Scaler**: Return an error from `getMaxMsgLag` when the configured consumer is missing instead of falling back to the stream's last sequence, preventing incorrect scale-up to `maxReplicaCount` ([#​7657](kedacore/keda#7657)) - **NATS JetStream Scaler**: URL-encode user input in monitoring URL construction ([#​7483](kedacore/keda#7483)) - **PostgreSQL Scaler**: Quote whitespace-containing connection parameters in generated connection strings ([#​7784](kedacore/keda#7784)) - **PredictKube Scaler**: Bump `dysnix/predictkube-libs` to `v0.1.0` (drops the predictkube path to the archived/EOL `go-grpc-prometheus` and to the deprecated `golang/protobuf`) and use a portable Prometheus-API instant query for the health check so the scaler works against VictoriaMetrics, Thanos and other Prometheus-API-compatible backends ([#​7745](kedacore/keda#7745)) - **Prometheus Scaler**: Handle NaN results in the same manner as Inf ([#​7475](kedacore/keda#7475)) - **Prometheus Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **Pulsar Scaler**: Drop bearer/basic auth headers on redirects to a different host or on https->http downgrades to prevent credential leakage ([#​7686](kedacore/keda#7686)) - **RabbitMQ Scaler**: Fix AMQP connection leak by recovering channels on the existing connection and closing connections properly ([#​6266](kedacore/keda#6266)) - **RabbitMQ Scaler**: Use SASL EXTERNAL for RabbitMQ AMQP TLS without credentials ([#​6840](kedacore/keda#6840)) - **Redis Scaler**: Use literal command names in Lua script to fix compatibility with Alibaba Cloud Redis Cluster ([#​7758](kedacore/keda#7758)) - **Solace Scaler**: Fix URL escaping for Message VPN and Queue names ([#​7481](kedacore/keda#7481)) - **Solr Scaler**: Use net/url to safely encode query parameters ([#​7467](kedacore/keda#7467)) - **Splunk Observability Scaler**: Add MTS stream handling with context timeout ([#​7799](kedacore/keda#7799)) ##### Deprecations You can find all deprecations in [this overview](https://github.com/kedacore/keda/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Abreaking-change) and [join the discussion here](https://github.com/kedacore/keda/discussions/categories/deprecations). ##### Breaking Changes - **GCP PubSub Scaler**: The `subscriptionSize` setting is DEPRECATED and is removed in v2.20 - Use `mode` and `value` instead ([#​7720](kedacore/keda#7720)) - **Huawei Cloudeye Scaler**: The `minMetricValue` setting is DEPRECATED and is removed - Use `activationTargetMetricValue` instead ([#​7436](kedacore/keda#7436)) - **IBM MQ Scaler**: The `tls` setting code is removed ([#​6094](kedacore/keda#6094)) - **InfluxDB Scaler**: The `authToken` setting from `triggerMetadata` is DEPRECATED and is removed in v2.20 - Use `authToken` from `resolvedEnv` or `authParams` instead ([#​7722](kedacore/keda#7722)) ##### Other - **General**: Migrate event recording RBAC from core `events` to `events.k8s.io` ([#​7781](kedacore/keda#7781)) - **General**: Migrate metrics service gRPC response away from Kubernetes API protobuf types for Kubernetes 0.35 ([#​7781](kedacore/keda#7781)) - **General**: Remove dead code from authentication package and drop unused `authModes` field from ArangoDB, Loki, Prometheus and PredictKube scalers ([#​7726](kedacore/keda#7726)) - **General**: Use informer cache for ReplicaSet lookups in GetCurrentReplicas to reduce API server load ([#​7466](kedacore/keda#7466)) - **External Scaler**: Fix race condition in `TestWaitForState` causing flaky test under `-race` detector ([#​7542](kedacore/keda#7542)) - **GCP Scaler**: Replace `credentialsFromJSON` with `credentialsFromJSONWithType` ([#​7523](kedacore/keda#7523)) - **Kafka Scaler**: Refactor Kafka Scaler ([#​7528](kedacore/keda#7528)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/779
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [authentik](https://goauthentik.io) ([source](https://github.com/goauthentik/helm)) | minor | `2026.2.x` → `2026.5.x` | | [fluxcd/flux2](https://github.com/fluxcd/flux2) | patch | `v2.8.5` → `v2.8.8` | | [ghcr.io/fluxcd/helm-controller](https://github.com/fluxcd/helm-controller) | patch | `v1.5.3` → `v1.5.5` | | [ghcr.io/fluxcd/kustomize-controller](https://github.com/fluxcd/kustomize-controller) | patch | `v1.8.3` → `v1.8.5` | | [ghcr.io/fluxcd/notification-controller](https://github.com/fluxcd/notification-controller) | patch | `v1.8.3` → `v1.8.4` | | [ghcr.io/fluxcd/source-controller](https://github.com/fluxcd/source-controller) | patch | `v1.8.2` → `v1.8.5` | | [keda](https://github.com/kedacore/keda) | minor | `2.19.0` → `2.20.0` | | [renovate/renovate](https://renovatebot.com) ([source](https://github.com/renovatebot/renovate)) | patch | [`43.209.1` → `43.209.2`](https://octochangelog.com/compare?repo=renovatebot%2Frenovate&from=43.209.1&to=43.209.2) | | [victoria-metrics-k8s-stack](https://github.com/VictoriaMetrics/helm-charts) | minor | `0.76.x` → `0.81.x` | --- ### Release Notes <details> <summary>goauthentik/helm (authentik)</summary> ### [`v2026.5.2`](https://github.com/goauthentik/helm/releases/tag/authentik-2026.5.2) [Compare Source](goauthentik/helm@authentik-2026.5.0...authentik-2026.5.2) authentik is an open-source Identity Provider focused on flexibility and versatility #### What's Changed - charts/authentik: bump to 2026.5.2 by [@​authentik-automation](https://github.com/authentik-automation)\[bot] in [#​476](goauthentik/helm#476) **Full Changelog**: <goauthentik/helm@authentik-2026.5.0...authentik-2026.5.2> ### [`v2026.5.0`](https://github.com/goauthentik/helm/releases/tag/authentik-2026.5.0) [Compare Source](goauthentik/helm@authentik-2026.2.3...authentik-2026.5.0) authentik is an open-source Identity Provider focused on flexibility and versatility See <https://docs.goauthentik.io/releases/2026.5/> #### What's Changed - charts/authentik: bump postgresql subchart from 16.7.27 to 18.6.5 by [@​renovate](https://github.com/renovate)\[bot] in [#​410](goauthentik/helm#410) - charts/authentik: remove hardcoded AUTHENTIK\_LISTEN variables by [@​rissson](https://github.com/rissson) in [#​468](goauthentik/helm#468) - charts/authentik: update docker.io/library/postgres Docker tag to v17.10 by [@​renovate](https://github.com/renovate)\[bot] in [#​470](goauthentik/helm#470) - charts/authentik: bump postgresql subchart to v18.6.7 by [@​renovate](https://github.com/renovate)\[bot] in [#​469](goauthentik/helm#469) - charts/authentik: bump to 2026.5.0 by [@​authentik-automation](https://github.com/authentik-automation)\[bot] in [#​471](goauthentik/helm#471) **Full Changelog**: <goauthentik/helm@authentik-2026.2.3...authentik-2026.5.0> </details> <details> <summary>fluxcd/flux2 (fluxcd/flux2)</summary> ### [`v2.8.8`](https://github.com/fluxcd/flux2/releases/tag/v2.8.8) [Compare Source](fluxcd/flux2@v2.8.7...v2.8.8) #### Highlights Flux v2.8.8 is a patch release that includes CVE fixes via go-git v5.19.1 (source-controller, image-automation-controller), reliability fixes in helm-controller and source-controller, the move of Helm back to upstream v4.2.0, support for GCP sovereign cloud artifact registries, and dependency updates. Users are encouraged to upgrade for the best experience. ℹ️ Please follow the [Upgrade Procedure for Flux v2.7+](fluxcd/flux2#5572) for a smooth upgrade from Flux v2.6 to the latest version. Fixes: - Add a configurable HTTP timeout for artifact fetching, preventing fetches that could block indefinitely and stall reconciliations (helm-controller) - Fix unbounded memory growth caused by a Kubernetes client transport retry wrapper accumulating on every reconcile (helm-controller) - Stop force-applying non-CRD objects placed under a chart's `crds/` directory (helm-controller) - Fix the Helm test action failing to find releases with names longer than 53 characters (helm-controller) - Improve path handling in the source reconcilers (source-controller) - Support Helm semver build-metadata encoding in OCIRepository tags (source-controller) Improvements: - Update go-git to v5.19.1 which fixes [CVE-2026-45571](GHSA-crhj-59gh-8x96) and [CVE-2026-45570](GHSA-m7cr-m3pv-hgrp) (source-controller, image-automation-controller) - Move Helm back to upstream v4.2.0 (source-controller, helm-controller) - Add support for GCP sovereign cloud artifact registries (source-controller, image-reflector-controller) - Upgrade Kubernetes to 1.36.1 (source-controller, helm-controller) - Update fluxcd/pkg dependencies #### Components changelog - helm-controller [v1.5.5](https://github.com/fluxcd/helm-controller/blob/v1.5.5/CHANGELOG.md) - image-automation-controller [v1.1.4](https://github.com/fluxcd/image-automation-controller/blob/v1.1.4/CHANGELOG.md) - image-reflector-controller [v1.1.2](https://github.com/fluxcd/image-reflector-controller/blob/v1.1.2/CHANGELOG.md) - source-controller [v1.8.5](https://github.com/fluxcd/source-controller/blob/v1.8.5/CHANGELOG.md) #### CLI changelog - Update toolkit components by [@​fluxcdbot](https://github.com/fluxcdbot) in [#​5904](fluxcd/flux2#5904) **Full Changelog**: <fluxcd/flux2@v2.8.7...v2.8.8> ### [`v2.8.7`](https://github.com/fluxcd/flux2/releases/tag/v2.8.7) [Compare Source](fluxcd/flux2@v2.8.6...v2.8.7) #### Highlights Flux v2.8.7 is a patch release that includes a bug fix in kustomize-controller, a CVE fix in source-controller and image-automation-controller via go-git v5.19.0, and dependency updates. Users are encouraged to upgrade for the best experience. ℹ️ Please follow the [Upgrade Procedure for Flux v2.7+](fluxcd/flux2#5572) for a smooth upgrade from Flux v2.6 to the latest version. Fixes: - Fix management of objects annotated with `kustomize.toolkit.fluxcd.io/ssa: IfNotPresent` where non-namespaced resources were being deleted and recreated on each reconciliation (kustomize-controller) Improvements: - Update go-git to v5.19.0 which fixes [CVE-2026-45022](GHSA-389r-gv7p-r3rp) (source-controller, image-automation-controller) - Update fluxcd/pkg dependencies (source-controller, kustomize-controller, image-automation-controller) #### Components changelog - helm-controller [v1.5.4](https://github.com/fluxcd/helm-controller/blob/v1.5.4/CHANGELOG.md) - image-automation-controller [v1.1.3](https://github.com/fluxcd/image-automation-controller/blob/v1.1.3/CHANGELOG.md) - kustomize-controller [v1.8.5](https://github.com/fluxcd/kustomize-controller/blob/v1.8.5/CHANGELOG.md) - notification-controller [v1.8.4](https://github.com/fluxcd/notification-controller/blob/v1.8.4/CHANGELOG.md) - source-controller [v1.8.4](https://github.com/fluxcd/source-controller/blob/v1.8.4/CHANGELOG.md) #### CLI changelog - Update toolkit components by [@​fluxcdbot](https://github.com/fluxcdbot) in [#​5891](fluxcd/flux2#5891) **Full Changelog**: <fluxcd/flux2@v2.8.6...v2.8.7> ### [`v2.8.6`](https://github.com/fluxcd/flux2/releases/tag/v2.8.6) [Compare Source](fluxcd/flux2@v2.8.5...v2.8.6) #### Highlights Flux v2.8.6 is a patch release that includes bug fixes and improvements across helm-controller, image-automation-controller, kustomize-controller, notification-controller, and source-controller. Users are encouraged to upgrade for the best experience. ℹ️ Please follow the [Upgrade Procedure for Flux v2.7+](fluxcd/flux2#5572) for a smooth upgrade from Flux v2.6 to the latest version. Fixes: - Fix a post-renderer conflict between overlapping hooks and templates (helm-controller) - Ignore force replace when server-side apply is enabled (helm-controller) - Fix a regression where generic providers would not forward commit status events (notification-controller) - Require the `audience` field on the GCR Receiver secret for tighter verification — will become mandatory in Flux v2.9 (notification-controller) Improvements: - Introduce the `MigrateAPIVersion` feature gate for migrating the API version of resources in managed field entries (kustomize-controller) - Update go-git to v5.18.0 bringing performance improvements for Git operations (source-controller, image-automation-controller) #### Components changelog - helm-controller [v1.5.4](https://github.com/fluxcd/helm-controller/blob/v1.5.4/CHANGELOG.md) - image-automation-controller [v1.1.2](https://github.com/fluxcd/image-automation-controller/blob/v1.1.2/CHANGELOG.md) - kustomize-controller [v1.8.4](https://github.com/fluxcd/kustomize-controller/blob/v1.8.4/CHANGELOG.md) - notification-controller [v1.8.4](https://github.com/fluxcd/notification-controller/blob/v1.8.4/CHANGELOG.md) - source-controller [v1.8.3](https://github.com/fluxcd/source-controller/blob/v1.8.3/CHANGELOG.md) #### CLI changelog - Update toolkit components by [@​fluxcdbot](https://github.com/fluxcdbot) in [#​5857](fluxcd/flux2#5857) **Full Changelog**: <fluxcd/flux2@v2.8.5...v2.8.6> </details> <details> <summary>fluxcd/helm-controller (ghcr.io/fluxcd/helm-controller)</summary> ### [`v1.5.5`](https://github.com/fluxcd/helm-controller/releases/tag/v1.5.5) [Compare Source](fluxcd/helm-controller@v1.5.4...v1.5.5) #### Changelog [v1.5.5 changelog](https://github.com/fluxcd/helm-controller/blob/v1.5.5/CHANGELOG.md) #### Container images - `docker.io/fluxcd/helm-controller:v1.5.5` - `ghcr.io/fluxcd/helm-controller:v1.5.5` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). ### [`v1.5.4`](https://github.com/fluxcd/helm-controller/releases/tag/v1.5.4) [Compare Source](fluxcd/helm-controller@v1.5.3...v1.5.4) #### Changelog [v1.5.4 changelog](https://github.com/fluxcd/helm-controller/blob/v1.5.4/CHANGELOG.md) #### Container images - `docker.io/fluxcd/helm-controller:v1.5.4` - `ghcr.io/fluxcd/helm-controller:v1.5.4` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). </details> <details> <summary>fluxcd/kustomize-controller (ghcr.io/fluxcd/kustomize-controller)</summary> ### [`v1.8.5`](https://github.com/fluxcd/kustomize-controller/releases/tag/v1.8.5) [Compare Source](fluxcd/kustomize-controller@v1.8.4...v1.8.5) #### Changelog [v1.8.5 changelog](https://github.com/fluxcd/kustomize-controller/blob/v1.8.5/CHANGELOG.md) #### Container images - `docker.io/fluxcd/kustomize-controller:v1.8.5` - `ghcr.io/fluxcd/kustomize-controller:v1.8.5` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). ### [`v1.8.4`](https://github.com/fluxcd/kustomize-controller/releases/tag/v1.8.4) [Compare Source](fluxcd/kustomize-controller@v1.8.3...v1.8.4) #### Changelog [v1.8.4 changelog](https://github.com/fluxcd/kustomize-controller/blob/v1.8.4/CHANGELOG.md) #### Container images - `docker.io/fluxcd/kustomize-controller:v1.8.4` - `ghcr.io/fluxcd/kustomize-controller:v1.8.4` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). </details> <details> <summary>fluxcd/notification-controller (ghcr.io/fluxcd/notification-controller)</summary> ### [`v1.8.4`](https://github.com/fluxcd/notification-controller/releases/tag/v1.8.4) [Compare Source](fluxcd/notification-controller@v1.8.3...v1.8.4) #### Changelog [v1.8.4 changelog](https://github.com/fluxcd/notification-controller/blob/v1.8.4/CHANGELOG.md) #### Container images - `docker.io/fluxcd/notification-controller:v1.8.4` - `ghcr.io/fluxcd/notification-controller:v1.8.4` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). </details> <details> <summary>fluxcd/source-controller (ghcr.io/fluxcd/source-controller)</summary> ### [`v1.8.5`](https://github.com/fluxcd/source-controller/releases/tag/v1.8.5) [Compare Source](fluxcd/source-controller@v1.8.4...v1.8.5) #### Changelog [v1.8.5 changelog](https://github.com/fluxcd/source-controller/blob/v1.8.5/CHANGELOG.md) #### Container images - `docker.io/fluxcd/source-controller:v1.8.5` - `ghcr.io/fluxcd/source-controller:v1.8.5` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). ### [`v1.8.4`](https://github.com/fluxcd/source-controller/releases/tag/v1.8.4) [Compare Source](fluxcd/source-controller@v1.8.3...v1.8.4) #### Changelog [v1.8.4 changelog](https://github.com/fluxcd/source-controller/blob/v1.8.4/CHANGELOG.md) #### Container images - `docker.io/fluxcd/source-controller:v1.8.4` - `ghcr.io/fluxcd/source-controller:v1.8.4` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). ### [`v1.8.3`](https://github.com/fluxcd/source-controller/releases/tag/v1.8.3) [Compare Source](fluxcd/source-controller@v1.8.2...v1.8.3) #### Changelog [v1.8.3 changelog](https://github.com/fluxcd/source-controller/blob/v1.8.3/CHANGELOG.md) #### Container images - `docker.io/fluxcd/source-controller:v1.8.3` - `ghcr.io/fluxcd/source-controller:v1.8.3` Supported architectures: `linux/amd64`, `linux/arm64` and `linux/arm/v7`. The container images are built on GitHub hosted runners and are signed with cosign and GitHub OIDC. To verify the images and their provenance (SLSA level 3), please see the [security documentation](https://fluxcd.io/flux/security/). </details> <details> <summary>kedacore/keda (keda)</summary> ### [`v2.20.0`](https://github.com/kedacore/keda/blob/HEAD/CHANGELOG.md#v2200) [Compare Source](kedacore/keda@v2.19.0...v2.20.0) ##### New - **General**: Add `scalingModifiers` fallback behavior ([#​7366](kedacore/keda#7366)) - **General**: Introduce Elastic Forecast Scaler ([#​7494](kedacore/keda#7494)) - **General**: Introduce new OpenSearch Scaler ([#​7456](kedacore/keda#7456)) ##### Improvements - **General**: Add cooldownPeriod and pollingInterval checks for ScaledObject ([#​7271](kedacore/keda#7271)) - **General**: Add CRD-level validation markers (Minimum, MinLength, MinItems, Enum) for ScaledObject, ScaledJob, ScaleTriggers, and TriggerAuthentication API types ([#​7533](kedacore/keda#7533)) - **General**: Add `--leader-election-id` flag to allow configuring the leader election Lease name ([#​7564](kedacore/keda#7564)) - **General**: Add scaler HTTP request metrics (`keda_scaler_http_requests_total`, `keda_scaler_http_request_duration_seconds`) for outbound HTTP requests made during scaler metric collection ([#​6600](kedacore/keda#6600)) - **General**: Allow more control of TLS versions & ciphers via `KEDA_HTTP_TLS_CIPHER_LIST`, `KEDA_SERVICE_TLS_CIPHER_LIST` and `KEDA_SERVICE_MIN_TLS_VERSION` env vars ([#​7617](kedacore/keda#7617)) - **General**: Cap each scalers-cache reader at a per-reader budget derived from `globalHTTPTimeout` so `ScalersCache.Close` cannot block indefinitely ([#​7574](kedacore/keda#7574)) - **General**: Make APIService cert injections optional ([#​7559](kedacore/keda#7559)) - **General**: Remove unconditional `json.MarshalIndent` calls from admission webhook validation hot paths; replace spec-comparison `MarshalIndent`-and-string-compare in `isRemovingFinalizer` variants with `reflect.DeepEqual`. Prevents webhook OOM under sustained admission load at large scale (observed at \~60k ScaledObjects) ([#​7670](kedacore/keda#7670)) - **AWS Scalers**: Add support for AWS External ID in TriggerAuthentication podIdentity for all AWS scalers (SQS, Kinesis, DynamoDB, CloudWatch, etc.) to enable cross-account access scenarios ([#​6921](kedacore/keda#6921)) - **Elasticsearch Scaler**: Add HTTP status check for Elasticsearch errors ([#​7480](kedacore/keda#7480)) - **Github Runner Scaler**: Handle rate limit errors by respecting X-RateLimit-Reset and Retry-After headers and returning cached queue length ([#​7683](kedacore/keda#7683)) - **Kubernetes Workload Scaler**: Add `groupByNode` parameter ([#​7628](kedacore/keda#7628)) - **Metrics API Scaler**: Add custom HTTP client timeout ([#​7549](kedacore/keda#7549)) - **MSSQL Scaler**: Add Azure Workload Identity support for Azure SQL authentication ([#​6104](kedacore/keda#6104)) - **Prometheus Scaler**: Emit metric tracking empty responses from Prometheus ([#​7062](kedacore/keda#7062)) - **RabbitMQ Scaler**: Add support for OAuth2 authentication for RabbitMQ over HTTP ([#​7379](kedacore/keda#7379)) - **Temporal Scaler**: Add support for scaling based on Worker Deployment Version backlog via new `workerDeploymentName` and `workerDeploymentBuildId` fields. Deprecate `buildId`, `selectAllActive`, and `selectUnversioned` because those parameters are used for Rules-Based Worker Versioning, which was a short-lived experimental feature that has been deprecated in the Temporal server since December 2024 and will stop being supported soon. Users of Rules-Based Worker Versioning should use Worker Deployments instead. ([#​7672](kedacore/keda#7672)) ##### Fixes - **General**: Check updated status for Fallback condition instead of ScaledObject ([#​7488](kedacore/keda#7488)) - **General**: Fail fast in `GetMetrics` when the gRPC connection is in Shutdown state instead of waiting for context timeout ([#​7251](kedacore/keda#7251)) - **General**: Fix int64 overflow in milli-quantity conversion for very large metric values ([#​7441](kedacore/keda#7441)) - **General**: Fix `keda_scaler_active` not being emitted for CPU and memory triggers ([#​4945](kedacore/keda#4945)) - **General**: Fix misleading namespace in error log when secret access is restricted ([#​7739](kedacore/keda#7739)) - **General**: Fix race in scalers cache rebuild that caused transient scaler errors ([#​7574](kedacore/keda#7574)) - **General**: Fix ScaledJob emitting wrong CloudEvent type (`ScaledObjectReadyType` instead of `ScaledJobReadyType`) when transitioning to ready state ([#​7792](kedacore/keda#7792)) - **General**: Fix ScaledObject admission webhook to return validation error from `verifyReplicaCount`, preventing invalid ScaledObjects from being created ([#​5954](kedacore/keda#5954)) - **General**: Fix ScaledObject Ready condition not reflecting HPA status ([#​7649](kedacore/keda#7649)) - **General**: Handle paused scaling directly in reconciler ([#​7663](kedacore/keda#7663)) - **General**: Honor `stderrthreshold` when `logtostderr` is enabled by updating klog to v2.140.0 ([#​7568](kedacore/keda#7568)) - **General**: Limit projected service account token reads during Vault authentication ([#​7783](kedacore/keda#7783)) - **General**: Reject ScaledObject creation and update when the name exceeds 63 characters ([#​6998](kedacore/keda#6998)) - **AWS Scalers**: Fix TCP connection leak by closing HTTP idle connections on scaler `Close()` for SQS, Kinesis, DynamoDB, DynamoDB Streams, and CloudWatch scalers ([#​7756](kedacore/keda#7756)) - **Azure Data Explorer Scaler**: Remove clientSecretFromEnv support ([#​7554](kedacore/keda#7554)) - **Azure Event Hub Scaler**: Reject non-positive `unprocessedEventThreshold` to prevent integer division by zero when computing lag ([#​7732](kedacore/keda#7732)) - **Azure Pipelines Scaler**: Exclude already-assigned jobs from queue length ([#​7747](kedacore/keda#7747)) - **Cron Scaler**: Fix metric name generation so cron expressions with comma-separated values no longer produce invalid metric names ([#​7448](kedacore/keda#7448)) - **External Scaler**: gRPC Pool uses TLS context in the key ([#​7687](kedacore/keda#7687)) - **Forgejo Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **Forgejo Scaler**: Return correct activity to enable scale-to-zero ([#​7527](kedacore/keda#7527)) - **GCP Cloud Tasks Scaler**: Implement escapeFilterValue for metric filtering ([#​7482](kedacore/keda#7482)) - **GCP Scaler**: Validate Pub/Sub resource name in BuildMQLQuery ([#​7468](kedacore/keda#7468)) - **GCP Storage Scaler**: Metadata is not printed in the log ([#​7688](kedacore/keda#7688)) - **Github Runner Scaler**: Bound etag and per-repo caches to prevent unbounded memory growth when `enableEtags` is on ([#​7685](kedacore/keda#7685)) - **Github Runner Scaler**: Improve URL construction and error handling ([#​7495](kedacore/keda#7495)) - **Github Runner Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **InfluxDB Scaler**: Make `authToken` optional to support unauthenticated InfluxDB instances ([#​7616](kedacore/keda#7616)) - **Loki Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **Loki Scaler**: `serverAddress` now appends `/loki/api/v1/query` to the end of existing path instead of overriding ([#​7648](kedacore/keda#7648)) - **Metrics API Scaler**: Fix `aggregateFromKubeServiceEndpoints` using empty label selector that matched all EndpointSlices in the namespace instead of only the target service's ([#​7641](kedacore/keda#7641)) - **Metrics API Scaler**: Fix division by zero in average aggregation when all kube service endpoints fail ([#​7742](kedacore/keda#7742)) - **Metrics API Scaler**: Prevent response value reflection in scaler errors ([#​7693](kedacore/keda#7693)) - **NATS JetStream Scaler**: Return an error from `getMaxMsgLag` when the configured consumer is missing instead of falling back to the stream's last sequence, preventing incorrect scale-up to `maxReplicaCount` ([#​7657](kedacore/keda#7657)) - **NATS JetStream Scaler**: URL-encode user input in monitoring URL construction ([#​7483](kedacore/keda#7483)) - **PostgreSQL Scaler**: Quote whitespace-containing connection parameters in generated connection strings ([#​7784](kedacore/keda#7784)) - **PredictKube Scaler**: Bump `dysnix/predictkube-libs` to `v0.1.0` (drops the predictkube path to the archived/EOL `go-grpc-prometheus` and to the deprecated `golang/protobuf`) and use a portable Prometheus-API instant query for the health check so the scaler works against VictoriaMetrics, Thanos and other Prometheus-API-compatible backends ([#​7745](kedacore/keda#7745)) - **Prometheus Scaler**: Handle NaN results in the same manner as Inf ([#​7475](kedacore/keda#7475)) - **Prometheus Scaler**: Limit HTTP error response logging ([#​7469](kedacore/keda#7469)) - **Pulsar Scaler**: Drop bearer/basic auth headers on redirects to a different host or on https->http downgrades to prevent credential leakage ([#​7686](kedacore/keda#7686)) - **RabbitMQ Scaler**: Fix AMQP connection leak by recovering channels on the existing connection and closing connections properly ([#​6266](kedacore/keda#6266)) - **RabbitMQ Scaler**: Use SASL EXTERNAL for RabbitMQ AMQP TLS without credentials ([#​6840](kedacore/keda#6840)) - **Redis Scaler**: Use literal command names in Lua script to fix compatibility with Alibaba Cloud Redis Cluster ([#​7758](kedacore/keda#7758)) - **Solace Scaler**: Fix URL escaping for Message VPN and Queue names ([#​7481](kedacore/keda#7481)) - **Solr Scaler**: Use net/url to safely encode query parameters ([#​7467](kedacore/keda#7467)) - **Splunk Observability Scaler**: Add MTS stream handling with context timeout ([#​7799](kedacore/keda#7799)) ##### Deprecations You can find all deprecations in [this overview](https://github.com/kedacore/keda/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Abreaking-change) and [join the discussion here](https://github.com/kedacore/keda/discussions/categories/deprecations). ##### Breaking Changes - **GCP PubSub Scaler**: The `subscriptionSize` setting is DEPRECATED and is removed in v2.20 - Use `mode` and `value` instead ([#​7720](kedacore/keda#7720)) - **Huawei Cloudeye Scaler**: The `minMetricValue` setting is DEPRECATED and is removed - Use `activationTargetMetricValue` instead ([#​7436](kedacore/keda#7436)) - **IBM MQ Scaler**: The `tls` setting code is removed ([#​6094](kedacore/keda#6094)) - **InfluxDB Scaler**: The `authToken` setting from `triggerMetadata` is DEPRECATED and is removed in v2.20 - Use `authToken` from `resolvedEnv` or `authParams` instead ([#​7722](kedacore/keda#7722)) ##### Other - **General**: Migrate event recording RBAC from core `events` to `events.k8s.io` ([#​7781](kedacore/keda#7781)) - **General**: Migrate metrics service gRPC response away from Kubernetes API protobuf types for Kubernetes 0.35 ([#​7781](kedacore/keda#7781)) - **General**: Remove dead code from authentication package and drop unused `authModes` field from ArangoDB, Loki, Prometheus and PredictKube scalers ([#​7726](kedacore/keda#7726)) - **General**: Use informer cache for ReplicaSet lookups in GetCurrentReplicas to reduce API server load ([#​7466](kedacore/keda#7466)) - **External Scaler**: Fix race condition in `TestWaitForState` causing flaky test under `-race` detector ([#​7542](kedacore/keda#7542)) - **GCP Scaler**: Replace `credentialsFromJSON` with `credentialsFromJSONWithType` ([#​7523](kedacore/keda#7523)) - **Kafka Scaler**: Refactor Kafka Scaler ([#​7528](kedacore/keda#7528)) </details> <details> <summary>renovatebot/renovate (renovate/renovate)</summary> ### [`v43.209.2`](https://github.com/renovatebot/renovate/releases/tag/43.209.2) [Compare Source](renovatebot/renovate@43.209.1...43.209.2) ##### Bug Fixes - **deps:** update ghcr.io/renovatebot/base-image docker tag to v13.55.6 (main) ([#​43751](renovatebot/renovate#43751)) ([160e9f9](renovatebot/renovate@160e9f9)) </details> <details> <summary>VictoriaMetrics/helm-charts (victoria-metrics-k8s-stack)</summary> ### [`v0.81.0`](https://github.com/VictoriaMetrics/helm-charts/releases/tag/victoria-metrics-k8s-stack-0.81.0) [Compare Source](VictoriaMetrics/helm-charts@victoria-metrics-k8s-stack-0.80.0...victoria-metrics-k8s-stack-0.81.0) ### Release notes for version 0.81.0 **Release date:** 28 May 2026   **Update note 1**: `defaultRules.create` is renamed to `defaultRules.enabled`; per-group `create` is renamed to `enabled`. Old `create` key is still respected as a fallback if `enabled` is not set. **Update note 2**: `defaultRules.additionalGroupByLabels` is renamed to `defaultRules.extraGroupByLabels`. Old `additionalGroupByLabels` is still respected as a fallback if `extraGroupByLabels` is not set. - rename `defaultRules.create` and per-group `create` to `enabled`, with fallback to `create` for backward compatibility. - add per-group extraGroupByLabels, that replace defaultRules.extraGroupByLabels (if absent defaults to defaultRules.additionalGroupByLabels). See [#​2832](VictoriaMetrics/helm-charts#2832). ### [`v0.80.0`](https://github.com/VictoriaMetrics/helm-charts/releases/tag/victoria-metrics-k8s-stack-0.80.0) [Compare Source](VictoriaMetrics/helm-charts@victoria-metrics-k8s-stack-0.79.1...victoria-metrics-k8s-stack-0.80.0) ### Release notes for version 0.80.0 **Release date:** 25 May 2026   - bump version of VM components to [v1.144.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.144.0) ### [`v0.79.1`](https://github.com/VictoriaMetrics/helm-charts/releases/tag/victoria-metrics-k8s-stack-0.79.1) [Compare Source](VictoriaMetrics/helm-charts@victoria-metrics-k8s-stack-0.79.0...victoria-metrics-k8s-stack-0.79.1) ### Release notes for version 0.79.1 **Release date:** 20 May 2026   - support Grafana HTTPRoute when resolving grafanaAddr - bump operator dependency chart to version 0.63.1 ### [`v0.79.0`](https://github.com/VictoriaMetrics/helm-charts/releases/tag/victoria-metrics-k8s-stack-0.79.0) [Compare Source](VictoriaMetrics/helm-charts@victoria-metrics-k8s-stack-0.78.0...victoria-metrics-k8s-stack-0.79.0) ### Release notes for version 0.79.0 **Release date:** 18 May 2026   - bump victoria-metrics-operator dependency chart to version 0.63.0 - bump grafana dependency chart to version 12.3.3 - bump node-exporter dependency chart to version 4.55.0 ### [`v0.78.0`](https://github.com/VictoriaMetrics/helm-charts/releases/tag/victoria-metrics-k8s-stack-0.78.0) [Compare Source](VictoriaMetrics/helm-charts@victoria-metrics-k8s-stack-0.77.0...victoria-metrics-k8s-stack-0.78.0) ### Release notes for version 0.78.0 **Release date:** 11 May 2026   - bump version of VM components to [v1.143.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.143.0) - fix Alertmanager templates path to match VM Operator mount. See [#​2883](VictoriaMetrics/helm-charts#2883). ### [`v0.77.0`](https://github.com/VictoriaMetrics/helm-charts/releases/tag/victoria-metrics-k8s-stack-0.77.0) [Compare Source](VictoriaMetrics/helm-charts@victoria-metrics-k8s-stack-0.76.0...victoria-metrics-k8s-stack-0.77.0) ### Release notes for version 0.77.0 **Release date:** 03 May 2026   - set default securityContext for Alertmanager, when persistence is enabled to prevent from permissions issues. See [#​2846](VictoriaMetrics/helm-charts#2846). - default operator `admissionWebhooks.policy` to `Ignore` so the stack can be installed and upgraded in a single pass without races against the operator's webhook server. Override to `Fail` for strict validation. See [#​2874](VictoriaMetrics/helm-charts#2874). </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMDkuMSIsInVwZGF0ZWRJblZlciI6IjQzLjIwOS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Reviewed-on: https://forgejo.maio-tech.com/Sammy/Servers/pulls/2
…tion hot paths (kedacore#7670) * perf(webhooks): remove eager json.MarshalIndent from admission validation hot paths Every admission call on ScaledObject, ScaledJob, TriggerAuthentication, ClusterTriggerAuthentication, CloudEventSource, and ClusterCloudEventSource was unconditionally calling json.MarshalIndent on the full object for a .V(1).Info debug log — which is off by default, so the marshaled bytes are discarded. At high admission throughput this generates significant transient garbage; Go's runtime lazily returns memory to the kernel via MADV_FREE, so under sustained burst the container RSS grows faster than it shrinks and the webhook OOMs. Additionally, four near-identical isRemovingFinalizer-style helpers were using json.MarshalIndent on both specs and comparing the resulting strings — an O(N) allocating/walking operation where reflect.DeepEqual is both correct and cheap. Changes: - Replace the eager marshal+Sprintf pattern with structured logr kv-args (`log.V(1).Info("msg", "key", value)`). Values are passed as interface{} and only serialized by the sink if the verbosity level is enabled. - Fix two pre-existing copy-paste bugs where the wrong logger was used (scaledobjectlog in scaledjob_webhook.go and triggerauthentication_webhook.go). - Replace four isRemovingFinalizer-style JSON-string-compare implementations with reflect.DeepEqual(spec, oldSpec). Same semantics (both treat nil slice and empty slice as different), much less work per call. - Remove encoding/json import from files that no longer need it. Observed impact at Netflix: during a 60k-ScaledObject KWOK+Karpenter load test, the admission webhook was OOMKilled in a crash loop at 4 GiB during the creation burst (~2000 admissions/sec sustained). Heap profile captured between restarts showed only a few MiB of inuse_space — the memory pressure was on the runtime, not live Go heap, confirming the MarshalIndent-generated garbage was the driver rather than cache growth. This change is net -35 lines and has no behavior change other than slightly different log keys at V(1). Signed-off-by: Greg Garber <[email protected]> * perf(webhooks): use equality.Semantic.DeepEqual in finalizer-removal helpers Replace reflect.DeepEqual with k8s.io/apimachinery/pkg/api/equality.Semantic.DeepEqual in the four isRemovingFinalizer helpers. Semantic.DeepEqual handles Kubernetes types (Quantity, nil vs empty slices) correctly, matching the behavior users expect. Also fix copy-paste variable name oldTa -> oldSj in scaledjob_webhook.go. Signed-off-by: Greg Garber <[email protected]> --------- Signed-off-by: Greg Garber <[email protected]> Signed-off-by: Yurii Shcherbak <[email protected]>
Problem
Every admission request on ScaledObject, ScaledJob, TriggerAuthentication,
ClusterTriggerAuthentication, CloudEventSource, and ClusterCloudEventSource
unconditionally calls
json.MarshalIndenton the full object for a.V(1).Infodebug log, e.g.:The marshaled bytes are discarded at the default log level. Under sustained
admission throughput this generates large amounts of transient garbage;
Go's runtime lazily returns memory to the kernel (MADV_FREE) so the
container's RSS grows faster than it shrinks and the webhook OOMs.
Four
isRemovingFinalizer-style helpers do additional damage — each callmarshals both specs to indented JSON, converts to strings, and compares
the strings, where
reflect.DeepEqualis correct and cheap.Observed impact at Netflix
During a KWOK+Karpenter scale test with 60k ScaledObjects created at
30/sec, the admission webhook was OOMKilled in a crash loop at a 4 GiB
memory limit (~10 restarts during a single run, kube-burner completed
via
failurePolicy: Ignore). A heap profile captured between restartsshowed only a few MiB of
inuse_space— the Go heap was fine; thepressure was RSS from the MarshalIndent-generated garbage that hadn't
been returned to the kernel yet. Sustained rate was ~2000 admissions/sec.
After mitigating on the operations side by raising the limit to 10 GiB,
we pursued this fix to reduce the underlying cost.
Changes
json.MarshalIndent(...) + fmt.Sprintf(...) + V(1).Info(...)pattern with structured logr kv-args across all four webhook files.
Values are passed as
interface{}; only serialized by the sink if theverbosity level is actually enabled.
isRemovingFinalizer-stylejson.MarshalIndentstringcomparisons with
reflect.DeepEqual(spec, oldSpec). Both approachesagree on nil-vs-empty-slice (both treat them as different), so behavior
is preserved.
logger (
scaledobjectlog) was used inscaledjob_webhook.goandtriggerauthentication_webhook.go.encoding/jsonimport from files that no longer need it.Net: +32 / -67 lines across 5 files.
Tests
go test ./apis/...passes on my branch. The kv-args refactor is adrop-in replacement; the
reflect.DeepEqualsubstitution preservessemantics for the
isRemovingFinalizercall site.Happy to add benchmarks if helpful, but the production 60k evidence is
already quite strong.
Related
the same large-scale KEDA load test.