fix(scaling): guard GetCurrentReplicas against nil ScaleTargetGVKR#7661
Merged
rickbrouwer merged 6 commits intoJun 19, 2026
Conversation
|
Thank you for your contribution! 🙏 Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected. While you are waiting, make sure to:
Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient. Learn more about our contribution guide. |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
ggarb
force-pushed
the
fix-getcurrentreplicas-nil-gvkr
branch
from
3d1ea68 to
4414c85
Compare
ggarb
force-pushed
the
fix-getcurrentreplicas-nil-gvkr
branch
2 times, most recently
from
f55e568 to
db90396
Compare
JorTurFer
reviewed
Apr 26, 2026
JorTurFer
left a comment
JorTurFer
left a comment
Member
There was a problem hiding this comment.
Thanks for the fix!
As now it's starting to be in multiple places, does it make sense to extract that code into a funciton that we can reuse and implicitly tracks all the usages of this hack?
When the informer cache races with ScaledObject creation, scaledObject.Status.ScaleTargetGVKR can be nil at the point the scale loop invokes GetCurrentReplicas. The current code then dereferences .Group / .Kind on a nil pointer and panics, taking down the operator. This applies the same defensive pattern already used in ResolveScaleTargetPodSpec: re-fetch the ScaledObject via the client when Status.ScaleTargetGVKR is nil, and if it is still nil after re-fetch, return a descriptive error instead of panicking. Observed in a 10k-ScaledObject KWOK load test where kube-burner created ScaledObjects at 10/s; the cache-race window opened wide enough that the panic fired reliably within the first 750 objects. Refs: kedacore#4389, kedacore#4955, kedacore#6176 Signed-off-by: Greg Garber <[email protected]>
ggarb
force-pushed
the
fix-getcurrentreplicas-nil-gvkr
branch
from
db90396 to
07cd22a
Compare
JorTurFer
approved these changes
May 7, 2026
rickbrouwer
added
Awaiting/2nd-approval
Signed-off-by: Rick Brouwer <[email protected]>
Member
|
/run-e2e passed tests: 176failed tests: 16 |
Member
|
/run-e2e |
Signed-off-by: Zbynek Roubalik <[email protected]>
Member
|
/run-e2e passed tests: 133failed tests: 62 |
zroubalik
approved these changes
May 26, 2026
zroubalik
added
waiting-for-e2e
and removed
Awaiting/2nd-approval
|
@zroubalik the e2e tests are a bit suspect. Can we run one more time to verify? The new ensureScaleTargetGVKR calls kubeClient.Get from inside GetCurrentReplicas, which is invoked on the SO reconcile path. If that Get ever blocks (e.g. informer cache mid-rebuild, watch reconnect), it ties up reconcile workers. |
Member
|
/run-e2e passed tests: 101failed tests: 94 |
Member
|
/run-e2e passed tests: 188failed tests: 7 |
Member
|
/run-e2e kafka*|gcp*|splunk|hashicorp|sequential |
rickbrouwer
added
merge-conflict
…s-nil-gvkr # Conflicts: # pkg/scaling/resolver/scale_resolvers_test.go
Contributor
Author
|
@rickbrouwer are these e2e test failures just flaky? |
Member
|
/run-e2e passed tests: 192failed tests: 5 |
|
Thanks for the re-runs! The e2e failures look like environmental flakes, not a regression from this change — the evidence is that the two most recent full runs fail on completely disjoint sets of tests:
No test fails in both runs. The only two that were plausibly related to this PR's scaling path — Mechanically a regression isn't possible here either: Happy to re-trigger if you'd like a clean run, but I believe the remaining failures are unrelated flakes. |
8 tasks
Member
|
/run-e2e internals passed tests: 36failed tests: 0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
At ~10k ScaledObjects being created at 10/s, the KEDA operator panics:
panic: runtime error: invalid memory address or nil pointer dereference
pkg/scaling/resolver/scale_resolvers.go:731 GetCurrentReplicas
pkg/scaling/executor/scale_scaledobjects.go:44 RequestScale
pkg/scaling/scale_handler.go:282 checkScalers
pkg/scaling/scale_handler.go:199 startScaleLoop
Root cause is the same informer-cache race described in #4389 / tracked
in #4955:
scaledObject.Status.ScaleTargetGVKRcan be nil when thescale loop first invokes
GetCurrentReplicas. The existing codedereferences
.Group/.Kindon the nil pointer, crashing the wholeoperator process and taking down every other scale loop with it.
ResolveScaleTargetPodSpecalready defends against this race(scale_resolvers.go L103-L119).
GetCurrentReplicasdoes not.This PR applies the same pattern.
Fix
If
Status.ScaleTargetGVKRis nil on entry, re-fetch the ScaledObjectvia the client. If it is still nil after re-fetch, return a descriptive
error instead of panicking.
Repro context
Observed during a 10k-ScaledObject KWOK load test at Netflix, after
raising --kube-api-qps/burst to eliminate client-go throttling
(previous 1k bottleneck). Once client-side throttling was gone, fast
ScaledObject creation widened the cache-race window enough that the
nil-pointer panic fired reliably before the 750th object was created.
Tests
TestGetCurrentReplicas_NilScaleTargetGVKRwith three cases:correct replica count (Deployment path)
"probably invalid ScaledObject cache" error
./pkg/scaling/...pass.Fixes / refs
site, doesn't close the underlying race)