Repository navigation
fix(listener): stop perpetual diff on server-filled default actions - #101
ack-prow[bot] merged 5 commits into
Conversation
DescribeListeners returns default actions with members the spec never
declared: an Order for every action, a ForwardConfig with weight 1 and
stickiness disabled for a bare targetGroupARN, the #{...} keywords for
unset redirect components and the authentication defaults, while the
k8s-only targetGroupRef and the OIDC client secret never come back. The
generated DeepEqual on Spec.DefaultActions therefore reported a diff on
every reconcile and drove a no-op ModifyListener each time. Skip the
generated comparison and compare DefaultActions in a delta_pre_compare
hook on copies with those members removed, the same shape aws-controllers-k8s#63 uses for
Rule actions.
Signed-off-by: Youssef Omar Bouden <[email protected]>
|
Hi @blacksoxx. Thanks for your PR. I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
gustavodiaz7722
left a comment
There was a problem hiding this comment.
Thanks for tracking this down. I reproduced it against a real ALB, with a local controller and on an EKS cluster where kro v0.9.3 manages the Listener. This PR stops the loop for the shapes in your tests, but three cases still need work.
What drives the rate. The delta alone doesn't produce ~3 calls/s. With no other writer, main makes no ModifyListener calls after creating an HTTP listener. The rate needs a second writer. ACK writes the DescribeListeners form of defaultActions into the spec, and kro server-side-applies the declared form back. defaultActions has no x-kubernetes-list-type, so the applier owns the whole list. Each write bumps the generation and triggers the other controller. On EKS with kro, main made 195 ModifyListener calls in 60s. This PR made 0 for the same Listener.
1. Unset members are ignored, so removals never reach AWS. I removed targetGroupStickinessConfig from a 90/10 forward action. This PR made no call, stickiness stayed enabled, and the CR reported Synced=True. main turned it off. Unit tests show the same for a removed redirect host, a removed weight and a removed OIDC scope. Instead, fill unset members on both sides with what DescribeListeners returns, then compare. Measured defaults: weight 1 for a single target group, stickiness {enabled: false}, redirect #{protocol}, #{port}, #{host}, /#{path}, #{query}, fixed-response contentType: text/plain. The OIDC/Cognito defaults are in the API docs.
2. Target groups are compared by position. DescribeListeners returned forwardConfig.targetGroups in ARN order every time, not in declared order. CreateListener's response echoes the declared order, so the loop starts at the first reconcile after create. A controller restart triggered it at ~4 calls/s. Sorting both sides by ARN fixes it.
3. HTTPS listeners still get AWS's form in the spec (aws-controllers-k8s/community#3115). For an HTTP listener this PR keeps the declared form, because the spec patch is skipped when the comparator says equal. An HTTPS listener without sslPolicy or mutualAuthentication is different. AWS fills both (ELBSecurityPolicy-2016-08 and {mode: off}), the controller writes them into the spec, and that merge patch carries the whole defaultActions list. With this PR and no other writer, such a listener ended with the expanded list in its spec. Adding set: [{method: Create, ignore: from}, {method: Update, ignore: from}] on DefaultActions, the config Rule.Conditions already uses, kept the declared form in the same test. ReadOne still fills latest for drift detection. You offered on aws-controllers-k8s/community#3115 to look at the write-back on top of this PR; I'd include it here, so both issues close together.
I prototyped all three on this branch, regenerated at v0.65.0, and ran it against the same setup. Under kro it made no ModifyListener calls for defaultActions in any shape I tried, including the reordered weighted forward. Removing stickiness took one call. An out-of-band target group change was reverted with one call, then nothing for 20 minutes. The spec kept its declared form in every case, HTTPS included. The six test cases I added fail on this branch and pass on the prototype. I can push it to your branch if that helps.
Smaller items:
- The
{mode: off}delta also costs one ModifyListener after creating any HTTPS listener. It converges, so it can wait for a follow-up. - For a single forward action, DescribeListeners returned no
Order. I didn't test multiple actions. - AWS never returns the OIDC client secret, so changing only the secret produces no delta. That can't be avoided; a code comment and a note in the description would help.
- #63 is still open, so Rule on main uses the generated comparison and loops under kro the same way. Filed as aws-controllers-k8s/community#3125.
- Please rebase onto main (v0.67.0) and add
Fixes aws-controllers-k8s/community#3115next to aws-controllers-k8s/community#3057.
|
/ok-to-test |
… the declared form in the spec The comparator now fills both sides with the members DescribeListeners returns for unset ones (order, the one-entry forwardConfig and action-level targetGroupARN, weight 1, stickiness disabled, the redirect keywords, text/plain, the OIDC and Cognito session defaults), sorts target groups by ARN, and leaves out only the k8s-only references and the OIDC client secret, so a member removed from the spec reaches ModifyListener and a reordered weighted forward does not. DefaultActions is no longer written back from the Create and Update responses, so an HTTPS listener keeps its declared form while ReadOne still fills latest. Regenerated with code-generator v0.67.0 after merging main. Signed-off-by: Youssef Omar Bouden <[email protected]>
gustavodiaz7722
left a comment
There was a problem hiding this comment.
Thanks, this addresses all three points. I re-tested 25acd16 against a real ALB with a local controller and kro v0.9.4:
| Case | ModifyListener calls |
|---|---|
kro Listener, targetGroupARN only |
0 over 4 controller starts, apart from the AWS-side change below; generation stayed 1 |
| kro Listener, 90/10 split declared in non-ARN order | 0 over 4 controller starts |
| Stickiness removed from the spec | 1, and AWS disabled it |
| Target group changed directly in AWS | 1 to revert it, then 0 |
authenticate-oidc + forward listener that AWS returns in another order |
1 no-op call on every reconcile |
The declared defaultActions stayed in the spec for HTTP and HTTPS listeners.
One case remains. With several actions, DescribeListeners doesn't always return them in declared order or by Order. For 3 of the 11 two-action listeners I created, including one made with the plain AWS CLI, it returned forward first although authenticate-oidc was declared first with the lower Order. The order stayed the same across reads. The comparison is positional, so those listeners make one no-op ModifyListener on every reconcile. Since the spec is no longer written back, kro doesn't turn this into a fast loop. Sorting both sides by Order fixes it; see the suggestions below. With them applied, the same listener made 0 calls over two controller starts, and the new test case fails without the sort.
Nit: the description says DescribeListeners returns "an Order for every action", but for a single action it returned none. Tide uses the description as the squash commit message, so it's worth correcting, and mentioning the action sort if you take it.
| normalizeCognito(c.AuthenticateCognitoConfig) | ||
| out[i] = c | ||
| } | ||
| return out |
There was a problem hiding this comment.
Sorting by Order makes the comparison independent of the order DescribeListeners returns the actions in.
| return out | |
| // DescribeListeners returns multiple actions in an order of its own, | |
| // neither declared nor by Order, so compare them sorted by Order. | |
| sort.SliceStable(out, func(i, j int) bool { | |
| var oi, oj int64 | |
| if out[i] != nil { | |
| oi = aws.ToInt64(out[i].Order) | |
| } | |
| if out[j] != nil { | |
| oj = aws.ToInt64(out[j].Order) | |
| } | |
| return oi < oj | |
| }) | |
| return out |
| forwardAsReturned(2, "arn:tg/a"), | ||
| }, | ||
| expectDelta: true, | ||
| }, |
There was a problem hiding this comment.
A case for it, built like the existing ones. It fails on this head and passes with the sort.
| }, | |
| }, | |
| { | |
| name: "multiple actions returned in another order than declared", | |
| desired: []*svcapitypes.Action{ | |
| {Type: aws.String("authenticate-oidc"), Order: aws.Int64(1), AuthenticateOIDCConfig: oidc(aws.String("s3cret"), false)}, | |
| {Type: aws.String("forward"), Order: aws.Int64(2), TargetGroupARN: aws.String("arn:tg/a")}, | |
| }, | |
| observed: []*svcapitypes.Action{ | |
| forwardAsReturned(2, "arn:tg/a"), | |
| {Type: aws.String("authenticate-oidc"), Order: aws.Int64(1), AuthenticateOIDCConfig: oidc(nil, true)}, | |
| }, | |
| expectDelta: false, | |
| }, |
DescribeListeners returns multiple actions in an order of its own, neither declared nor by Order, so a two-action listener compared by position made a no-op ModifyListener on every reconcile. Both sides are sorted by Order before the comparison; the new test case fails without the sort. Merged main (v0.68.0) and regenerated with code-generator v0.68.0, no generated change. Signed-off-by: Youssef Omar Bouden <[email protected]>
|
/LGTM |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: blacksoxx, gustavodiaz7722 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/release label/patch |
Issue #, if available: Fixes aws-controllers-k8s/community#3057, Fixes aws-controllers-k8s/community#3115
Description of changes:
DescribeListeners returns default actions in its own form: an
Orderfor each action when there are several (none for a single one), the actions in an order of its own, a one-entryforwardConfigplus the action-leveltargetGroupARNfor a single target group, weight1, stickiness{enabled: false}, the#{...}keywords for unset redirect components,text/plainfor a fixed response, the OIDC and Cognito session defaults, and target groups in ARN order. The k8s-onlytargetGroupRefand the OIDC client secret never come back. The generated DeepEqual onSpec.DefaultActionstherefore reported a diff on every reconcile and drove a no-op ModifyListener each time, and the write-back of the AWS form into the spec made GitOps tools see the whole list as drifted.DefaultActionsiscompare: is_ignoredand compared in adelta_pre_comparehook on copies normalized to that form: both sides get the members the service fills in when the spec leaves them unset, actions are sorted byOrderand target groups by ARN, and only the references and the client secret are left out. A member removed from the spec, such as stickiness, a weight, a redirect host or an OIDC scope, then differs from what the load balancer still holds and reaches ModifyListener; a reordered weighted forward does not.DefaultActionsis alsoset: ignore: fromon Create and Update, the configRule.Conditionsalready uses, so the spec keeps its declared form for HTTPS listeners too while ReadOne still fillslatestfor drift detection.AWS never returns the OIDC client secret, so a spec that changes only the secret produces no delta; the new secret reaches AWS with the next other change.
hooks_test.gobuilds the desired and observed shapes by hand for forward (ARN, ref, weighted, reordered), redirect, fixed-response and authenticate-oidc actions, with the removal cases, the reordered-actions case and the no-delta cases; seven of the cases fail on the original head. Regenerated with code-generator v0.68.0 after merging main;go build ./...andgo test ./pkg/...pass.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.