Skip to content

fix(listener): stop perpetual diff on server-filled default actions - #101

Merged
ack-prow[bot] merged 5 commits into
aws-controllers-k8s:mainfrom
blacksoxx:listener-default-actions-delta
Nov 11, 2026
Merged

ack-prow[bot] merged 5 commits into
aws-controllers-k8s:mainfrom
blacksoxx:listener-default-actions-delta

Conversation

@blacksoxx

@blacksoxx blacksoxx commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available: Fixes aws-controllers-k8s/community#3057, Fixes aws-controllers-k8s/community#3115

Description of changes:
DescribeListeners returns default actions in its own form: an Order for each action when there are several (none for a single one), the actions in an order of its own, a one-entry forwardConfig plus the action-level targetGroupARN for a single target group, weight 1, stickiness {enabled: false}, the #{...} keywords for unset redirect components, text/plain for a fixed response, the OIDC and Cognito session defaults, and target groups in ARN order. The k8s-only targetGroupRef and the OIDC client secret never come back. The generated DeepEqual on Spec.DefaultActions therefore reported a diff on every reconcile and drove a no-op ModifyListener each time, and the write-back of the AWS form into the spec made GitOps tools see the whole list as drifted.

DefaultActions is compare: is_ignored and compared in a delta_pre_compare hook on copies normalized to that form: both sides get the members the service fills in when the spec leaves them unset, actions are sorted by Order and target groups by ARN, and only the references and the client secret are left out. A member removed from the spec, such as stickiness, a weight, a redirect host or an OIDC scope, then differs from what the load balancer still holds and reaches ModifyListener; a reordered weighted forward does not. DefaultActions is also set: ignore: from on Create and Update, the config Rule.Conditions already uses, so the spec keeps its declared form for HTTPS listeners too while ReadOne still fills latest for drift detection.

AWS never returns the OIDC client secret, so a spec that changes only the secret produces no delta; the new secret reaches AWS with the next other change.

hooks_test.go builds the desired and observed shapes by hand for forward (ARN, ref, weighted, reordered), redirect, fixed-response and authenticate-oidc actions, with the removal cases, the reordered-actions case and the no-delta cases; seven of the cases fail on the original head. Regenerated with code-generator v0.68.0 after merging main; go build ./... and go test ./pkg/... pass.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

DescribeListeners returns default actions with members the spec never
declared: an Order for every action, a ForwardConfig with weight 1 and
stickiness disabled for a bare targetGroupARN, the #{...} keywords for
unset redirect components and the authentication defaults, while the
k8s-only targetGroupRef and the OIDC client secret never come back. The
generated DeepEqual on Spec.DefaultActions therefore reported a diff on
every reconcile and drove a no-op ModifyListener each time. Skip the
generated comparison and compare DefaultActions in a delta_pre_compare
hook on copies with those members removed, the same shape aws-controllers-k8s#63 uses for
Rule actions.

Signed-off-by: Youssef Omar Bouden <[email protected]>
@ack-prow
ack-prow Bot requested review from jlbutler and michaelhtm October 13, 2026 10:06
@ack-prow ack-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 13, 2026
@ack-prow

ack-prow Bot commented Oct 13, 2026

Copy link
Copy Markdown

Hi @blacksoxx. Thanks for your PR.

I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@gustavodiaz7722 gustavodiaz7722 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down. I reproduced it against a real ALB, with a local controller and on an EKS cluster where kro v0.9.3 manages the Listener. This PR stops the loop for the shapes in your tests, but three cases still need work.

What drives the rate. The delta alone doesn't produce ~3 calls/s. With no other writer, main makes no ModifyListener calls after creating an HTTP listener. The rate needs a second writer. ACK writes the DescribeListeners form of defaultActions into the spec, and kro server-side-applies the declared form back. defaultActions has no x-kubernetes-list-type, so the applier owns the whole list. Each write bumps the generation and triggers the other controller. On EKS with kro, main made 195 ModifyListener calls in 60s. This PR made 0 for the same Listener.

1. Unset members are ignored, so removals never reach AWS. I removed targetGroupStickinessConfig from a 90/10 forward action. This PR made no call, stickiness stayed enabled, and the CR reported Synced=True. main turned it off. Unit tests show the same for a removed redirect host, a removed weight and a removed OIDC scope. Instead, fill unset members on both sides with what DescribeListeners returns, then compare. Measured defaults: weight 1 for a single target group, stickiness {enabled: false}, redirect #{protocol}, #{port}, #{host}, /#{path}, #{query}, fixed-response contentType: text/plain. The OIDC/Cognito defaults are in the API docs.

2. Target groups are compared by position. DescribeListeners returned forwardConfig.targetGroups in ARN order every time, not in declared order. CreateListener's response echoes the declared order, so the loop starts at the first reconcile after create. A controller restart triggered it at ~4 calls/s. Sorting both sides by ARN fixes it.

3. HTTPS listeners still get AWS's form in the spec (aws-controllers-k8s/community#3115). For an HTTP listener this PR keeps the declared form, because the spec patch is skipped when the comparator says equal. An HTTPS listener without sslPolicy or mutualAuthentication is different. AWS fills both (ELBSecurityPolicy-2016-08 and {mode: off}), the controller writes them into the spec, and that merge patch carries the whole defaultActions list. With this PR and no other writer, such a listener ended with the expanded list in its spec. Adding set: [{method: Create, ignore: from}, {method: Update, ignore: from}] on DefaultActions, the config Rule.Conditions already uses, kept the declared form in the same test. ReadOne still fills latest for drift detection. You offered on aws-controllers-k8s/community#3115 to look at the write-back on top of this PR; I'd include it here, so both issues close together.

I prototyped all three on this branch, regenerated at v0.65.0, and ran it against the same setup. Under kro it made no ModifyListener calls for defaultActions in any shape I tried, including the reordered weighted forward. Removing stickiness took one call. An out-of-band target group change was reverted with one call, then nothing for 20 minutes. The spec kept its declared form in every case, HTTPS included. The six test cases I added fail on this branch and pass on the prototype. I can push it to your branch if that helps.

Smaller items:

  • The {mode: off} delta also costs one ModifyListener after creating any HTTPS listener. It converges, so it can wait for a follow-up.
  • For a single forward action, DescribeListeners returned no Order. I didn't test multiple actions.
  • AWS never returns the OIDC client secret, so changing only the secret produces no delta. That can't be avoided; a code comment and a note in the description would help.
  • #63 is still open, so Rule on main uses the generated comparison and loops under kro the same way. Filed as aws-controllers-k8s/community#3125.
  • Please rebase onto main (v0.67.0) and add Fixes aws-controllers-k8s/community#3115 next to aws-controllers-k8s/community#3057.

@gustavodiaz7722

Copy link
Copy Markdown
Member

/ok-to-test

@ack-prow ack-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 3, 2026
… the declared form in the spec

The comparator now fills both sides with the members DescribeListeners returns for unset ones (order, the one-entry forwardConfig and action-level targetGroupARN, weight 1, stickiness disabled, the redirect keywords, text/plain, the OIDC and Cognito session defaults), sorts target groups by ARN, and leaves out only the k8s-only references and the OIDC client secret, so a member removed from the spec reaches ModifyListener and a reordered weighted forward does not. DefaultActions is no longer written back from the Create and Update responses, so an HTTPS listener keeps its declared form while ReadOne still fills latest. Regenerated with code-generator v0.67.0 after merging main.

Signed-off-by: Youssef Omar Bouden <[email protected]>

@gustavodiaz7722 gustavodiaz7722 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this addresses all three points. I re-tested 25acd16 against a real ALB with a local controller and kro v0.9.4:

Case ModifyListener calls
kro Listener, targetGroupARN only 0 over 4 controller starts, apart from the AWS-side change below; generation stayed 1
kro Listener, 90/10 split declared in non-ARN order 0 over 4 controller starts
Stickiness removed from the spec 1, and AWS disabled it
Target group changed directly in AWS 1 to revert it, then 0
authenticate-oidc + forward listener that AWS returns in another order 1 no-op call on every reconcile

The declared defaultActions stayed in the spec for HTTP and HTTPS listeners.

One case remains. With several actions, DescribeListeners doesn't always return them in declared order or by Order. For 3 of the 11 two-action listeners I created, including one made with the plain AWS CLI, it returned forward first although authenticate-oidc was declared first with the lower Order. The order stayed the same across reads. The comparison is positional, so those listeners make one no-op ModifyListener on every reconcile. Since the spec is no longer written back, kro doesn't turn this into a fast loop. Sorting both sides by Order fixes it; see the suggestions below. With them applied, the same listener made 0 calls over two controller starts, and the new test case fails without the sort.

Nit: the description says DescribeListeners returns "an Order for every action", but for a single action it returned none. Tide uses the description as the squash commit message, so it's worth correcting, and mentioning the action sort if you take it.

normalizeCognito(c.AuthenticateCognitoConfig)
out[i] = c
}
return out

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorting by Order makes the comparison independent of the order DescribeListeners returns the actions in.

Suggested change
return out
// DescribeListeners returns multiple actions in an order of its own,
// neither declared nor by Order, so compare them sorted by Order.
sort.SliceStable(out, func(i, j int) bool {
var oi, oj int64
if out[i] != nil {
oi = aws.ToInt64(out[i].Order)
}
if out[j] != nil {
oj = aws.ToInt64(out[j].Order)
}
return oi < oj
})
return out

forwardAsReturned(2, "arn:tg/a"),
},
expectDelta: true,
},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A case for it, built like the existing ones. It fails on this head and passes with the sort.

Suggested change
},
},
{
name: "multiple actions returned in another order than declared",
desired: []*svcapitypes.Action{
{Type: aws.String("authenticate-oidc"), Order: aws.Int64(1), AuthenticateOIDCConfig: oidc(aws.String("s3cret"), false)},
{Type: aws.String("forward"), Order: aws.Int64(2), TargetGroupARN: aws.String("arn:tg/a")},
},
observed: []*svcapitypes.Action{
forwardAsReturned(2, "arn:tg/a"),
{Type: aws.String("authenticate-oidc"), Order: aws.Int64(1), AuthenticateOIDCConfig: oidc(nil, true)},
},
expectDelta: false,
},

DescribeListeners returns multiple actions in an order of its own, neither declared nor by Order, so a two-action listener compared by position made a no-op ModifyListener on every reconcile. Both sides are sorted by Order before the comparison; the new test case fails without the sort. Merged main (v0.68.0) and regenerated with code-generator v0.68.0, no generated change.

Signed-off-by: Youssef Omar Bouden <[email protected]>
@gustavodiaz7722

Copy link
Copy Markdown
Member

/LGTM

@ack-prow ack-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 11, 2026
@ack-prow

ack-prow Bot commented Nov 11, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: blacksoxx, gustavodiaz7722

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ack-prow ack-prow Bot added the approved label Nov 11, 2026
@gustavodiaz7722

Copy link
Copy Markdown
Member

/release label/patch

@ack-prow
ack-prow Bot merged commit 30036a4 into aws-controllers-k8s:main Nov 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test.

Projects

None yet

3 participants