Infer SupportedFeatures in Conformance Tests (GEP-2162) [#3759] #3848

bexxmodd · 2025-06-10T04:53:45Z

What type of PR is this?
/kind feature
/area conformance-test

What this PR does / why we need it:
This change automatically infers SupportedFeatures from the GatewayClass' Status to run conformance tests against. This eliminates need for manually supplying features you are testing and any confusion for what features are supported that pass conformance.

Inferred SupportedFeatures will take precedence over all other test related flags. Existing flags like, --supported-features, --all-features, --conformance-profiles, or --exempt-features should only be used for development/testing. Final report for submission should have inferred supported features.

Generated Conformance report will have InferredSupportedFeatures that determines the validity of conformance tests.

Feature was testing using Kong's gateway-operator. Steps:

Install specific version of experimental CRDs (v1.3.0 in this case):

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/experimental-install.yaml

Clone kong’s gateway operator repo (currently >= v1.6.0) and cd into it.
make build
Create Gateway and GatewayClass

Run with experimental flag enabled:

GATEWAY_OPERATOR_ENABLE_GATEWAY_API_EXPERIMENTAL=true make run

Run conformance tests from gateway-api folder:

go test -v ./conformance --gateway-class istio --run TestConformance --cleanup-base-resources=false

Does this PR introduce a user-facing change?:
The change is the user doesn't need to manually supply features test if they want to generate passing report. They still can use flags like --supported-features and --exempt-features but they will be purely for development purposes

/fixes #3759

This change automatically infers SupportedFeatures from the GatewayClass' Status to run conformance tests against. This eliminates need for manually supplying features you are testing and any confusion for what features are supported that pass conformance. If any other features related flags are supplied Status' SupportedFeatures will not be inferred and test report will get failing result.

…r tests were manually supplied or inferred from GatewayClass

…ied and refactored some code around it.

…esting and if they were inferred or not.

k8s-ci-robot · 2025-06-10T04:53:49Z

@bexxmodd: The label(s) area/conformance cannot be applied, because the repository doesn't have them.

In response to this:

What type of PR is this?
/kind feature
/area conformance

What this PR does / why we need it:
This features automatically infers SupportedFeatures from the GatewayClass' Status for to run conformance tests against. This eliminates need for manually supplying features you are testing and any confusion for what features are supported that pass conformance.

Does this PR introduce a user-facing change?:
The change is the user doesn't need to manually supply features test if they want to generate passing report. They still can use flags like --supported-features and --exempt-features but they will be purely for development purposes

P.S. There are some decisions to make that will be addressed in follow up PRs:

Should/How inferred features should be prioritized over Conformance Profiles?

What happens to --all-features flag as if user claims that they support all the features they will also be inferred.

How to report the Conformance Test failure if features where not inferred (manually supplied or some were except).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-06-10T04:53:55Z

Hi @bexxmodd. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bexxmodd · 2025-06-10T04:56:37Z

Hi @LiorLieberman Please take a look

LiorLieberman · 2025-06-10T05:43:36Z

/cc @LiorLieberman

…ature flags.

robscott · 2025-06-10T20:57:09Z

/ok-to-test

LiorLieberman · 2025-06-10T22:25:41Z

suggest linking to the PR to #3759

And retitle to indicate the change as well.

…hSupportedFeatures function.

…mance suite.

bexxmodd · 2025-06-12T04:43:42Z

Added unit tests.

LiorLieberman

Thanks @bexxmodd! looks good to me with a few small comments.

Also suggest re-title based on my comment here #3848 (comment)

holding lgtm as it will be removed after you add more commits

conformance/apis/v1/conformancereport.go

conformance/conformance.go

conformance/utils/suite/suite.go

conformance/utils/suite/reports.go

bexxmodd · 2025-06-12T16:42:58Z

thanks for reviewing @LiorLieberman . Resolved comments and updated title.

LiorLieberman

This is awesome and LGTM, thanks @bexxmodd !

Would be useful if you update the PR description with exactly what its doing, and with the name and version of implementation that you checked (I believe it was Kong)

/lgtm

LiorLieberman · 2025-06-12T20:15:36Z

/hold
oh and one more things - can you update the conformance website guide?
Feel free to remove the hold after that

LiorLieberman · 2025-06-12T20:16:03Z

Also, can you fix release note by either say what you have done or adding NONE in that code block?

k8s-ci-robot · 2025-06-12T21:35:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bexxmodd, LiorLieberman, youngnick

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~conformance/OWNERS~~ [LiorLieberman,youngnick]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bexxmodd · 2025-06-12T21:36:14Z

-> Update description with steps how it was tested.
-> Added release notes.

<- I'll update conformance docs in a separate PR.

/hold cancel

LiorLieberman · 2025-06-12T21:55:10Z

/lgtm

…igs#3759] (kubernetes-sigs#3848) * SupportedFeatures * Added inferredSupportedFeatures to determine and report if feature for tests were manually supplied or inferred from GatewayClass * Corrected suite unit tests after addition of a new field. * Added logic of determining if supportedFeatures are inferred or supplied and refactored some code around it. * moved SupportedFeatures init after scheme addition * Cleaned up and organized code to properly determine features we are testing and if they were inferred or not. * Cleanup v2 * Make inferring supportedFeatures to take precedence over all other feature flags. * Fix when no profile was setting inferred to false. * remove debug log * Switched args arrangement so ctx is the first argument passed to fetchSupportedFeatures function. * Updated context as it's unclear where it should be supplied from * removed SupportedFeatures struct and method for determining supported features. * Refactored logic of determining supported features and add flag for the report in cSuite. * Reversed flag parsing for some values. * Tweaked logging options. * Cleaned up last places for SupportedFeatures struct. * Wrote unit tests for supported features determination logic in conformance suite. * Formatting fixes. * Resolved comments. * Fixed formatting. * Updated conformance docs. * Removed doc update so it can be submitted as a seperate PR.

bexxmodd added 7 commits June 5, 2025 15:25

SupportedFeatures

2c3b74f

Added inferredSupportedFeatures to determine and report if feature fo…

a04fd94

…r tests were manually supplied or inferred from GatewayClass

Corrected suite unit tests after addition of a new field.

1e4b6f8

Added logic of determining if supportedFeatures are inferred or suppl…

d57efcc

…ied and refactored some code around it.

moved SupportedFeatures init after scheme addition

581c12a

Cleaned up and organized code to properly determine features we are t…

92b0599

…esting and if they were inferred or not.

Cleanup v2

b061105

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 10, 2025

k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 10, 2025

k8s-ci-robot requested review from candita and kflynn June 10, 2025 04:53

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 10, 2025

k8s-ci-robot requested a review from LiorLieberman June 10, 2025 05:43

Make inferring supportedFeatures to take precedence over all other fe…

d0cfde9

…ature flags.

bexxmodd changed the title ~~[WIP] Implementing SupportedFeatures (GEP-2162)~~ Implementing SupportedFeatures (GEP-2162) Jun 10, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2025

bexxmodd added 2 commits June 10, 2025 09:13

Fix when no profile was setting inferred to false.

d90127e

remove debug log

d35725c

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 10, 2025

Switched args arrangement so ctx is the first argument passed to fetc…

fb37630

…hSupportedFeatures function.

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 12, 2025

Wrote unit tests for supported features determination logic in confor…

420681f

…mance suite.

Formatting fixes.

29facbb

LiorLieberman approved these changes Jun 12, 2025

View reviewed changes

Resolved comments.

39c3f40

bexxmodd changed the title ~~Implementing SupportedFeatures (GEP-2162)~~ Inference of SupportedFeatures in Conformance Tests (GEP-2162) [#3759] Jun 12, 2025

bexxmodd changed the title ~~Inference of SupportedFeatures in Conformance Tests (GEP-2162) [#3759]~~ Infer SupportedFeatures in Conformance Tests (GEP-2162) [#3759] Jun 12, 2025

Fixed formatting.

2309852

LiorLieberman approved these changes Jun 12, 2025

View reviewed changes

k8s-ci-robot assigned LiorLieberman Jun 12, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 12, 2025

Updated conformance docs.

525f3cb

Removed doc update so it can be submitted as a seperate PR.

ba6cc8d

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 12, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 12, 2025

k8s-ci-robot merged commit ed2bd6b into kubernetes-sigs:main Jun 12, 2025
13 checks passed

bexxmodd deleted the GEP-2162 branch June 13, 2025 02:29

timflannagan mentioned this pull request Jun 17, 2025

Switch conformance from allowlist to denylist kgateway-dev/kgateway#11437

Closed

bexxmodd mentioned this pull request Oct 14, 2025

REQUEST: New membership for bexxmodd kubernetes/org#5926

Closed

11 tasks

Infer SupportedFeatures in Conformance Tests (GEP-2162) [#3759] #3848

Infer SupportedFeatures in Conformance Tests (GEP-2162) [#3759] #3848

Uh oh!

Conversation

bexxmodd commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jun 10, 2025

Uh oh!

k8s-ci-robot commented Jun 10, 2025

Uh oh!

bexxmodd commented Jun 10, 2025

Uh oh!

LiorLieberman commented Jun 10, 2025

Uh oh!

robscott commented Jun 10, 2025

Uh oh!

LiorLieberman commented Jun 10, 2025

Uh oh!

bexxmodd commented Jun 12, 2025

Uh oh!

LiorLieberman left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bexxmodd commented Jun 12, 2025

Uh oh!

LiorLieberman left a comment

Choose a reason for hiding this comment

Uh oh!

LiorLieberman commented Jun 12, 2025

Uh oh!

LiorLieberman commented Jun 12, 2025

Uh oh!

k8s-ci-robot commented Jun 12, 2025

Uh oh!

bexxmodd commented Jun 12, 2025

Uh oh!

LiorLieberman commented Jun 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bexxmodd commented Jun 10, 2025 •

edited

Loading

LiorLieberman left a comment •

edited

Loading