Skip to content
This repository was archived by the owner on Nov 5, 2024. It is now read-only.

Add support for setting adapter Affinity#1381

Merged
FranBarrera merged 7 commits into
triggermesh:mainfrom
FranBarrera:add-affinity
Apr 13, 2023
Merged

Add support for setting adapter Affinity#1381
FranBarrera merged 7 commits into
triggermesh:mainfrom
FranBarrera:add-affinity

Conversation

@FranBarrera

Copy link
Copy Markdown
Contributor

Closes #768.

Example:

spec:
  # ...
  adapterOverrides:
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
            - key: zone
              operator: In
              values:
              - zone-a

Result:

kubectl get deployment awss3source-sample -o jsonpath='{.spec.template.spec.affinity}'
{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"zone","operator":"In","values":["zone-a"]}]},"weight":1}]}}

@FranBarrera FranBarrera requested a review from a team March 29, 2023 15:02
@FranBarrera FranBarrera self-assigned this Mar 29, 2023
odacremolbap
odacremolbap previously approved these changes Mar 29, 2023

@odacremolbap odacremolbap left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done.

The CRDs are now humongous, but this is the way of implementing this. ✔️

@tzununbekov

Copy link
Copy Markdown
Member

Oh my... are you sure that we want our CRDs to look like this? This update makes them 5 times bigger (~12Kb vs 62Kb) 😨
Initially, it was about the overrides documentation because it was a custom set of knobs, like a public switch, but do we really need to duplicate a well-known affinity object for this reason?
My main concern is that CRDs are now used in the CLI for object generation and commands auto-completion, so it is often loaded in memory. Making it this big without a clear reason may negatively impact the UX.

@FranBarrera

Copy link
Copy Markdown
Contributor Author

@tzununbekov @odacremolbap I've updated the PR to make the affinity arbitrary and I've added checks at the Webhook to verify things like some wrong names in the affinity.

@tzununbekov

Copy link
Copy Markdown
Member

Thanks. I see a lot of things done for the webhook and I'm struggling to find the reason. Could you point me to the code that implements webhook validation logic?

@odacremolbap odacremolbap requested review from a team and odacremolbap and removed request for odacremolbap April 5, 2023 09:27
@FranBarrera

FranBarrera commented Apr 5, 2023

Copy link
Copy Markdown
Contributor Author

@tzununbekov There are some Unmarshal that are done automatically.

if I add an affinity like this:

  adapterOverrides:
    affinity:
      nodeAffinityy:

it will return this error at the creation time:

admission webhook "defaulting.webhook.triggermesh.io" denied the request: mutation failed: cannot decode incoming new object: json: unknown field "nodeAffinityy"

We could add more checks in the Validate func but I don't see what could we checks at this point

@tzununbekov

Copy link
Copy Markdown
Member

I'm pretty sure that the Defaulting webhook error is a side-effect of internal object processing and not its direct function; the Validation webhook should do that kind of validation. Nevertheless, we can live with that.

I tested the Affinity function and it doesn't work for me. I did some digging and it looks like it is not the way it works in the Knative. Are you sure that it works for you? Please try to create the Kn-based adapter and check if it is online.

@FranBarrera

FranBarrera commented Apr 5, 2023

Copy link
Copy Markdown
Contributor Author

@tzununbekov What do you mean that it doesn't works for you? I've tested creating a source, the adapter is running and checking if the deployment has the affinity set:

kubectl get deployment awss3source-sample -o jsonpath='{.spec.template.spec}'|grep affinity
{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"zone","operator":"In","values":["zone-a"]}]},"weight":1}]}},"

About the Webhook, to check our objects with the webhook we need to add them to the NewDefaultingAdmissionController and NewValidationAdmissionController, if we don't add our objects there it doesn't check anything.

@tzununbekov

Copy link
Copy Markdown
Member

It looks like you tested only a Deployment-based component, not a Kn-based one. Knative serving, by default, does not allow setting the affinity parameter, it should be enabled through the Features and Extensions configuration. Otherwise, adapter deployment won't pass Serving validation. Btw, the same goes for the nodeSelector attribute:

Failed to create adapter Service "foo-transformation": admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: must not set the field(s): spec.template.spec.affinity

IMO, this detail must have been mentioned because without knowing that, setting these adapter overrides may cost some hours of debugging the "outdated adapter specs" kind of issue.

Regarding the webhook implementation, I think we're on different pages. What I'm saying is that there is not much sense in having a webhook controller up and running while none of our components have Validation and Default methods implemented (those numerous SetDefaults and Validate methods that you created for all components and left empty - they are responsible for admission controller logic).
I see two options here:

  1. Do not do any validation on our side and let the K8s controller handle incorrect overrides,
  2. Define a common mechanism to run validation on shared objects (such as adapterOverrides) not to duplicate it on all resources.

I leaning towards the first option. Looks like Knative also does not care about validating these features (here).

@FranBarrera

Copy link
Copy Markdown
Contributor Author

We already have support for tolerations in the same way, I just follow that pattern.

Webhook:
The first option seems wrong to be, if we don't do any validations, k8s will not fail, the result will be the deployment created without the affinity attribute, and I guess that will require some more debug to check for users.

@tzununbekov tzununbekov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of small comments before the merge.

Comment thread cmd/triggermesh-webhook/main.go
Comment thread config/300-awscloudwatchlogssource.yaml
Comment thread pkg/apis/extensions/v1alpha1/register.go Outdated
Comment thread pkg/apis/flow/v1alpha1/register.go
Comment thread pkg/apis/routing/v1alpha1/register.go
Comment thread pkg/apis/sources/v1alpha1/register.go
Comment thread pkg/apis/targets/v1alpha1/register.go
@FranBarrera FranBarrera requested a review from tzununbekov April 12, 2023 14:44
Comment thread config/300-awscloudwatchlogssource.yaml Outdated
@FranBarrera FranBarrera requested a review from tzununbekov April 13, 2023 07:20
@FranBarrera FranBarrera merged commit a3d47a1 into triggermesh:main Apr 13, 2023
@FranBarrera FranBarrera deleted the add-affinity branch April 13, 2023 08:07
@mukundjalan

Copy link
Copy Markdown

@FranBarrera can you please add an example of how can I add nodeSelector to the adapter?

@sebgoa

sebgoa commented May 16, 2023

Copy link
Copy Markdown
Member

Hi @mukundjalan nodeSelector was added in #1380

The syntax in the spec is

spec:
  # ...
  adapterOverrides:
    nodeSelector:
      disktype: ssd

Does that help ? We can discuss more on Slack if you want

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow setting nodeSelector and affinity

5 participants