Skip to content

Handle configs override for meshed workloads#2500

Closed
ihcsim wants to merge 9 commits intomasterfrom
isim/configs-override-on-update
Closed

Handle configs override for meshed workloads#2500
ihcsim wants to merge 9 commits intomasterfrom
isim/configs-override-on-update

Conversation

@ihcsim
Copy link
Contributor

@ihcsim ihcsim commented Mar 14, 2019

This PR is a different take to handle configs override while updating meshed workloads.

Assumptions

The proxy injector is only interested in two three types of workloads:

  • unmeshed workload with the linkerd.io/inject: enabled annotation
  • meshed workload with the config.linkerd.io/* annotations
  • NEW: unmeshed workload with the linkerd.io/inject: enabled and config.linkerd.io/* annotations

Everything else should be a passthrough.

Implementations

  • Introduces a check to verify if a workload should be injected and/or config-overridden, OR if it just needs to be config-overridden. If neither, the rest of the injection process will be skipped.
  • Separated out the logic to generate the different patches for unmeshed and meshed workloads.

PS The whole thing probably doesn't work yet and the unit tests are failing. Just thought I will share it to get some early feedback.

Signed-off-by: Ivan Sim [email protected]

@ihcsim ihcsim added priority/P0 Release Blocker area/inject labels Mar 14, 2019
@ihcsim ihcsim self-assigned this Mar 14, 2019
@ihcsim ihcsim requested review from alpeb, klingerf and siggy March 14, 2019 22:46
}

// AddRootLabels adds all the pod labels into the root workload (e.g. Deployment)
func (conf *ResourceConfig) AddRootLabels(patch *Patch) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just grouping all the exported methods together.

@ihcsim ihcsim force-pushed the isim/configs-override-on-update branch from 35b4ba0 to 55613bf Compare March 14, 2019 22:51
Copy link
Member

@alpeb alpeb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a great start 👍
After a first review pass I added a couple of remarks below.

case envVarProxyInboundListener:
container.Env[i].Value = conf.proxyInboundListener()
case envVarProxyDestinationProfileSuffixes:
container.Env[i].Value = conf.proxyDestinationProfileSuffixes()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are calls here that were already made in newProxyContainer(); not sure if there's a way to avoid that without making this snipped unnecessarily more complex.


// populate the proxy's configurable properties with either overrides or
// defaults
proxy := conf.setProxyConfigs(identity)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say shouldInject() returned false because HostNetwork is true, yet the user added config annotations unknowingly. This will inject the proxy when it shouldn't. I think shouldInject() needs to be split into different methods to be able to distinguish when pods should not be injected at all, and when they are already injected.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might also help clarifying a bit the logic to decide when to inject/override inside GetPatch(), which is currently not obvious IMO.

Copy link
Contributor Author

@ihcsim ihcsim Mar 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, definitely a bug in the if statement. As a matter of fact, it breaks linkerd inject, because the received YAML will already have a proxy spec, and if config annotations are present, the bug you described will occur, leading to some downstream complication in the YAML (i.e. the CLI calls GetPatch() once, and then the webhook calls it again when the deployment is created.) 💥

I'm not sure how splitting up the shouldInject() will help, as I imagine it will require some clever ways to chain these different methods together, to satisfy both the webhook and CLI code path.

What I ended up doing was to perform the shouldInject() checks earlier in both the CLI and webhhook. So if shouldInject() failed, we won't even call the GetPatch() function. This also helps to keep GetPatch() to focus only on generating JSON patches, and the decision on whether the configs need to be overridden.

The commit is at dde345f the HEAD of this branch. This seems to work for the webhook, but I'm now battling with how the CLI reports unit tests. Maybe you can take a look at that commit to see if it makes sense? Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I ended up doing was to perform the shouldInject() checks earlier in both the CLI and webhhook. So if shouldInject() failed, we won't even call the GetPatch() function. This also helps to keep GetPatch() to focus only on generating JSON patches, and the decision on whether the configs need to be overridden.

Yes, this looks much better to me 👍
Note though that your shouldInject() functions will return false whenever there's already a proxy attached because healthcheck.HasExistingSidecars() doesn't distinguish between linkerd proxies and 3rd party proxies, so this will prevent config overriding. That should be an easy fix 😉

Ivan Sim added 4 commits March 17, 2019 21:06
Introduces a check to verify if a workload should be injected OR if it's
configs should be overridden. If neither, the rest of the injection
process will be skipped.

Separated out the different logic to generate the patches for unmeshed
and meshed workloads.

Signed-off-by: Ivan Sim <[email protected]>
Signed-off-by: Ivan Sim <[email protected]>
When the CLI submits a meshed YAML, it will be intercepted by the
webhook. This change ensures that any config override annotations in the
YAML won't trigger the JSON patch creation process.

Signed-off-by: Ivan Sim <[email protected]>
Signed-off-by: Ivan Sim <[email protected]>
@ihcsim ihcsim force-pushed the isim/configs-override-on-update branch 4 times, most recently from dbcdc88 to 04f9da7 Compare March 18, 2019 05:47
This avoids the problem of workload meshed by the CLI getting re-meshed
again by the proxy injector as a result of the webhook intercepting the
pod creation and update events.

Signed-off-by: Ivan Sim <[email protected]>
@ihcsim ihcsim force-pushed the isim/configs-override-on-update branch from 04f9da7 to ca6b774 Compare March 18, 2019 06:19
Ivan Sim added 4 commits March 18, 2019 09:12
Signed-off-by: Ivan Sim <[email protected]>
The shouldInject check applies only to create operation, in order to
support config override during updates.

Also. ensure that the created-by annotation and 'root' labels are added to the JSON patch
during create operations only.

Signed-off-by: Ivan Sim <[email protected]>
@ihcsim ihcsim force-pushed the isim/configs-override-on-update branch from ad547c0 to 48646df Compare March 18, 2019 21:38
Copy link
Contributor

@klingerf klingerf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a good start, and I left a bunch of miscellaneous feedback below.

if err != nil {
return nil, nil, err
if err == inject.ErrUnsupportedResourceKind {
report.UnsupportedResource = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code is unreachable as written. The shouldInject call returns false if conf.pod.spec is nil, and that causes this function to return immediately. GetPatch only returns ErrUnsupportedResourceKind if conf.pod.spec is nil, but it can't be nil by the time we call it.

if err == inject.ErrUnsupportedResourceKind {
report.UnsupportedResource = true
}
return bytes, []inject.Report{*report}, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we've encountered an error, shouldn't we return nil, nil, err here instead? That's how we're handling errors further down in this function.

return nil, nil, err
}
return injectedYAML, reports, nil
return injectedYAML, []inject.Report{*report}, nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than build the report slice multiple times inside this function, my preference to build it once, immediately after the report is returned. Something like:

diff --git a/cli/cmd/inject.go b/cli/cmd/inject.go
index 275ce53d..9822b46a 100644
--- a/cli/cmd/inject.go
+++ b/cli/cmd/inject.go
@@ -110,9 +110,10 @@ func (rt resourceTransformerInject) transform(bytes []byte) ([]byte, []inject.Re
 	if err != nil {
 		return nil, nil, err
 	}
+	reports := []inject.Report{*report}
 
 	if !shouldInject(conf) {
-		return bytes, []inject.Report{*report}, nil
+		return bytes, reports, nil
 	}
 
 	p, err := conf.GetPatch()
@@ -121,7 +122,7 @@ func (rt resourceTransformerInject) transform(bytes []byte) ([]byte, []inject.Re
 	}
 
 	if p.IsEmpty() {
-		return bytes, []inject.Report{*report}, nil
+		return bytes, reports, nil
 	}
 
 	p.AddCreatedByPodAnnotation(k8s.CreatedByAnnotationValue())
@@ -130,7 +131,7 @@ func (rt resourceTransformerInject) transform(bytes []byte) ([]byte, []inject.Re
 		return nil, nil, err
 	}
 	if patchJSON == nil {
-		return bytes, []inject.Report{*report}, nil
+		return bytes, reports, nil
 	}
 	log.Infof("patch generated for: %s", conf)
 	log.Debugf("patch: %s", patchJSON)
@@ -150,7 +151,7 @@ func (rt resourceTransformerInject) transform(bytes []byte) ([]byte, []inject.Re
 	if err != nil {
 		return nil, nil, err
 	}
-	return injectedYAML, []inject.Report{*report}, nil
+	return injectedYAML, reports, nil
 }
 
 func (resourceTransformerInject) generateReport(reports []inject.Report, output io.Writer) {

}
if !nonEmpty {
return admissionResponse, nil
log.Infof("received %s", conf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind changing this to log.Debugf? It's a bit too verbose to be at the info level.

Allowed: true,
}
admissionResponse.Patch = patchJSON
admissionResponse.PatchType = &patchType
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can set these two fields when you initialize the AdmissionResponse struct. In fact, I'd just return that struct directly:

diff --git a/controller/proxy-injector/webhook.go b/controller/proxy-injector/webhook.go
index 044d059e..7d167bb5 100644
--- a/controller/proxy-injector/webhook.go
+++ b/controller/proxy-injector/webhook.go
@@ -148,14 +148,12 @@ func (w *Webhook) inject(request *admissionv1beta1.AdmissionRequest) (*admission
 	log.Debugf("patch: %s", patchJSON)
 
 	patchType := admissionv1beta1.PatchTypeJSONPatch
-	admissionResponse := &admissionv1beta1.AdmissionResponse{
-		UID:     request.UID,
-		Allowed: true,
-	}
-	admissionResponse.Patch = patchJSON
-	admissionResponse.PatchType = &patchType
-
-	return admissionResponse, nil
+	return &admissionv1beta1.AdmissionResponse{
+		UID:       request.UID,
+		Allowed:   true,
+		Patch:     patchJSON,
+		PatchType: &patchType,
+	}, nil
 }
 
 func shouldInject(conf *inject.ResourceConfig) bool {


// PodUsingHostNetwork returns true if the HostNetwork property is true.
func (conf *ResourceConfig) PodUsingHostNetwork() bool {
return conf.pod.spec != nil && conf.pod.spec.HostNetwork
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to my comment in the CLI code, it looks like this func is only ever called after a call to HasPayload, so it should never be the case the conf.pod.spec is nil. What if we make HasPayload package private (e.g. hasPayload), and just build it into these functions directly so that external packages don't have to do their own validation that the payload is present?

r.HostNetwork = conf.pod.spec.HostNetwork
r.Sidecar = healthcheck.HasExistingSidecars(conf.pod.spec)
r.UDP = checkUDPPorts(conf.pod.spec)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This func now duplicates a bunch of code that exists in pkg/inject/inject.go. Can you instead write it like:

func (r *Report) update(conf *ResourceConfig) {
	r.InjectDisabled = conf.InjectDisabled()
	r.HostNetwork = conf.PodUsingHostNetwork()
	r.Sidecar = conf.HasExistingProxy()
	r.UDP = checkUDPPorts(conf.pod.spec)
}


// CreatedByCLI is used by the CreatedByAnnotation to indicate that the
// proxy is injected by the CLI.
CreatedByCLI = "linkerd/cli"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unused?

// ShouldOverrideConfig returns true if the workload has existing sidecars and
// config override annotations.
func (conf *ResourceConfig) ShouldOverrideConfig() bool {
return healthcheck.HasExistingSidecars(conf.pod.spec) && hasOverrideAnnotations(conf.pod.meta)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than only doing the override if at least config.linkerd.io annotation is present, I think it would better to use something like k8s.IsMeshed to determine if the pod is inject, and then always override if inject, even if it doesn't contain override annotations. My thinking is that eventually we might allow users to change the contents of the proxy configmap after install, and we won't pick up the change if we're only looking for override annotations.

@ihcsim
Copy link
Contributor Author

ihcsim commented Mar 19, 2019

Here's a summary of the discussion with @klingerf @alpeb:

  1. the ShouldInject checks will remain separated in the CLI and webhook, as seen in this PR
  2. the ParseMetaAndYaml will be used to generate the report early before any proxy injection, as seen in this PR
  3. the report struct will be extended to capture the value of any namespace level linkerd.io/inject annotation. This struct will be used to decide whether a proxy injection should happen
  4. the webhook's ShouldInject will be updated to check only for the presence of 3rd party proxies, implying that any YAML manifests with existing Linkerd proxy will be "re-injected" to support config override
  5. put this PR on-hold, and proceed with CLI Inject With Proxy Config Overrides Annotations #2447, so that when the webhook intercepts YAML manifests submitted by the CLI, the proxy will be "re-injected" with the correct values from the config annotations.

@ihcsim
Copy link
Contributor Author

ihcsim commented Mar 28, 2019

See #2590.

@ihcsim ihcsim closed this Mar 28, 2019
@ihcsim ihcsim deleted the isim/configs-override-on-update branch March 28, 2019 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants