Conversation
(Moved from the wiki to a PR so it can be commented more easily)
The current CLI install process uses Helm libraries under the hood, but just as a template library. There is currently a Helm chart under the `charts` directory that allows installing the control plane, but uninjected, and there's no `values.yaml` file provided.
The intention here is to provide a new chart with an injected control plane so that users can install linkerd through a simple `helm install incubator/linkerd2` command.
Helm charts can't rely on go code besides the functions provided by the go template library, the Sprig library, and a few extra functions provided by Helm itself. This implies a few compromises:
- We can't validate the install options provided in `values.yaml`. Instead, a new set of `linkerd check` checks could help catching invalid options, post-install.
- We should provide a comprehensive `values.yaml` file that would contain the most common settings, but heavily annotated to instruct users about alternate settings for advanced scenarios.
- Helm's crypto functions only allow us to use RSA certs/keys. We can move the cert/keys from the webhook configs (`proxy-injector` and `sp-validator`) to use RSA. As for the trust root for identity, we decided the user should provide their own in `values.yaml`. The docs should have instructions on how to generate that cert/key.
These compromises entail a less straightforward experience than what `linkerd install` provides, so the Helm installation alternative should be considered an "advanced" feature.
New alternative install workflow through Helm
-----------------------------------------------
```
helm install incubator/linkerd2
```
That would install linkerd using the most common settings. The `NOTES.txt` file (rendered and shown when this command completes) could provide the follow instructions/warnings:
- Warn that the identity trust root should have been provided, and show instructions on how to generate it.
- Instructions on how to, optionally, install the linkerd CLI
- Instructions on running `linkerd check` to verify everything is ok
- Instructions on how to change the most basic settings
- Instructions on how to get ahold of `values.yaml` containing all the possible settings?
This will replace the current single chart under the `charts` directory.
```
charts
├── control-plane
│ ├── charts
│ │ └── partials -> ../../partials
│ ├── Chart.yaml
│ ├── templates
│ │ ├── config.yaml
│ │ ├── controller-rbac.yaml
│ │ ├── controller.yaml
│ │ ├── grafana-rbac.yaml
│ │ ├── grafana.yaml
│ │ ├── identity-rbac.yaml
│ │ ├── identity.yaml
│ │ ├── namespace.yaml
│ │ ├── NOTES.txt
│ │ ├── prometheus-rbac.yaml
│ │ ├── prometheus.yaml
│ │ ├── proxy_injector-rbac.yaml
│ │ ├── proxy_injector.yaml
│ │ ├── psp.yaml
│ │ ├── _resources.yaml
│ │ ├── serviceprofile-crd.yaml
│ │ ├── sp_validator-rbac.yaml
│ │ ├── sp_validator.yaml
│ │ ├── tap-rbac.yaml
│ │ ├── tap.yaml
│ │ ├── trafficsplit-crd.yaml
│ │ ├── web-rbac.yaml
│ │ └── web.yaml
│ └── values.yaml
├── data-plane
│ ├── charts
│ │ └── partials -> ../../partials
│ ├── Chart.yaml
│ ├── templates
│ │ └── patch.json
│ └── values.yaml
└── partials
├── charts
├── Chart.yaml
├── templates
│ ├── _debug.yaml
│ ├── _metadata.yaml
│ ├── _proxy.yaml
│ ├── _proxy-init.yaml
│ └── _volumes.yaml
└── values.yaml
```
Current mechanisms and changes
-------------------------------
The user experience for the current way of doing things remains the same. There will be some changes under the hood though.
Currently the `proxy-injector` webhook is invoked for any pod that gets created. If the pod's namespace or the pod itself contains a `linkerd.io/inject: enabled` annotation then the webhook relies on the library `pkg/inject/inject.go` to programatically generate a proxy container (and a proxy-init container if necessary) using go-client structs. Those structs are transformed into JSON-patch format which is returned to Kubernetes, which will do the actual injection of the container into the pod.
The functionality and contract for `pkg/inject/inject.go` will remain the same but with a different mechanism underneath. The JSON-patch will be generated through the new `data-plane` chart which itself depends on the `_proxy.yaml` template under the `charts/partials` chart. `_proxy.yaml` will be the sole place containing the structure of the proxy container and it will replace the go-client structs currently used. This partial will also be used when doing `helm install` (see below) thus avoiding having more than one place for the proxy structure source-of-truth.
Currently this only adds the `linkerd.io/inject: enabled` annotation to the pod template, and when Kuberenetes creates the pod it invokes the webhook as just explained. This remains as-is.
Currently this calls `pkg/inject/inject.go` to perform the injection, just as the webhook does. So the changes detailed above also affect this execution path, but the experience remains the same.
Currently this relies on the Helm go library and the charts under the `charts` directory to generate the control plane resources. The options passed as CLI arguments are converted into template values that are passed to the Helm template engine to be replaced in the placeholders inside those charts. Then, the generated yaml is fed into the `pkg/inject/inject.go` library to inject the proxies, just as `linkerd inject --manual` would do.
Here we will be replacing the current chart with a new one under `charts/control-plane` to create the control plane resources, and that chart depends on various templates under the `charts/partials` charts, in particular the `_proxy.yaml` template for inserting the yaml for the proxy into all the control plane resources. Note that `charts/control-plane` doesn't depend on `charts/data-plane` because the latter's sole purpose is to generate a JSON-patch, which Helm can't interpret.
The user experience for `linkerd install` remains the same as well.
New alternative `helm install` mechanism
---------------------------------------------
The mechanism is the same as `linkerd install` just explained; the main chart will be `charts/control-plane` which depends on `charts/partials` for, among other things, the proxy insertion. The main difference will be that the chart values will come from `values.yml` (or provided by the user through `--set` on Helm's CLI).
Published charts
----------------
The `control-plane` chart should be the main chart, published under `https://github.com/helm/charts/incubator/linkerd2`, copying the `partials` chart to `control-plane/charts` prior to publication. I'm not sure if there's a better way of doing this, given `partials` isn't suitable as as stand-alone public chart.
`data-plane` can remain unpublished.
Tasks
---------
- Refactor `injectPodSpec()` and `injectProxyInit()` in `pkg/inject/inject.go` that currently generates a JSON patch, but have it use the `data-plane` chart instead of the hard-coded go-client structs.
- Refactor the TLS libraries relied upon by the `proxy-injector` and `sp-validor` webhooks to have them work with RSA as well (they currently only deal with EC).
- Refactor the `proxy-injector` and `sp-validator` charts so that they generate the certs/keys with Helm's `genSelfSignedCert()`.
- Create `values.yaml` with all the default values, by hand (later, we can have this be automated based off of protobuf for the config part). The trust root for identity is expected to be provided by the user in this file.
- Have a well annotated main `values.yaml` file with the most common settings by default.
- Create a detailed `NOTES.txt` file with the instructions/warnings detailed above.
- Create a new website doc for Helm. A section should have a tutorial for generating the cert/key for identity.
- Enhance `linkerd check` with new checks that cover the options validations currently done in `linkerd install` that can't be performed with `helm install` (Maybe be leave this for later?)
To-do
------
- Validate how all this plays with Helm v3
Signed-off-by: Alejandro Pedraza <[email protected]>
|
|
||
| Helm charts can't rely on go code besides the functions provided by the go template library, the Sprig library, and a few extra functions provided by Helm itself. This implies a few compromises: | ||
|
|
||
| - We can't validate the install options provided in `values.yaml`. Instead, a new set of `linkerd check` checks could help catching invalid options, post-install. |
There was a problem hiding this comment.
I really like this idea and hadn’t thought of it myself! We could have a CLI command that consumes a values.yaml file and runs checks on it linkerd check chart or something like that. It could be run pre-install and maybe even highlight changes between versions (as I’m sure we’ll be mutating the values pretty extensively).
There was a problem hiding this comment.
Will this check run through the existing install validation code; i.e. validate the values in values.yaml as if they are CLI options (so a bit like linkerd install --values values.yaml)? I'd imagine this plus some additional validation code, since it'd it won't be a 1:1 mapping between the values and the CLI options.
There was a problem hiding this comment.
Yeah that sounds like it; the more code we can share here, the better 👍
| ``` | ||
| charts | ||
| ├── control-plane | ||
| │ ├── charts |
There was a problem hiding this comment.
I assume we’ll want helm to manage charts and do a requirements.yaml here, right?
There was a problem hiding this comment.
I thought requirements.yaml is only needed if the dependency (in this case, charts/partials) is published?
There was a problem hiding this comment.
Helm assumes that it manages the charts directory and it is only populated on helm package. We could hack it with some symlinks, and that might be the best solution, I just wanted to call out the one true way (tm).
You can have local paths in requirements.yaml as well, fwiw. No need to publish.
There was a problem hiding this comment.
Let me know if any of this makes sense:
When pushing the control-plane chart to the helm charts repo (https://github.com/helm/charts), some CI job there will package it to publish it to https://kubernetes-charts.storage.googleapis.com.
The only charts that job will have access to are the already published ones and the ones under control-plane/charts. For that reason it seems to me having a requirements.yaml file can't work here. OTOH when I test helm package with symlinks, it produces a tgz file that when unpacked will have charts/partials populated. I guess that unpacked directory is what we'd have to push to the helm charts repo.
Also note that helm package complains if the directory name doesn't match the chart name. So I'll have to rename control-plane to linkerd2.
There was a problem hiding this comment.
Makes sense to me to package up partials as .tgz file inside control-plane/charts, since we aren't publishing partials.
So I'll have to rename control-plane to linkerd2.
Makes sense. The data-plane folder name may not even make sense; maybe just patch?
There was a problem hiding this comment.
@alpeb in the past I've written a helm package script that just copies the directory to tmp and renames it. I don't mind calling it linkerd2.
This raises a good point though, if we want this to be helm install stable/linkerd2, we won't be able to run helm package or push the compiled release anywhere. It will end up being in raw form in the stable repo (https://github.com/helm/charts/tree/master/stable/prometheus-adapter). This would require publishing both the partials and control-plane charts.
My proposal - don't worry about it, do what we need to do internally and write a publishing script for stable/linkerd2 that copies the partials into the control-plane chart on "publish".
There was a problem hiding this comment.
I've just pushed a change with the charts renaming and a new task for the chart publication.
I've left the symlinks under charts. @grampelberg Let me know if you still deem necessary to have a requirements.yml file instead.
Signed-off-by: Alejandro Pedraza <[email protected]>
|
Integration test results for f290d67: fail 😕 |
Signed-off-by: Alejandro Pedraza <[email protected]>
|
|
||
| Helm charts can't rely on go code besides the functions provided by the go template library, the Sprig library, and a few extra functions provided by Helm itself. This implies a few compromises: | ||
|
|
||
| - We can't validate the install options provided in `values.yaml`. Instead, a new set of `linkerd check` checks could help catching invalid options, post-install. |
There was a problem hiding this comment.
Will this check run through the existing install validation code; i.e. validate the values in values.yaml as if they are CLI options (so a bit like linkerd install --values values.yaml)? I'd imagine this plus some additional validation code, since it'd it won't be a 1:1 mapping between the values and the CLI options.
|
Integration test results for 3f9a1bd: fail 😕 |
Signed-off-by: Alejandro Pedraza <[email protected]>
|
Integration test results for 7a8300d: success 🎉 |
Signed-off-by: Alejandro Pedraza <[email protected]>
|
Integration test results for a17ecb9: success 🎉 |
Signed-off-by: Alejandro Pedraza <[email protected]>
Signed-off-by: Alejandro Pedraza <[email protected]>
|
Integration test results for a2b0691: success 🎉 |
|
Integration test results for 909d4e4: fail 😕 |
Signed-off-by: Alejandro Pedraza <[email protected]>
|
Integration test results for a16a32a: success 🎉 |
4 participants
Note: Helm support for Linkerd 2 is now being tracked under the project https://github.com/orgs/linkerd/projects/13
Project kick-off presentation:Helm Support.pdf
The current CLI install process uses Helm libraries under the hood, but just as a template library. There is currently a Helm chart under the
chartsdirectory that allows installing the control plane, but uninjected, and there's novalues.yamlfile provided.The intention here is to provide a new chart with an injected control plane so that users can install linkerd through a simple
helm install incubator/linkerd2command.Helm charts can't rely on go code besides the functions provided by the go template library, the Sprig library, and a few extra functions provided by Helm itself. This implies a few compromises:
linkerd installcurrently provides. Some of the basic checks can be performed and havehelm installfail using Sprig's fail. The other checks can be performed with a new commandlinkerd check --chart values.yaml.values.yaml. Instead, a new set oflinkerd checkchecks could help catching invalid options, post-install.values.yamlfile that would contain the most common settings, but heavily annotated to instruct users about alternate settings for advanced scenarios.values.yamlfiles should be available, for different configuration scenarios like HA for example.proxy-injectorandsp-validator) to use RSA. As for the trust root for identity, we decided the user should provide their own invalues.yaml. The docs should have instructions on how to generate that cert/key.These compromises entail a less straightforward experience than what
linkerd installprovides, so the Helm installation alternative should be considered an "advanced" feature.New alternative install workflow through Helm
values.yml validation
This is an optional pre-install check that validates the options in the provided
values.yamlfile. Some of these validations are also performed whenhelm installruns, but given Helm's limitations some others can only be done through go with this new check.Installing the chart
That would install linkerd using the most common settings. The
NOTES.txtfile (rendered and shown when this command completes) could provide the follow instructions/warnings:linkerd checkto verify everything is okvalues.yamlcontaining all the possible settings?New Chart Layout
This will replace the current single chart under the
chartsdirectory.Current mechanisms and changes
The user experience for the current way of doing things remains the same. There will be some changes under the hood though.
Automated proxy injection
Currently the
proxy-injectorwebhook is invoked for any pod that gets created. If the pod's namespace or the pod itself contains alinkerd.io/inject: enabledannotation then the webhook relies on the librarypkg/inject/inject.goto programatically generate a proxy container (and a proxy-init container if necessary) using go-client structs. Those structs are transformed into JSON-patch format which is returned to Kubernetes, which will do the actual injection of the container into the pod.The functionality and contract for
pkg/inject/inject.gowill remain the same but with a different mechanism underneath. The JSON-patch will be generated through the newpatchchart which itself depends on the_proxy.yamltemplate under thecharts/partialschart._proxy.yamlwill be the sole place containing the structure of the proxy container and it will replace the go-client structs currently used. This partial will also be used when doinghelm install(see below) thus avoiding having more than one place for the proxy structure source-of-truth.linkerd injectCurrently this only adds the
linkerd.io/inject: enabledannotation to the pod template, and when Kuberenetes creates the pod it invokes the webhook as just explained. This remains as-is.linkerd inject --manualCurrently this calls
pkg/inject/inject.goto perform the injection, just as the webhook does. So the changes detailed above also affect this execution path, but the experience remains the same.linkerd installCurrently this relies on the Helm go library and the charts under the
chartsdirectory to generate the control plane resources. The options passed as CLI arguments are converted into template values that are passed to the Helm template engine to be replaced in the placeholders inside those charts. Then, the generated yaml is fed into thepkg/inject/inject.golibrary to inject the proxies, just aslinkerd inject --manualwould do.Here we will be replacing the current chart with a new one under
charts/linkerd2to create the control plane resources, and that chart depends on various templates under thecharts/partialscharts, in particular the_proxy.yamltemplate for inserting the yaml for the proxy into all the control plane resources. Note thatcharts/linkerd2doesn't depend oncharts/patchbecause the latter's sole purpose is to generate a JSON-patch, which Helm can't interpret.The user experience for
linkerd installremains the same as well.New alternative
helm installmechanismThe mechanism is the same as
linkerd installjust explained; the main chart will becharts/linkerd2which depends oncharts/partialsfor, among other things, the proxy insertion. The main difference will be that the chart values will come fromvalues.yml(or provided by the user through--seton Helm's CLI).Tasks
linkerd check --chart values.yamlcommand to fully validate the options invalues.yaml. This includes ensuring the trust root for identity has been provided (Introduce New Command To Check Helm Chart Config Values File #3130).injectPodSpec()andinjectProxyInit()inpkg/inject/inject.gothat currently generates a JSON patch, but have it use thepatchchart instead of the hard-coded go-client structs (Refactor Proxy Injection To Work With 2.5 Helm Chart #3128).proxy-injectorandsp-validorwebhooks to have them work with RSA as well (they currently only deal with EC) (Add RSA support to TLS libraries #3131).proxy-injectorandsp-validatorcharts so that they generate the certs/keys with Helm'sgenSelfSignedCert()(Include Partial Templates and Defaults In Existing Helm Chart #3126)values.yamlwith all the default values, by hand (later, we can have this be automated based off of protobuf for the config part). The trust root for identity is expected to be provided by the user in this file (Include Partial Templates and Defaults In Existing Helm Chart #3126).linkerd installto work with the newlinkerd2chart (Refactor CLI Install Command To Work With 2.5 Helm Chart #3127).values.yamlfile with the most common settings by default (Include Partial Templates and Defaults In Existing Helm Chart #3126)NOTES.txtfile with the instructions/warnings detailed above (Create NOTES.txt file for the new Helm chart #3132).helm upgrade,helm deleteandhelm rollback(Test Helm Upgrade, Delete And Rollback #3129).Not necessarily for the first iteration of this project
linkerd check --charthighlight changes between versions.linkerd2chart to https://github.com/helm/chartsDefinitely for later
install-cni. Could be as a separate stage (--set stage=cni).To-do