Repository navigation
Add the option to install a large-cluster manifest and cordon control plane to dedicated machines - #1632
Conversation
| echo " --podcert-workers-per-signer N Concurrent workers per podcertificate-controller signer (default: 1)" | ||
| echo " --cluster-size size0|size10 Cluster size profile (default: size0). \"size10\" assumes a dedicated postgres node" | ||
| echo " --cordon-control-plane Pin each control plane pod to its own node: assumes a pool labeled and tainted" | ||
| echo " ate.dev/workloadType=ate-control-plane:NoSchedule with one node per pod (7 at the" |
There was a problem hiding this comment.
nit: we need to get this into the docs somewhere, I nearly had https://github.com/agent-substrate/substrate/pull/1651/changes tolerate all taints for atelet.
There was a problem hiding this comment.
added to achitecture.md
| echo " --delete-all Delete core system and all registered demos" | ||
| echo " --atenet-router=envoy|agentgateway Select the ingress and egress dataplane (default: envoy)" | ||
| echo " --podcert-workers-per-signer N Concurrent workers per podcertificate-controller signer (default: 1)" | ||
| echo " --cluster-size size0|size10 Cluster size profile (default: size0). \"size10\" assumes a dedicated postgres node" |
There was a problem hiding this comment.
what do these names mean? this doesn't seem intuitive?
There was a problem hiding this comment.
I was discouraged from "small" and "large", so "size0" is default (smallest) size. "size10" is larger. There is room for "size5", etc...
There was a problem hiding this comment.
I see. This feels unintuitive, and just "small" / "large" by another name ...
Maybe we should just require you to pass custom resizing patches ...?
Can defer to a follow-up to unblock work
There was a problem hiding this comment.
Yeah, it's essentially "small" and "large", but with the ability for more gradations later if necessary.
It looks like most of it could be a kustomize file, with the Postgres connection string being the one exception.
One good thing about "sizing"-type flags is that they should make it transparent when/if manifests change later.
Ultimately I'm guessing we'll need to have some generic way of customizing the footprint during install for production use cases?
be918e4 to
8518096
Compare
| requests: | ||
| cpu: "2" | ||
| cpu: "4" | ||
| memory: "1Gi" |
There was a problem hiding this comment.
is 1Gi request / 2GI limit memory reasonable with 4 CPU?
There was a problem hiding this comment.
Ooops, leftover from experimentation. Reverted this file.
Bowei Du (bowei)
added
kind/feature
30d11b5 to
429e41b
Compare
|
Bowei Du (@bowei) re-worked it to work with the golang-only install process |
Michelle Au (msau42)
left a comment
Michelle Au (msau42)
left a comment
There was a problem hiding this comment.
This is ok from a testing/dev perspective, but I do anticipate we'll need something more customizable for production.
The bundled postgres ships sized for small clusters (--cluster-size=size0, the default): max_connections=100 and modest CPU and memory requests. Larger clusters saturate that and cap ate-api-server well below the load they generate. postgres-size10/postgres-config-patch.yaml carries a postgresql.conf tuned for a dedicated node, merged over the postgres-config ConfigMap when --cluster-size=size10 is selected. The benchmark orchestrator selects it for the 10k tests.
The podcert-size10 kustomize overlay raises the podcertificate-controller's client-go rate limits to --kube-api-qps=100 / --kube-api-burst=200, so signing keeps up with the request volume the size10 postgres profile enables. It is installed on --cluster-size=size10 clusters.
Under --cordon-control-plane, every control plane workload (ate-api-server, ate-controller, atenet-router, atenet-egress, podcertificate-controller, postgres) gets a nodeSelector and toleration for ate.dev/workloadType=ate-control-plane and a hostname anti-affinity against the other control plane pods, so each runs alone on a node of a dedicated pool. Without the flag the manifests are applied unchanged. The pinning lives in one kustomize component with a name-regex target. The install applies these workloads through several different streams (the base bundle, a lone ate-api-server redeploy, the podcert overlay, the postgres file, and the egress variants), so the installer composes whichever path is in use with the component; kustomize leaves a stream alone when nothing in it matches.
--podcert-workers-per-signer was applied as a Get/Update on the live Deployment after the manifest's server-side apply. That rolled the controller out twice per install, first with the manifest's WORKERS_PER_SIGNER=1 and then with the requested value, and let the rollout wait watch a pod that was about to be replaced. Because the apply is forced, every later install reset the value to 1 before the Update raised it again, so the double rollout repeated even when nothing changed. The value is now set on the rendered objects before the one apply, as the size10 postgres resize already is: one rollout, and the rollout wait sees the pod that will serve.
f9fe07e to
d93dbaa
Compare
|
100% WRT needing something more customizable for production. |
This change allows users to install Substrate on larger clusters. It adds a
--cluster-sizeflag to enable more t-shirt-style sizing in the future to accommodate clusters of different sizes.It also adds a --cordon-control-plane flag that allows taints/tolerances/antiaffinity/node labels to have each control plane element run on its own dedicated machine.
Fixes #<issue_number_goes_here>