Repository navigation
Add --fake-data-plane to install a fake data plane for control-plane benchmarks - #2469
Conversation
8d9b682 to
6f3679b
Compare
|
LGTM, but looks like there are merge conflicts |
6f3679b to
df2aaae
Compare
Rebased and verified locally on my own GKE cluster. PTAL. |
df2aaae to
0263d1a
Compare
The fake data plane, fake-atelet and fake-workersync, lets a benchmark drive ate-api-server and Postgres past what real nodes can restore. ate-setup --fake-data-plane installs it in place of the real one, so nothing has to be grafted onto a live install. A new fake-data-plane kustomize component, layered over the atelet and ate-controller applies the way cordon-control-plane is: - runs fake-atelet in atelet's own DaemonSet, so it keeps atelet's name, service account, and node selector; fake-atelet writes snapshot placeholders in place of the objectstore-plugin sidecar, so it takes that sidecar's env, the storage settings each overlay gives it, and the patch drops atelet's args, host mounts, sidecar, and scrape annotations; - scales ate-controller to zero, whose worker syncer would delete the fake Workers and whose WorkerPool controller would make real worker pods; its NetworkPolicy and egress MITM trust controllers stop too, which fake actors do not need; - adds fake-workersync in ate-system, one at a time through the Recreate strategy, pinned with the control plane under --cordon-control-plane. The installer only chooses the data plane; how slowly fake-atelet answers is a benchmark parameter. fake-atelet reads each of its tuning flags from a FAKE_ATELET_<FLAG> variable, a flag on the command line winning, and the component loads those variables from an optional fake-atelet-tuning ConfigMap that no install creates or resets. Only deploy ate-system switches a cluster between the real and fake data planes: - deploy atelet and deploy ate-controller refuse a data plane other than the installed one, since each alone would switch only half of it; - --fake-data-plane refuses a cluster with live worker pods, which would outlive ate-controller beside a fake atelet; - a real install stops fake-workersync, and waits for its pod to go, before applying the bundle, so it never runs beside ate-controller, and deletes the rest of its manifest every time. delete ate-system always deletes fake-workersync. Both fakes join images.Components, so a pre-built install can use them. fake-workersync places Workers on the nodes that run a fake-atelet pod rather than on a node label, which would also match the tainted control plane nodes under --cordon-control-plane. A node keeps its Workers for --node-grace after its fake-atelet pod goes, so a DaemonSet rollout does not move them. fake-workersync no longer deletes its Workers on shutdown; it adopts them when it starts, retrying while ate-api-server does not serve yet, and reports ready only once it has, so the install's rollout wait covers it. Its relay identity moves to ate-system with it. kube.NewFromClients builds a Client on existing clients, so the step tests can exercise the removal through fake typed and dynamic clients; client-go's dynamic fake is vendored for them.
Isolate mode takes the nodes' restore capacity out of a benchmark, so it can drive ate-api-server and Postgres past what real nodes restore. An isolate block on a tests.yaml entry installs substrate with --fake-data-plane; a bare `isolate:` selects the mode with the defaults. The block is validated with the other fields, before any cluster work, and the teardown needs no change, since --delete-ate-system removes the fake data plane either way. The entry's workerCount and workerMemory size the fake fleet as they size a real one. The benchmark, not the installer, tunes fake-atelet. workloads/deploy.sh --fake-atelet-env NAME=VALUE, repeatable, writes the fake-atelet-tuning ConfigMap to exactly the given FAKE_ATELET_<FLAG> entries, and restarts fake-atelet when that changes it, before the WorkerPool is applied. The isolate block's keys become those entries: delay, the per-call delayRun, delayRestore, delayCheckpoint, delayPromoteSnapshot, and delayTerminate, and mintActorCertificate. fake-atelet ignores a variable it does not know, so a fake-atelet test requires the keys to set exactly its tuning variables, and each of those to name one of its flags. deploy.sh deploys the same WorkerPool on either install. On the fake data plane no worker Deployment is made for it, so deploy.sh waits instead until the pool's readyReplicas reaches --worker-count, which fake-workersync counts only once a fake Worker's capacity is reported, so the templates' golden actors find capacity. No actor exists in isolate mode, so the mode suits only a user class that sends the actors no requests. The benchmarking README describes the mode.
…k policies The fake data plane predated two control-plane changes and could not come up beside them. atelet's probes moved to a named health port on :9091. fake-atelet keeps atelet's probes but replaces its ports, so the health port went missing and the probes could not resolve. The replacement now declares it, and fake-atelet and fake-workersync serve their health endpoints on :9091 like the real binaries, fake-workersync through a named health port of its own. ate-system now denies ingress by default. fake-workersync calls ate-api-server in ate-controller's place and relays capacity reports to each fake-atelet, and neither flow was admitted. The fake-data-plane component adds fake-workersync to the ate-api-server policy's clients and ships a fake-atelet-relay policy admitting it to the relay port. The relay policy lives in fake-workersync's manifest, so switching back to the real data plane deletes it with the rest. The system NetworkPolicy tests now render every install with and without the fake data plane, and require both of fake-workersync's flows to be admitted and the relay port to admit nothing else. The fake-data-plane render test requires every named port fake-atelet's probes target to be declared.
0263d1a to
8f51f6b
Compare
| exit 1 | ||
| fi | ||
|
|
||
| for entry in "${FAKE_ATELET_ENV[@]}"; do |
There was a problem hiding this comment.
🤖 should-fix 🟡 – with nounset, expanding an empty array like "${FAKE_ATELET_ENV[@]}" is an unbound variable error on bash before 4.4, which includes the stock macOS bash. This loop runs on every invocation, so deploy.sh would fail there even without --fake-atelet-env. The two array expansions in tune_fake_atelet have the same problem. We use ${arr[@]+"${arr[@]}"} for this in hack/run-e2e.sh and hack/install-ate.sh.
There was a problem hiding this comment.
Thanks for flagging this. I will fix this in a follow-up PR.
| if err != nil && !apierrors.IsNotFound(err) { | ||
| return fmt.Errorf("while deleting deployment %s/%s: %w", namespace, name, err) | ||
| } | ||
| err = poll(ctx, timeout, func(ctx context.Context) (bool, error) { |
There was a problem hiding this comment.
🤖 nit 🟢 – this poll duplicates WaitDeleted in wait.go. Can we do the foreground delete and then call that instead?
There was a problem hiding this comment.
Thanks for flagging this. I will fix this in a follow-up PR.
| // requireNoWorkerPods refuses to install the fake data plane while worker | ||
| // pods exist, terminating ones included. Their WorkerPools own them, so | ||
| // stopping ate-controller would leave them running, or leave the Workers of | ||
| // those terminating unregistered, taking actors fake-workersync knows nothing | ||
| // of, beside a fake atelet. A pod that has finished, as an evicted one has, | ||
| // holds no Worker, so a pool scaled to zero does not stand in the way. |
There was a problem hiding this comment.
🤖 nit 🟢 – these doc comments (here and on removeFakeWorkersync) are a lot to read for what the code does. Can we trim them to the one line of why, e.g. that worker pods would keep running beside a fake atelet? The comments in the component's kustomization.yaml could use the same pass.
There was a problem hiding this comment.
Thanks for flagging this. I will fix this in a follow-up PR.
Benjamin Elder (BenTheElder)
left a comment
There was a problem hiding this comment.
would like that shell bug fixed because it regresses stock macOS support but in the fullness of time that's not critical enough to block the PR
ate-setup --fake-data-planeinstalls the fake data plane from #2300 in place of the real one, so a benchmark's isolate mode can drive ate-api-server and Postgres past what real nodes can restore.This replaces #2366, which grafted the fakes onto a live install from
deploy.sh. Following the review there, the installer installs the fakes directly and never installs the real atelet. The installer only chooses the data plane; the benchmark tunes fake-atelet.Usage
In
tests.yaml,isolate: {delay: 500ms}does both.What's in this PR
--fake-data-plane.fake-data-planekustomize component:fake-atelet-tuningConfigMap that no install creates or resets.deploy ate-systemswitches between the real and fake data planes:deploy.sh --fake-atelet-envwrites the tuning ConfigMap.tests.yamlisolateblock selects the mode, and its keys become that tuning.deploy.shwaits on the pool'sreadyReplicasinstead of a worker Deployment.Things to look at
images.Components, so release builds publish them.ate_api.py.Testing
isolatemapping.Addresses #2261