Skip to content

Add --fake-data-plane to install a fake data plane for control-plane benchmarks - #2469

Merged
Benjamin Elder (BenTheElder) merged 3 commits into
agent-substrate:mainfrom
haiyanmeng:fake-data-plane-install
Oct 14, 2026
Merged

Benjamin Elder (BenTheElder) merged 3 commits into
agent-substrate:mainfrom
haiyanmeng:fake-data-plane-install

Conversation

@haiyanmeng

@haiyanmeng haiyanmeng commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

ate-setup --fake-data-plane installs the fake data plane from #2300 in place of the real one, so a benchmark's isolate mode can drive ate-api-server and Postgres past what real nodes can restore.

This replaces #2366, which grafted the fakes onto a live install from deploy.sh. Following the review there, the installer installs the fakes directly and never installs the real atelet. The installer only chooses the data plane; the benchmark tunes fake-atelet.

Usage

hack/install-ate.sh --deploy-ate-system --fake-data-plane
benchmarking/workloads/deploy.sh --deploy --worker-count 200 \
  --fake-atelet-env FAKE_ATELET_DELAY=500ms

In tests.yaml, isolate: {delay: 500ms} does both.

What's in this PR

  1. ate-setup: install the fake data plane with --fake-data-plane.
    • A fake-data-plane kustomize component:
      • runs fake-atelet in atelet's own DaemonSet, keeping atelet's identity, node selector and storage settings;
      • scales ate-controller to zero;
      • adds fake-workersync.
    • fake-atelet reads its tuning flags from an optional fake-atelet-tuning ConfigMap that no install creates or resets.
    • Only deploy ate-system switches between the real and fake data planes:
      • single-component redeploys refuse a mismatch;
      • a fake install refuses live worker pods;
      • a real install removes fake-workersync before ate-controller starts.
    • fake-workersync:
      • places Workers on the fake-atelet nodes;
      • keeps them across restarts;
      • reports ready only once ate-api-server answers.
  2. benchmarking: run isolate mode on the fake data plane install.
    • deploy.sh --fake-atelet-env writes the tuning ConfigMap.
    • A tests.yaml isolate block selects the mode, and its keys become that tuning.
    • On the fake data plane, deploy.sh waits on the pool's readyReplicas instead of a worker Deployment.

Things to look at

  • Release images: fake-atelet and fake-workersync join images.Components, so release builds publish them.
  • Switching: a fake install needs a cluster with no live worker pods, so delete any WorkerPools first.
  • ate-controller at zero: its NetworkPolicy and egress MITM trust controllers stop too. Fake actors need neither.
  • User class: fake actors don't exist, so the mode suits only a user class that sends them no requests, such as ate_api.py.

Testing

  • Unit tests cover:
    • the component's render over every bundle;
    • the switching guards and the fake-workersync removal;
    • fake-atelet's tuning variables;
    • fake-workersync's node grace, retry and readiness;
    • the isolate mapping.

Addresses #2261

  • Tests pass
  • Appropriate changes to documentation are included in the PR

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxsmythe

Copy link
Copy Markdown
Collaborator

LGTM, but looks like there are merge conflicts

@haiyanmeng

Copy link
Copy Markdown
Collaborator Author

LGTM, but looks like there are merge conflicts

Rebased and verified locally on my own GKE cluster. PTAL.

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxsmythe
Max Smythe (maxsmythe) added this pull request to the merge queue Oct 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 13, 2026
@HavenXia
Haven Xia (HavenXia) added this pull request to the merge queue Oct 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 13, 2026
@haiyanmeng
haiyanmeng force-pushed the fake-data-plane-install branch from df2aaae to 0263d1a Compare October 14, 2026 00:42
The fake data plane, fake-atelet and fake-workersync, lets a benchmark
drive ate-api-server and Postgres past what real nodes can restore.
ate-setup --fake-data-plane installs it in place of the real one, so
nothing has to be grafted onto a live install. A new fake-data-plane
kustomize component, layered over the atelet and ate-controller applies
the way cordon-control-plane is:

- runs fake-atelet in atelet's own DaemonSet, so it keeps atelet's
  name, service account, and node selector; fake-atelet writes snapshot
  placeholders in place of the objectstore-plugin sidecar, so it takes
  that sidecar's env, the storage settings each overlay gives it, and the
  patch drops atelet's args, host mounts, sidecar, and scrape
  annotations;
- scales ate-controller to zero, whose worker syncer would delete the
  fake Workers and whose WorkerPool controller would make real worker
  pods; its NetworkPolicy and egress MITM trust controllers stop too,
  which fake actors do not need;
- adds fake-workersync in ate-system, one at a time through the
  Recreate strategy, pinned with the control plane under
  --cordon-control-plane.

The installer only chooses the data plane; how slowly fake-atelet
answers is a benchmark parameter. fake-atelet reads each of its tuning
flags from a FAKE_ATELET_<FLAG> variable, a flag on the command line
winning, and the component loads those variables from an optional
fake-atelet-tuning ConfigMap that no install creates or resets.

Only deploy ate-system switches a cluster between the real and fake
data planes:

- deploy atelet and deploy ate-controller refuse a data plane other
  than the installed one, since each alone would switch only half of it;
- --fake-data-plane refuses a cluster with live worker pods, which
  would outlive ate-controller beside a fake atelet;
- a real install stops fake-workersync, and waits for its pod to go,
  before applying the bundle, so it never runs beside ate-controller,
  and deletes the rest of its manifest every time.

delete ate-system always deletes fake-workersync. Both fakes join
images.Components, so a pre-built install can use them.

fake-workersync places Workers on the nodes that run a fake-atelet pod
rather than on a node label, which would also match the tainted control
plane nodes under --cordon-control-plane. A node keeps its Workers for
--node-grace after its fake-atelet pod goes, so a DaemonSet rollout
does not move them. fake-workersync no longer deletes its Workers on
shutdown; it adopts them when it starts, retrying while ate-api-server
does not serve yet, and reports ready only once it has, so the
install's rollout wait covers it. Its relay identity moves to ate-system
with it.

kube.NewFromClients builds a Client on existing clients, so the step
tests can exercise the removal through fake typed and dynamic clients;
client-go's dynamic fake is vendored for them.
Isolate mode takes the nodes' restore capacity out of a benchmark, so it
can drive ate-api-server and Postgres past what real nodes restore. An
isolate block on a tests.yaml entry installs substrate with
--fake-data-plane; a bare `isolate:` selects the mode with the defaults.
The block is validated with the other fields, before any cluster work,
and the teardown needs no change, since --delete-ate-system removes the
fake data plane either way. The entry's workerCount and workerMemory
size the fake fleet as they size a real one.

The benchmark, not the installer, tunes fake-atelet.
workloads/deploy.sh --fake-atelet-env NAME=VALUE, repeatable, writes the
fake-atelet-tuning ConfigMap to exactly the given FAKE_ATELET_<FLAG>
entries, and restarts fake-atelet when that changes it, before the
WorkerPool is applied. The isolate block's keys become those entries:
delay, the per-call delayRun, delayRestore, delayCheckpoint,
delayPromoteSnapshot, and delayTerminate, and
mintActorCertificate. fake-atelet ignores a variable it does not know, so
a fake-atelet test requires the keys to set exactly its tuning variables,
and each of those to name one of its flags.

deploy.sh deploys the same WorkerPool on either install. On the fake
data plane no worker Deployment is made for it, so deploy.sh waits
instead until the pool's readyReplicas reaches --worker-count, which
fake-workersync counts only once a fake Worker's capacity is reported,
so the templates' golden actors find capacity.

No actor exists in isolate mode, so the mode suits only a user class
that sends the actors no requests. The benchmarking README describes the
mode.
…k policies

The fake data plane predated two control-plane changes and could not
come up beside them.

atelet's probes moved to a named health port on :9091. fake-atelet
keeps atelet's probes but replaces its ports, so the health port went
missing and the probes could not resolve. The replacement now declares
it, and fake-atelet and fake-workersync serve their health endpoints on
:9091 like the real binaries, fake-workersync through a named health
port of its own.

ate-system now denies ingress by default. fake-workersync calls
ate-api-server in ate-controller's place and relays capacity reports
to each fake-atelet, and neither flow was admitted. The fake-data-plane
component adds fake-workersync to the ate-api-server policy's clients
and ships a fake-atelet-relay policy admitting it to the relay port.
The relay policy lives in fake-workersync's manifest, so switching back
to the real data plane deletes it with the rest.

The system NetworkPolicy tests now render every install with and
without the fake data plane, and require both of fake-workersync's
flows to be admitted and the relay port to admit nothing else. The
fake-data-plane render test requires every named port fake-atelet's
probes target to be declared.
@haiyanmeng
haiyanmeng force-pushed the fake-data-plane-install branch from 0263d1a to 8f51f6b Compare October 14, 2026 01:05
exit 1
fi

for entry in "${FAKE_ATELET_ENV[@]}"; do

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 should-fix 🟡 – with nounset, expanding an empty array like "${FAKE_ATELET_ENV[@]}" is an unbound variable error on bash before 4.4, which includes the stock macOS bash. This loop runs on every invocation, so deploy.sh would fail there even without --fake-atelet-env. The two array expansions in tune_fake_atelet have the same problem. We use ${arr[@]+"${arr[@]}"} for this in hack/run-e2e.sh and hack/install-ate.sh.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging this. I will fix this in a follow-up PR.

if err != nil && !apierrors.IsNotFound(err) {
return fmt.Errorf("while deleting deployment %s/%s: %w", namespace, name, err)
}
err = poll(ctx, timeout, func(ctx context.Context) (bool, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 nit 🟢 – this poll duplicates WaitDeleted in wait.go. Can we do the foreground delete and then call that instead?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging this. I will fix this in a follow-up PR.

Comment on lines +383 to +388
// requireNoWorkerPods refuses to install the fake data plane while worker
// pods exist, terminating ones included. Their WorkerPools own them, so
// stopping ate-controller would leave them running, or leave the Workers of
// those terminating unregistered, taking actors fake-workersync knows nothing
// of, beside a fake atelet. A pod that has finished, as an evicted one has,
// holds no Worker, so a pool scaled to zero does not stand in the way.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 nit 🟢 – these doc comments (here and on removeFakeWorkersync) are a lot to read for what the code does. Can we trim them to the one line of why, e.g. that worker pods would keep running beside a fake atelet? The comments in the component's kustomization.yaml could use the same pass.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging this. I will fix this in a follow-up PR.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would like that shell bug fixed because it regresses stock macOS support but in the fullness of time that's not critical enough to block the PR

@BenTheElder
Benjamin Elder (BenTheElder) added this pull request to the merge queue Oct 14, 2026
Merged via the queue into agent-substrate:main with commit 6b64505 Oct 14, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants