Repository navigation
Description
Summary
Benchmark ate-api-server and Postgres without a real data plane by replacing atelet, and atecontroller's WorkerPool controller and worker syncer, with two small fakes under cmd/benchmarking/isolate/. The fakes speak the existing APIs and serve the real WorkerPool CR, so the production binaries gain no benchmark flags or code paths, and a benchmark sizes the fake fleet the way it sizes a real one.
Motivation
A real cold resume is limited by the node's restore capacity, which hides the limits of ate-api-server and Postgres. To find those limits, a benchmark has to take the data plane out: lifecycle calls must succeed quickly, without running any workload, and the control plane must see enough workers to place actors on.
We could add a stub AteomHerder behind an atelet flag. But it has two problems:
- It puts benchmark-only code in a production binary, behind a flag that breaks a cluster if set by mistake.
- It does not go far enough. Every Worker still needs a real, Ready worker pod to be registered, so the worker count, and with it the load the control plane can be driven to, is still bound by node capacity.
Both components sit behind well-defined APIs, so fakes can stand in for them from outside.
Goals
- Let every RPC ate-api-server serves run through its real handlers, workflows, store and scheduler at rates the data plane cannot sustain:
- The RPCs that reach atelet (
ResumeActor,SuspendActor,PauseActorandDeleteActor) are answered by fake-atelet. - The ones that call object storage directly (
CreateTag,DeleteTagandRevertActor) work against fake-atelet's placeholder objects. - The rest
ControlRPCs need nothing from the data plane and run unchanged. WorkerServiceserves only atelet callers for their own node's Workers, so its RPCs are called with a fake-atelet's certificate.SetWorkerCapacityis: fake-atelet relays fake-workersync's reports.MintAteomActorCertificateis: fake-atelet mints on each activation, as ateom does through atelet.RequestActorSuspendis not driven; no ateom sends it yet.
- The RPCs that reach atelet (
- Register many more Workers than the cluster has nodes, each with a reported capacity, so placement is never the bottleneck unless the benchmark wants it to be.
- Size and shape the fake fleet through the WorkerPool, as a real one is:
spec.replicas, labels, sandbox class and resource limits, with the pool's status reporting the fleet's readiness. - Keep each fake call's latency configurable, so actor leases are held about as long as they would be in production.
- No changes to
cmd/atelet,cmd/atecontrollerorcmd/ateapi.
Non-goals
- Exercising atelet, ateom, real snapshots or the router. Actors "run" by the fakes do not exist and cannot be pinged. Object storage is touched only by ate-api-server's own snapshot bookkeeping and the fake's placeholder objects .
- Running on a cluster that serves real actors. The fakes replace cluster-wide components.
- Simulating failures. Every fake call succeeds after its delay; fault injection can be added later behind the same seams.
- Reproducing ate-api-server's connection fan-out to atelets.
Activity
haiyanmeng commentedon Oct 6, 2026
Julian Gutierrez Oschmann (@juli4n) Max Smythe (@maxsmythe) Benjamin Elder (@BenTheElder) , I created this issue and added it into the M3 milestone.
fake-dataplane: add a fake dataplane skeleton (agent-substrate#2272)
fake-dataplane: let the fake data plane host actors (agent-substrate#…
Merge upstream main through 25a0b88 (#67)
Sync upstream sandbox, snapshot, and HTTPS egress updates (#69)
benchmarking: pace boomer to a target rate and add the resumepaused u…
Add --fake-data-plane to install a fake data plane for control-plane …
Benchmarking: measure ate-api-server cold resume capacity on the fake…