Skip to content

Add a fake data plane for control-plane benchmarks #2261

Closed

Description

@haiyanmeng

Summary

Benchmark ate-api-server and Postgres without a real data plane by replacing atelet, and atecontroller's WorkerPool controller and worker syncer, with two small fakes under cmd/benchmarking/isolate/. The fakes speak the existing APIs and serve the real WorkerPool CR, so the production binaries gain no benchmark flags or code paths, and a benchmark sizes the fake fleet the way it sizes a real one.

Motivation

A real cold resume is limited by the node's restore capacity, which hides the limits of ate-api-server and Postgres. To find those limits, a benchmark has to take the data plane out: lifecycle calls must succeed quickly, without running any workload, and the control plane must see enough workers to place actors on.

We could add a stub AteomHerder behind an atelet flag. But it has two problems:

  1. It puts benchmark-only code in a production binary, behind a flag that breaks a cluster if set by mistake.
  2. It does not go far enough. Every Worker still needs a real, Ready worker pod to be registered, so the worker count, and with it the load the control plane can be driven to, is still bound by node capacity.

Both components sit behind well-defined APIs, so fakes can stand in for them from outside.

Goals

  • Let every RPC ate-api-server serves run through its real handlers, workflows, store and scheduler at rates the data plane cannot sustain:
    • The RPCs that reach atelet (ResumeActor, SuspendActor, PauseActor and DeleteActor) are answered by fake-atelet.
    • The ones that call object storage directly (CreateTag, DeleteTag and RevertActor) work against fake-atelet's placeholder objects.
    • The rest Control RPCs need nothing from the data plane and run unchanged.
    • WorkerService serves only atelet callers for their own node's Workers, so its RPCs are called with a fake-atelet's certificate. 
      1. SetWorkerCapacity is: fake-atelet relays fake-workersync's reports. 
      2. MintAteomActorCertificate is: fake-atelet mints on each activation, as ateom does through atelet. 
      3. RequestActorSuspend is not driven; no ateom sends it yet.
  • Register many more Workers than the cluster has nodes, each with a reported capacity, so placement is never the bottleneck unless the benchmark wants it to be.
  • Size and shape the fake fleet through the WorkerPool, as a real one is: spec.replicas, labels, sandbox class and resource limits, with the pool's status reporting the fleet's readiness.
  • Keep each fake call's latency configurable, so actor leases are held about as long as they would be in production.
  • No changes to cmd/atelet, cmd/atecontroller or cmd/ateapi.

Non-goals

  • Exercising atelet, ateom, real snapshots or the router. Actors "run" by the fakes do not exist and cannot be pinged. Object storage is touched only by ate-api-server's own snapshot bookkeeping and the fake's placeholder objects .
  • Running on a cluster that serves real actors. The fakes replace cluster-wide components.
  • Simulating failures. Every fake call succeeds after its delay; fault injection can be added later behind the same seams.
  • Reproducing ate-api-server's connection fan-out to atelets.

Activity

  1. added this to the M3 milestone on Oct 6, 2026
  2. self-assigned this
    on Oct 6, 2026
  3. haiyanmeng commented on Oct 6, 2026

    @haiyanmeng
    CollaboratorAuthor

    Julian Gutierrez Oschmann (@juli4n) Max Smythe (@maxsmythe) Benjamin Elder (@BenTheElder) , I created this issue and added it into the M3 milestone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions