Skip to content

benchmarking/atunnel: atunnel CPU and memory benchmark - #2761

Merged
Bowei Du (bowei) merged 4 commits into
agent-substrate:mainfrom
chuangw6:atunnel-bench
Oct 19, 2026
Merged

Bowei Du (bowei) merged 4 commits into
agent-substrate:mainfrom
chuangw6:atunnel-bench

Conversation

@chuangw6

@chuangw6 Chuang Wang (chuangw6) commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

What

A benchmark that runs atunnel in its own process and drives load through it at rising rates, reporting atunnel's CPU, memory and latency per rate.

Includes a first full run: runs/20261016-123913/results.txt, and report.md, which explains the results and two findings (one filed as #2759).

How

  • atunnel is started and hosts actors with the worker's own code: tunnel.Start, then per actor PrepareEgress, actornet.ServeSandbox and Activate. Only atelet, the actors' programs, the router and the egress gateway are faked.
  • Everything is one program, atunnel-bench, with a subcommand per process: tunnel (atunnel), worker, sink, load, pki.
  • run-docker.sh runs three containers:
    • worker (privileged): atunnel in its own cgroup with CPU and memory limits, plus the fake atelet, an ingress actor in every actor's sandbox, and an egress actor in the first;
    • sink: the fake egress gateway;
    • load: the load generator, which sends ingress and reads atunnel's stats.
  • Seven scenarios cover ingress (HTTPS and CONNECT) and egress (reused, new, and idle connections, and throughput).

For reviewers

  • .golangci.yaml: adds a depguard exception so the benchmark can import internal/ateworker.

  • internal/workerhost has a root-gated test that runs traffic through a real sandbox.

  • Tests pass

  • Appropriate changes to documentation are included in the PR

@google-cla

google-cla Bot commented Oct 16, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@chuangw6
Chuang Wang (chuangw6) force-pushed the atunnel-bench branch 2 times, most recently from 0deb23e to 95eea20 Compare October 16, 2026 18:29
@bowei

Copy link
Copy Markdown
Collaborator

Let's make all of the executables into a single atunnel-bench executable and use Cobra flags to do the various subcommands

The first pieces of a benchmark that measures atunnel's CPU and memory on
its own: the peers atunnel talks to, the plumbing every benchmark process
shares, and the atunnel-bench program, which runs each process as a Cobra
subcommand.

- pki writes throwaway certificates shaped like the cluster's: a worker
  Pod certificate and an atelet certificate carrying Pod identities on the
  same node, as atunnel's atelet client checks, plus router and gateway
  certificates. atunnel-bench pki writes them.
- fakeatelet serves the two WorkerSupport calls atunnel makes, actor
  certificate minting and egress policy, so atunnel's own broker client
  runs unchanged.
- sink is the far end: an ingress actor, standing in for an actor's
  program receiving requests, and a fake egress gateway that accepts
  CONNECT over HTTP/2 and HTTP/1.1 with the real gateway's Envoy HTTP/2
  settings, answering or discarding by destination port. atunnel-bench
  sink runs the gateway.
- procstats reads a process's CPU, memory, goroutines, and its own cgroup
  v2 accounting, and serves them on /stats.
atunnel-bench load drives seven scenarios through atunnel at rising rates
and reports atunnel's own CPU and memory per rate, read from atunnel's
/stats.

Load is open loop: work is scheduled at a fixed rate whatever earlier
requests take, and latency is measured from when each unit was due, so a
backlog shows up as latency rather than as less load. Ingress traffic is
sent directly; egress is made by an egress actor that sits inside an
actor's sandbox, where the sandbox's redirect hands its connections to
atunnel, and that the generator drives over a Unix socket.

Each rung gets a verdict that separates atunnel running out of CPU from
the generator or far end being the bottleneck, and a rung where atunnel
dies, as when it is OOM-killed, is recorded rather than ending the run.
atunnel-bench tunnel starts atunnel and hosts actors with the calls
cmd/ateworker-gvisor makes, in its order: tunnel.Start with the worker's
own flags, then per actor PrepareEgress, actornet.ServeSandbox, and
Activate. Each actor gets real network namespaces, the nftables redirect,
and a certificate minted through atunnel's atelet client; only gVisor is
missing: an ingress actor, a small server, stands in for each actor's
program. The process runs nothing but atunnel, so its CPU and memory are
atunnel's.

atunnel-bench worker plays the rest of the worker pod in a privileged
container: the fake atelet, atunnel in a cgroup of its own with CPU and
memory limits and no swap, the ingress actors placed in every sandbox and
the egress actor in the first, and a fresh atunnel when one dies, with its
OOM kills logged. It starts atunnel and the actors from its own
executable. run-docker.sh runs the worker, the fake gateway, and the
generator as three containers.

Since every process is a subcommand of the one program, atunnel's process
also holds the others' code, which it never runs: at idle, hosting 10
actors, it holds about 29 MB resident against 26 MB built on its own.

The depguard rule that keeps internal/ateworker to the worker binaries
gains an exception for this benchmark, which hosts atunnel as a worker
does.
@chuangw6

Copy link
Copy Markdown
Collaborator Author

Let's make all of the executables into a single atunnel-bench executable and use Cobra flags to do the various subcommands

SG. everything is now one atunnel-bench binary with Cobra subcommands (tunnel, worker, sink, load, pki)

A full run of every scenario with atunnel limited to 1 CPU and 1 GiB:
results.txt is the generator's output, report.md explains it for readers
new to atunnel, with two findings: each idle egress connection holds about
157 KB of buffers, and ingress over HTTPS opens a new connection to the
actor for many requests under load, since atunnel keeps only two spare
connections per actor.
@bowei
Bowei Du (bowei) added this pull request to the merge queue Oct 19, 2026
Merged via the queue into agent-substrate:main with commit 9eb8ae5 Oct 19, 2026
15 checks passed
@bowei Bowei Du (bowei) added area/benchmarking area/network kind/feature An enhancement / feature request or implementation labels Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/benchmarking area/network kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants