router authn/z part 3: Convert all e2e tests so that they pass for static-mtls ingress auth mode - #2390

…ithout the router's alert (agent-substrate#2961)

`TestIngressAuthentication` fails about one e2e job in three since
static-mtls became the e2e default (5 of the 15 pr-workflow runs since
agent-substrate#2390 merged): the refused client's request ends with `http2: client
conn could not be established` instead of the router's TLS alert, and
the test calls the refusal unproven. The router closes the connection
right after the alert, and the port-forward (`portforward.go:549 ...
write: broken pipe`) or Go's HTTP/2 client reports that close first.

Recent instances, all on the `https` listener:
-
https://github.com/agent-substrate/substrate/actions/runs/39402878520/job/118561649470
(untrusted CA)
-
https://github.com/agent-substrate/substrate/actions/runs/39408627974/job/118580615105
(unlisted SPIFFE ID)
-
https://github.com/agent-substrate/substrate/actions/runs/39410976322/job/118588372580
(no client certificate)
-
https://github.com/agent-substrate/substrate/actions/runs/39411585756/job/118590529170
(no client certificate)
-
https://github.com/agent-substrate/substrate/actions/runs/39413905393/job/118598129904
(unlisted SPIFFE ID)

**Change.** A refusal that fails without a TLS alert is retried on a
fresh port-forward, three times at most. An alert is still required, and
a client the router lets through still fails the test.

**Tests.** `go vet` and the suite compile; the retry only shows under
the flake, which CI's e2e jobs exercise.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR (none
needed)

… data plane (agent-substrate#2686)

Adds what it takes to measure ate-api-server's and Postgres's cold
resume capacity on the fake data plane from the benchmark automation,
plus the first tests that use it.

- **boomer: record server-side latency.** `resumepaused` records
`locust_server_duration_milliseconds` from the `x-server-elapsed-us`
trailer that `ateinterceptors.ServerUnaryInterceptor` sets, next to the
client latency. The gap between the two is network and client time.
- **Scrape boomer in automation runner Jobs.** The runner container
declares port 8001 and `monitoring.yaml`'s `boomer-worker` job keeps
`substrate-benchmark-runner` pods, so automated runs get the same
latency histograms as manual ones.
- **Shape the control plane for capacity runs.** A locust test can carry
a `controlPlane:` block (`benchmarking/automation/controlplane.py`):
- `apiServer:` sets ate-api-server's replicas, CPU and memory (requests
= limits) and its Postgres pool size.
- `postgres:` loads `pg_stat_statements` with a `postgres_exporter`
sidecar that Prometheus scrapes, and passes extra Postgres settings.

Placement is left to the install: capacity tests install with
`--cordon-control-plane`, which keeps ate-api-server and Postgres on
their own tainted pools, away from the load. The block patches the
installed manifests after the deploy; a TODO in `controlplane.py`
records the drift guards that shim still lacks.
- **Summarize ladder runs per step, on the runner.** For a ladder run,
`benchmarking/locust/ladder_steps.py` reads each step's boundaries from
the runner's log, queries the in-cluster Prometheus over the step's
steady window, and appends a `ladder_step_NNN` row per step to
`stats.jsonl`: throughput, client and server latency, CPU and memory of
ate-api-server, Postgres and fake-atelet, and Postgres connections,
commits and waits. It is additive, like the server telemetry harvest it
follows: a failure is logged and the run's other results are kept.
- **Cold resume tests.** `tests.yaml` adds two:
- `ateapi_cold_resume_pause_fake_2cpu`: a 100–1500/s cold resume ladder
against one 2 CPU replica.
- `ateapi_cold_resume_pause_fake_smoke`: the same setup at 50 actors and
20 → 50/s, a quick end-to-end check.

Each actor books 10m CPU, since a fake Worker reports its node's
allocatable as its capacity and placement must never bind. The
benchmarking README explains the per-step rows.

**Running them.** The target cluster needs the `ate-control-plane` and
`ate-postgres` node pools `--cordon-control-plane` expects, and a node
with room for the runner. On current `main` the target cluster's env
also needs `SNAPSHOT_LOCATION` (agent-substrate#2544) and
`ATE_INGRESS_ALLOWED_SPIFFE_IDS` (agent-substrate#2390), which every install now
requires.

Addresses agent-substrate#2261

- [x] Tests pass (`go test -race ./internal/benchmarking/...`, the
`benchmarking/automation` and `benchmarking/locust` Python tests,
`hack/verify-all.sh`)
- [x] Appropriate changes to documentation are included in the PR

…gent-substrate#2976)

PR agent-substrate#2390 removed the hardcoded `--ingress-allowed-spiffe-ids` flag from
`manifests/ate-install/atenet-router.yaml` and made
`ingressAllowedSPIFFEIDs` (`ATE_INGRESS_ALLOWED_SPIFFE_IDS` /
`--ingress-allowed-spiffe-ids`) required in `ate-setup deploy
ate-system` when `--atenet-dataplane=envoy`.

`deploy_substrate()` in `benchmarking/automation/orchestrator.py` does
not set `ATE_INGRESS_ALLOWED_SPIFFE_IDS`, so automated benchmark runs
fail at `deploy_ate_system`. Set `ATE_INGRESS_ALLOWED_SPIFFE_IDS` to
`BENCHMARK_RUNNER_PRINCIPAL` alongside `ATE_API_AUTHZ_BOOTSTRAP_OWNERS`,
and update the benchmark install docs.

Fixes agent-substrate#2975 

- [x] Tests pass
- [x] Docs updated

TestActorArbitraryPortAccess has been flaking in e2e since the router
moved to static-mtls in agent-substrate#2390, e.g.
https://github.com/agent-substrate/substrate/actions/runs/39549120410/job/119054007125

client-go closes the whole port-forward when any one connection on it
errors:

https://github.com/kubernetes/client-go/blob/v0.37.0/tools/portforward/portforward.go#L549-L550

RouterClient shared one CONNECT port-forward for the life of the client.
So one reset from the router failed the tunnel in flight and every
Connect after it, which then got "connection refused" from the dead
local port. agent-substrate#2961 fixed the same thing in ingressauth by forwarding per
attempt.


<!-- The PR title becomes the release note. Write it for users. -->

Why is this needed? (Or link the issue. Open one first for features,
API, install or CI changes.)

Fixes a trivial test flake bug, test only change. Spotted while
reviewing PRs like
agent-substrate#2502

<!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author
trailers. -->
AI use: AI-assisted, I reviewed it.

<!-- Breaking change? Describe what users must do on upgrade and ask for
the `breaking-change` label. -->

- [x] Tests pass
- [x] Docs updated

… next steps

From Substrate v0.6.0 the default Envoy ingress router admits only clients
that present an allowlisted pod certificate, and ate-setup refuses to
deploy it without ATE_INGRESS_ALLOWED_SPIFFE_IDS
(agent-substrate/substrate#2390). A v0.6.0 install stopped at "Turn on
Substrate" with "at least one of ATE_INGRESS_ALLOWED_SPIFFE_IDS ... must
be set".

The control-plane deploy now passes the counter demos' driver identities
(agent-substrate/substrate#3178), spiffe://cluster.local/ns/ate-demo-counter/sa/default
and spiffe://cluster.local/ns/ate-demo-counter-microvm/sa/default, with
any IDs the caller exported in ATE_INGRESS_ALLOWED_SPIFFE_IDS appended
and duplicates dropped. ate-setup reads the variable as one value, so the
installer joins the lists. Both demo IDs are listed because the sandbox
runtime and demo are chosen after the control plane is deployed. The
pod identity signer fixes the trust domain, so the IDs are known in
advance. The control-plane screen shows the allowlist, and the upgrade
hand-over exports it. Older trees and the agentgateway router ignore it.

The Envoy router also serves only TLS, so the next steps' port-forward
to port 80 and plain curl no longer reach an actor. Next steps now follow
the router:

- agentgateway: unchanged (port-forward, kubectl ate, curl).
- Envoy, tree with demos/counter/driver: kubectl logs of the driver,
  which calls the actor through the router.
- Envoy, no driver (v0.6.0): install kubectl ate and create the actor,
  then a note that callers need an allowlisted pod certificate, linking
  upstream's router README at the install's commit.

The README has a section on the allowlist: the ID format, which
workloads need an entry, adding your own clients before or after the
install, and the agentgateway alternative.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

… next steps

From Substrate v0.6.0 the default Envoy ingress router admits only clients
that present an allowlisted pod certificate, and ate-setup refuses to
deploy it without ATE_INGRESS_ALLOWED_SPIFFE_IDS
(agent-substrate/substrate#2390). A v0.6.0 install stopped at "Turn on
Substrate" with "at least one of ATE_INGRESS_ALLOWED_SPIFFE_IDS ... must
be set".

The control-plane deploy now passes the counter demos' driver identities
(agent-substrate/substrate#3178), spiffe://cluster.local/ns/ate-demo-counter/sa/default
and spiffe://cluster.local/ns/ate-demo-counter-microvm/sa/default, with
any IDs the caller exported in ATE_INGRESS_ALLOWED_SPIFFE_IDS appended
and duplicates dropped. ate-setup reads the variable as one value, so the
installer joins the lists. Both demo IDs are listed because the sandbox
runtime and demo are chosen after the control plane is deployed. The
pod identity signer fixes the trust domain, so the IDs are known in
advance. The control-plane screen shows the allowlist, and the upgrade
hand-over exports it. Older trees and the agentgateway router ignore it.

The Envoy router also serves only TLS, so the next steps' port-forward
to port 80 and plain curl no longer reach an actor. Next steps now follow
the router:

- agentgateway: unchanged (port-forward, kubectl ate, curl).
- Envoy, tree with demos/counter/driver: kubectl logs of the driver,
  which calls the actor through the router.
- Envoy, no driver (v0.6.0): install kubectl ate and create the actor,
  then a note that callers need an allowlisted pod certificate, linking
  upstream's router README at the install's commit.

The README has a section on the allowlist: the ID format, which
workloads need an entry, adding your own clients before or after the
install, and the agentgateway alternative.