Repository navigation
Description
What happened?
Commit 5d5dd7ae (#2977) removed plaintext HTTP ingress (:80 / targetPort 8080) and its NetworkPolicy rule from atenet-router. atenet-router now only accepts SPIFFE mTLS connections on :443 (with the 3-label serving certificate SAN atenet-router.ate-system.svc).
The benchmark suite still dials http://atenet-router.ate-system.svc.cluster.local:80 without client certificates:
cmd/benchmarking/boomer-worker/main.godefaults--router-urltohttp://atenet-router.ate-system.svc.cluster.localand uses an unconfiguredhttp.Transport(glutton,durdir,spawn,agentsession,sweperf).internal/benchmarking/boomer/ateenv/lifecycle.gorejectshttps://inrouterGRPCTarget, defaults to port80, and dials withinsecure.NewCredentials().benchmarking/nighthawk-ingress(runner.py,actors.py,spec.py) warms actors and generates traffic againsthttp://atenet-router.ate-system.svc.cluster.local:80with noUpstreamTlsContext.benchmarking/locust/tests/counter_demo.pyposts tohttp://atenet-router.ate-system.svc.cluster.localwithout TLS credentials.benchmarking/automation/orchestrator.pydoes not setATE_INGRESS_ALLOWED_SPIFFE_IDSindeploy_substrate(), sohack/install-ate.sh deploy ate-systemfails or rejects the runner's SPIFFE ID (spiffe://cluster.local/ns/benchmarking/sa/benchmark-runner).
As a result, every data-plane request across these benchmarks times out dialing :80.
Expected Behavior
All benchmark clients connect to https://atenet-router.ate-system.svc (:443) over mTLS, verifying the router certificate against /run/servicedns-ca/ca.crt (atenet-router.ate-system.svc) and presenting /run/podidentity.podcert.ate.dev/credential-bundle.pem. deploy_substrate() and manual install docs configure ATE_INGRESS_ALLOWED_SPIFFE_IDS for the benchmark runner ServiceAccount.
Steps to Reproduce
- Deploy Substrate and benchmark workloads via
benchmarking/automation/orchestrator.py(orhack/install-ate.sh deploy ate-system+benchmarking/deploy_locust.sh --deploy). - Run a Boomer HTTP benchmark (e.g.
glutton) or gRPC benchmark (ate_env). - Observe 100% failure rate on data-plane calls (
GluttonPing,AteEnvExec) due to:80dial timeouts.
Sandbox Runtime
Both / Runtime Agnostic
Agent Substrate Version / Commit SHA
57443d3 (first broken at 5d5dd7a)
Kubernetes Version & Environment
GKE
Host OS & Architecture
Linux x86_64
Relevant Logs and Diagnostic Output
http GluttonPing: Post "http://atenet-router.ate-system.svc.cluster.local/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
AteEnvExec 0.020 client 4ba9e5967f7509d4000bb5ccc7406d68 StartProcess: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.xxxxx.253:80: i/o timeout"Additional Context
No response
Confirmation
- I have searched existing issues and verified that this is not a duplicate.
- I have verified that this issue occurs on the latest commit on
main.
Activity
benchmarking: dial atenet-router over SPIFFE mTLS on port 443 (agent-…
benchmarking: dial atenet-router over SPIFFE mTLS on port 443 (agent-…