Add agentgateway to CI & rename dataplane flag by keithmattix · Pull Request #1598 · agent-substrate/substrate

Keith Mattix II (keithmattix) · 2026-09-10T21:37:25Z

Run agentgateway data plane tests as a part of substrate CI (non-blocking to start so we can confirm it's not flaky). Also, change the --atenet-router flag to --atenet-dataplane to make it clearer that the flag controls ingress and egress.

I've run the e2es locally across gVisor and microVM plus the MITM variants for both. The only skip we do for agentgateway is TestIngressProtocolDowngrade because 1. the behavior its testing only exists on the non-CONNECT atunnel ingress path and agentgateway only sends CONNECT to atunnel and 2. I'm not sure that we want this to be a part of the contract that substrate is bound by (e.g. do we really want to commit to atunnel always parsing HTTP?).

My goal with getting both dataplanes into CI is to start taking steps to codify the proxy (router + egress PEP) contract for substrate. The telemetry they emit, atunnel expectations, etc. are all important contracts to explicitly call out so that they don't become too coupled to a single dataplane implementation.

It's a good idea to open an issue first for discussion.

Tests pass
Appropriate changes to documentation are included in the PR

Keith Mattix II (keithmattix) · 2026-09-14T14:28:41Z

Current failure is a postgres timeout. I think CI just needs to be re-kicked

Eitan Yarmush (EItanya)

🤖 Relocated with AI assistance; the original inline comment wording is preserved.

Moved from my review on #1628, which is stacked on this PR. The three inline comments and the ingress-contract portion of that review belong here:

I have some clarifying questions about what the actual ingress contract is since there are definitely dataplane specific checks in the tests

The benchmark-specific questions remain on #1628. The CI matrix suggestion is already addressed here.

Eitan Yarmush (EItanya) · 2026-09-14T15:14:06Z

-		t.Fatalf("creating statusz client: %v", err)
+	var statusz *e2e.StatuszClient
+	if os.Getenv("E2E_DATAPLANE") != "agentgateway" {
+		statusz, err = e2e.NewStatuszClient(ctx)


Moved from my comment on #1628; this feedback belongs to #1598.

Can we use a more generic solution here?

My plan is to just get both data planes to a working state in CI then iterate on a contract that both data planes implement. New contracts need would need to be persisted there instead of one off. So agree with the comment but will do it in a follow up

Eitan Yarmush (EItanya) · 2026-09-14T15:14:07Z

 	t.Fatalf("timed out waiting for the parking gauge to satisfy the condition (last active=%d)", last)
 }
+
+func agentGatewayParkingCount(ctx context.Context) (int, bool) {


Moved from my comment on #1628; this feedback belongs to #1598.

Same question about unifying the data gathering mechanism here

Keith’s reply on the original thread points to this PR’s description and its goal of defining a shared proxy contract.

Yeah +1 to my comment above; we'll define it in a shared contract

Lior Lieberman (LiorLieberman)

Thanks Keith, LGTM overall a few questions

Lior Lieberman (LiorLieberman) · 2026-09-14T23:20:40Z

+			case "envoy":
+				authority, ok := accessLogField(line.text, "authority")
+				if ok && strings.HasSuffix(authority, ":"+port) && strings.Contains(line.text, "/actor/"+actorName) {
+					t.Logf("egress gateway tunneled the request: %s", line.text)
+					return true
+				}
+			case "agentgateway":


is there a way we can wrap it somehow to avoid many switch cases

I can make an interface in the testbed to encode all the dataplane specific behavior. FWIW, my plan is to create a dataplane contract so some of these nuances/features get described in a central place, but we'll have the test wrapper to make this cleaner in the mentime

That being said it's a bigger change than the few conditionals we have today, so I'll push it in a new commit to get some feedback

Lior Lieberman (LiorLieberman) · 2026-09-14T23:20:52Z

-		gatewayContainer = "envoy"
-		// The access log's line prefix, from the HttpConnectionManager
-		// text_format_source in manifests/ate-install/atenet-egress.yaml.


where did these go to

Abstracted into accessLogField and the caller assertEgressGatewayConnect

Lior Lieberman (LiorLieberman) · 2026-09-14T23:22:47Z

-	if err != nil {
-		t.Fatalf("creating statusz client: %v", err)
+	var statusz *e2e.StatuszClient
+	if os.Getenv("E2E_DATAPLANE") != "agentgateway" {


We really need to avoid if envoy if agentgateway everywhere? is there a way we can initialize it in the suit once and avoid switch cases everywhere?

Implemented something in 25c30d6

Lior Lieberman (LiorLieberman) · 2026-09-14T23:23:05Z

 	collectorNamespace = "otel-system"
 	collectorService   = "opentelemetry-collector"
 	collectorPromPort  = 8889
+	routerStatsPort    = 15020


what is this port for?

Will rename to prefix with agentgateway. The current metrics reader in our test code relies on OTLP exported metrics while agentgateway just has native prom metrics

Signed-off-by: Keith Mattix II <[email protected]>

Lior Lieberman (LiorLieberman)

LGTM
Thanks Keith Mattix II (@keithmattix)

Keith Mattix II (keithmattix) mentioned this pull request Sep 12, 2026

benchmark: Make nighthawk more reproducible and add agentgateway #1628

Open

2 tasks done

Eitan Yarmush (EItanya) reviewed Sep 14, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) reviewed Sep 14, 2026

View reviewed changes

Keith Mattix II (keithmattix) requested a review from Lior Lieberman (LiorLieberman) September 15, 2026 01:35

Keith Mattix II (keithmattix) added 8 commits September 14, 2026 22:56

Add agentgateway to CI & rename dataplane flag

41badf8

Signed-off-by: Keith Mattix II <[email protected]>

Remove idle timeout

b657ecc

Signed-off-by: Keith Mattix II <[email protected]>

Deslop yaml list

ace1e0a

Signed-off-by: Keith Mattix II <[email protected]>

Migrate to matrix approach

5b1927b

Signed-off-by: Keith Mattix II <[email protected]>

Unblock CI

881c2b7

Signed-off-by: Keith Mattix II <[email protected]>

Patch one more dataplane contract drift

f20d90c

Signed-off-by: Keith Mattix II <[email protected]>

ADdress PR feedback

e063053

Signed-off-by: Keith Mattix II <[email protected]>

Restore AgentGateway egress policy coverage

67d6a87

Keith Mattix II (keithmattix) force-pushed the add-agentgateway-e2es-to-ci branch from 25c30d6 to 67d6a87 Compare September 15, 2026 14:13

Lior Lieberman (LiorLieberman) approved these changes Sep 15, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) merged commit abd45ad into agent-substrate:main Sep 15, 2026
11 checks passed

Conversation

Keith Mattix II (keithmattix) commented Sep 10, 2026 • edited by LiorLieberman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Keith Mattix II (keithmattix) commented Sep 14, 2026

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Keith Mattix II (keithmattix) commented Sep 10, 2026 •

edited by LiorLieberman

Loading