Skip to content

test(vertexai): keep RPC replay independent of external network - #1548

Merged
jjsasha63 merged 6 commits into
google:mainfrom
Ultroncn:fix/offline-vertex-replay
Oct 9, 2026
Merged

jjsasha63 merged 6 commits into
google:mainfrom
Ultroncn:fix/offline-vertex-replay

Conversation

@Ultroncn

@Ultroncn Ultroncn commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Linked issue

Related to #1507. Validation of its runner fix, #1547, exposed this separate VertexAI replay-harness problem

Description of change

Problem: setupReplay used Replayer.DialOptions(), including grpc.WithBlock(). Constructing the session client attempted to connect to the real Vertex AI endpoint before recorded RPCs could replay. The original regression only detected blocking construction: removing WithBlock let an external dial attempt pass the test

Solution: Replay mode supplies Replayer.Connection() through option.WithGRPCConn. Recording mode keeps its original dial options and authentication behavior. The regression now observes asynchronous dial attempts, drives the client connection, and waits for connection state changes instead of sleeping. It fails if the injected dialer is called, including when initialization itself returns immediately. The client is closed on cleanup. No dependencies, recordings or production files are changed

Behavior change

Nothing changes for applications using the current release. VertexAI replay tests initialize and replay without access to an external endpoint. This revision addresses the test weakness raised in the review at #1548 (comment)

Testing Plan

Current tip: 5c83e4e. Merged current upstream main 44d41c9c1c48ee16d953e143a288b32972d84493; the contribution remains one test file

A local Go workspace contains the root and plugin/agentanalytics modules. With Go 1.27.1, go build -mod=readonly work passes, and go test -race -mod=readonly -count=1 -shuffle=on -timeout=120s work passes across 97 packages with tests. go mod tidy -diff is empty in both modules

With the project's Go 1.26.6 and CI-pinned golangci-lint v2.3.1, both modules report 0 issues. go test -race -mod=readonly -count=1 -shuffle=on ./session/vertexai/ also passes on Go 1.26.6. The machine's Go 1.27 export format was incompatible with the pinned analyzer, so lint was run with the project's declared toolchain. golangci-lint fmt --diff and git diff --check are clean

With the replay helper change reverted and the test kept: TestSetupReplay_InitializesWithoutExternalNetwork fails with the original blocking dial path. It also fails when replay keeps its interceptors but drops WithBlock and omits WithGRPCConn, the non-blocking regression identified in review

Blocking regression:
--- FAIL: TestSetupReplay_InitializesWithoutExternalNetwork (1.00s)
    replay client must initialize without a network connection: could not establish connection to the aiplatform server: context deadline exceeded

Non-blocking regression:
--- FAIL: TestSetupReplay_InitializesWithoutExternalNetwork (0.00s)
    replay client attempted a network connection

Restored implementation, Go 1.26.6:
ok  google.golang.org/adk/v2/session/vertexai  2.406s

The old regression was independently run against the same non-blocking mutant and passed, confirming that the added assertion addresses the reviewed gap. Both mutations were removed before final validation. An independent whole-diff review found no blocking issue

Integration limits

Existing RPC recordings exercise the service flows with UPDATE_REPLAYS unset. No live cloud E2E or re-recording was performed. As noted by the reviewer, the existing emptyService(..., offline=true) cases can still start background external connections; that separate behavior is unchanged and outside this PR's scope. Python parity is not applicable to this Go-specific replay harness, which changes no production API behavior

Checklist

  • I have read CONTRIBUTING.md
  • I have performed a self-review and an independent whole-diff review
  • The regression rejects both blocking and non-blocking external-dial variants
  • New and existing local tests pass across both modules
  • Build, formatting, module tidy and CI-pinned lint pass
  • Current-tip hosted CI and maintainer re-review are complete
  • Live cloud end-to-end verification has been performed

@Ultroncn
Ultroncn force-pushed the fix/offline-vertex-replay branch from 790e688 to 8a9e71f Compare September 14, 2026 09:01

@karolpiotrowicz karolpiotrowicz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one thing: the new regression test passes against a version of setupReplay that dials the real Vertex AI endpoint, so it does not yet pin what its name and the description say it does.

Must change before merge:

  • TestSetupReplay_InitializesWithoutExternalNetwork should fail whenever replay-mode client construction attempts a connection to a non-loopback address, not only when that attempt blocks. Details are on the dialer line.

The fix itself checks out. With the PR applied, go test ./session/vertexai/ passes in 0.19s with networking disabled, where main times out with the dial parked in ClientConn.WaitForStateChange. The new test fails with only the setupReplay change reverted, the 33 recorded Test_vertexaiService cases still replay through the interceptors, and record mode builds the same options as before.

Separately, and not something this PR needs to change: the two offline: true cases in emptyService (service_test.go#L149-L150, unchanged here) build a client with only option.WithoutAuthentication(), and that client starts a background connection to aiplatform.googleapis.com on port 443. No test fails because no RPC is sent, but the package as a whole still reaches for the network.

Comment thread session/vertexai/service_test.go Resolved

@karolpiotrowicz karolpiotrowicz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this push the test fails against the variant from my earlier comment (replay interceptors kept, WithBlock dropped, no WithGRPCConn), and it fails at the dialer check itself, with the dialer called for aiplatform.googleapis.com on port 443. The exact revert of setupReplay still fails, every recorded Test_vertexaiService case still passes through the replayer, and go test ./session/vertexai/ passes with networking disabled in 0.2s where main hangs.

Two narrower gaps remain in what the test can see. Both are non-blocking and share one cause: the test treats any call to its own dialer as the failure, rather than looking at which address was dialed. Details are on the two lines:

  • A connection that setupReplay builds itself and hands over through option.WithGRPCConn never goes through the test's dialer, so the test passes whenever that connection reaches Ready, whatever its target.
  • The dialer rejects every address, so a setupReplay that only ever dials loopback fails the test.

The property both point at is that the test fails on a dial to a non-loopback address and only on that, whichever path opens the connection. A follow-up for that is fine if you would rather not extend this PR.

Comment thread session/vertexai/service_test.go Resolved
Comment thread session/vertexai/service_test.go Resolved
@jjsasha63
jjsasha63 merged commit 6c9a898 into google:main Oct 9, 2026
14 checks passed
@Ultroncn

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review and for merging the replay fix. I have noted the two non-blocking test limitations you identified: connections supplied through WithGRPCConn can bypass the test dialer, and loopback-only connections should not be treated as external-network regressions. Those remain follow-up considerations; I am not treating this merge as coverage of those cases.

@Ultroncn
Ultroncn deleted the fix/offline-vertex-replay branch October 28, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants