Repository navigation
test(vertexai): keep RPC replay independent of external network - #1548
Conversation
790e688 to
8a9e71f
Compare
karolpiotrowicz
left a comment
There was a problem hiding this comment.
Requesting changes for one thing: the new regression test passes against a version of setupReplay that dials the real Vertex AI endpoint, so it does not yet pin what its name and the description say it does.
Must change before merge:
TestSetupReplay_InitializesWithoutExternalNetworkshould fail whenever replay-mode client construction attempts a connection to a non-loopback address, not only when that attempt blocks. Details are on the dialer line.
The fix itself checks out. With the PR applied, go test ./session/vertexai/ passes in 0.19s with networking disabled, where main times out with the dial parked in ClientConn.WaitForStateChange. The new test fails with only the setupReplay change reverted, the 33 recorded Test_vertexaiService cases still replay through the interceptors, and record mode builds the same options as before.
Separately, and not something this PR needs to change: the two offline: true cases in emptyService (service_test.go#L149-L150, unchanged here) build a client with only option.WithoutAuthentication(), and that client starts a background connection to aiplatform.googleapis.com on port 443. No test fails because no RPC is sent, but the package as a whole still reaches for the network.
karolpiotrowicz
left a comment
There was a problem hiding this comment.
With this push the test fails against the variant from my earlier comment (replay interceptors kept, WithBlock dropped, no WithGRPCConn), and it fails at the dialer check itself, with the dialer called for aiplatform.googleapis.com on port 443. The exact revert of setupReplay still fails, every recorded Test_vertexaiService case still passes through the replayer, and go test ./session/vertexai/ passes with networking disabled in 0.2s where main hangs.
Two narrower gaps remain in what the test can see. Both are non-blocking and share one cause: the test treats any call to its own dialer as the failure, rather than looking at which address was dialed. Details are on the two lines:
- A connection that
setupReplaybuilds itself and hands over throughoption.WithGRPCConnnever goes through the test's dialer, so the test passes whenever that connection reachesReady, whatever its target. - The dialer rejects every address, so a
setupReplaythat only ever dials loopback fails the test.
The property both point at is that the test fails on a dial to a non-loopback address and only on that, whichever path opens the connection. A follow-up for that is fine if you would rather not extend this PR.
|
Thanks for the thorough review and for merging the replay fix. I have noted the two non-blocking test limitations you identified: connections supplied through |
Linked issue
Related to #1507. Validation of its runner fix, #1547, exposed this separate VertexAI replay-harness problem
Description of change
Problem:
setupReplayusedReplayer.DialOptions(), includinggrpc.WithBlock(). Constructing the session client attempted to connect to the real Vertex AI endpoint before recorded RPCs could replay. The original regression only detected blocking construction: removingWithBlocklet an external dial attempt pass the testSolution: Replay mode supplies
Replayer.Connection()throughoption.WithGRPCConn. Recording mode keeps its original dial options and authentication behavior. The regression now observes asynchronous dial attempts, drives the client connection, and waits for connection state changes instead of sleeping. It fails if the injected dialer is called, including when initialization itself returns immediately. The client is closed on cleanup. No dependencies, recordings or production files are changedBehavior change
Nothing changes for applications using the current release. VertexAI replay tests initialize and replay without access to an external endpoint. This revision addresses the test weakness raised in the review at #1548 (comment)
Testing Plan
Current tip: 5c83e4e. Merged current upstream main
44d41c9c1c48ee16d953e143a288b32972d84493; the contribution remains one test fileA local Go workspace contains the root and
plugin/agentanalyticsmodules. With Go 1.27.1,go build -mod=readonly workpasses, andgo test -race -mod=readonly -count=1 -shuffle=on -timeout=120s workpasses across 97 packages with tests.go mod tidy -diffis empty in both modulesWith the project's Go 1.26.6 and CI-pinned golangci-lint v2.3.1, both modules report 0 issues.
go test -race -mod=readonly -count=1 -shuffle=on ./session/vertexai/also passes on Go 1.26.6. The machine's Go 1.27 export format was incompatible with the pinned analyzer, so lint was run with the project's declared toolchain.golangci-lint fmt --diffandgit diff --checkare cleanWith the replay helper change reverted and the test kept:
TestSetupReplay_InitializesWithoutExternalNetworkfails with the original blocking dial path. It also fails when replay keeps its interceptors but dropsWithBlockand omitsWithGRPCConn, the non-blocking regression identified in reviewThe old regression was independently run against the same non-blocking mutant and passed, confirming that the added assertion addresses the reviewed gap. Both mutations were removed before final validation. An independent whole-diff review found no blocking issue
Integration limits
Existing RPC recordings exercise the service flows with
UPDATE_REPLAYSunset. No live cloud E2E or re-recording was performed. As noted by the reviewer, the existingemptyService(..., offline=true)cases can still start background external connections; that separate behavior is unchanged and outside this PR's scope. Python parity is not applicable to this Go-specific replay harness, which changes no production API behaviorChecklist