Skip to content

fix(vertexai): prevent RuntimeError from stale client after startup event loop#6072

Merged
cdoern merged 1 commit into
ogx-ai:mainfrom
goingforstudying-ctrl:fix/vertexai-client-event-loop-reset
Jun 12, 2026
Merged

fix(vertexai): prevent RuntimeError from stale client after startup event loop#6072
cdoern merged 1 commit into
ogx-ai:mainfrom
goingforstudying-ctrl:fix/vertexai-client-event-loop-reset

Conversation

@goingforstudying-ctrl

Copy link
Copy Markdown
Contributor

Ran into #6057 while setting up a VertexAI provider — the server would crash with RuntimeError: Event loop is closed on the first inference request after startup.

Turns out the issue is that during StackApp.__init__, the stack initialization runs in a temporary event loop, and refresh_registry_once() triggers model listing which calls _get_client() on the VertexAI adapter. The Google genai Client eagerly creates an httpx.AsyncClient internal to itself, binding it to that temporary loop. After the temp loop goes away and uvicorn starts on a fresh loop, the cached client is still holding connections tied to the dead loop.

Two things in this PR:

  1. Added _reset_client() on VertexAIInferenceAdapter — clears the cached default client and HTTP options. This is called from StackApp.__init__ right after reset_sqlstore_engines(), following the exact same pattern that already exists for SQL engines.

  2. Added a safety check in _get_client() itself — before returning the cached default client, it checks whether the underlying httpx transport has been closed. If it has (which happens when the event loop it was created on is terminated), it logs and recreates the client. This is defense-in-depth in case the reset isn't called.

Not entirely sure about the is_closed check — it relies on httpx's internal state tracking which seems stable across recent versions but could change. Happy to remove that part if you'd prefer to keep it simpler.

Test Plan

Ran python3.12 -m py_compile on both modified files — they compile cleanly. The existing test suite should cover the normal code paths since these changes only affect the initialization/recreation path. The event loop simulation is tricky to unit test without bringing up a full server, but the pattern mirrors the tested reset_sqlstore_engines() flow exactly.

@goingforstudying-ctrl
goingforstudying-ctrl force-pushed the fix/vertexai-client-event-loop-reset branch 10 times, most recently from 79a086d to ce6d05b Compare June 11, 2026 21:56
…r startup

During StackApp.__init__, stack.initialize() runs inside a temporary event
loop via ThreadPoolExecutor.  Model listing (refresh_registry_once) triggers
lazy VertexAI client creation which binds an internal httpx.AsyncClient to
the temporary loop.  When uvicorn later starts on a new loop, the cached
client causes 'RuntimeError: Event loop is closed' on the first inference
request.

Two fixes that work together:

1. Add _reset_client() to VertexAIInferenceAdapter — clears the cached
   default client and HTTP options after the temporary event loop exits.
   This follows the same pattern as reset_sqlstore_engines() for SQL
   engines.

2. Add defense-in-depth in _get_client() — before returning the cached
   default client, verify it is still usable by checking whether the
   underlying httpx transport has been closed.  If it has, recreate
   the client.

Fixes ogx-ai#6057

Signed-off-by: goingforstudying-ctrl <[email protected]>
@goingforstudying-ctrl
goingforstudying-ctrl force-pushed the fix/vertexai-client-event-loop-reset branch from ce6d05b to 0ec1a27 Compare June 12, 2026 15:44
@cdoern
cdoern added this pull request to the merge queue Jun 12, 2026
@cdoern

cdoern commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

@Mergifyio backport release-1.1.x

@mergify

mergify Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

backport release-1.1.x

✅ Backports have been created

Details

Merged via the queue into ogx-ai:main with commit a83f2a1 Jun 12, 2026
53 checks passed
@mattf

mattf commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

@cdoern how can we update this so the provider doesn't have impl details about how the server does startup sequencing?

@goingforstudying-ctrl
goingforstudying-ctrl deleted the fix/vertexai-client-event-loop-reset branch June 14, 2026 14:07
cdoern pushed a commit that referenced this pull request Jun 15, 2026
…vent loop (backport #6072) (#6105)

Ran into #6057 while setting up a VertexAI provider — the server would
crash with `RuntimeError: Event loop is closed` on the first inference
request after startup.

Turns out the issue is that during `StackApp.__init__`, the stack
initialization runs in a temporary event loop, and
`refresh_registry_once()` triggers model listing which calls
`_get_client()` on the VertexAI adapter. The Google genai `Client`
eagerly creates an `httpx.AsyncClient` internal to itself, binding it to
that temporary loop. After the temp loop goes away and uvicorn starts on
a fresh loop, the cached client is still holding connections tied to the
dead loop.

Two things in this PR:

1. Added `_reset_client()` on `VertexAIInferenceAdapter` — clears the
cached default client and HTTP options. This is called from
`StackApp.__init__` right after `reset_sqlstore_engines()`, following
the exact same pattern that already exists for SQL engines.

2. Added a safety check in `_get_client()` itself — before returning the
cached default client, it checks whether the underlying httpx transport
has been closed. If it has (which happens when the event loop it was
created on is terminated), it logs and recreates the client. This is
defense-in-depth in case the reset isn't called.

Not entirely sure about the `is_closed` check — it relies on httpx's
internal state tracking which seems stable across recent versions but
could change. Happy to remove that part if you'd prefer to keep it
simpler.

## Test Plan
Ran `python3.12 -m py_compile` on both modified files — they compile
cleanly. The existing test suite should cover the normal code paths
since these changes only affect the initialization/recreation path. The
event loop simulation is tricky to unit test without bringing up a full
server, but the pattern mirrors the tested `reset_sqlstore_engines()`
flow exactly.<hr>This is an automatic backport of pull request #6072
done by [Mergify](https://mergify.com).

Signed-off-by: goingforstudying-ctrl <[email protected]>
Co-authored-by: goingforstudying-ctrl <[email protected]>
Co-authored-by: goingforstudying-ctrl <[email protected]>
sahana-sreeram pushed a commit to sahana-sreeram/ogx-testing that referenced this pull request Jun 23, 2026
…x-ai#6148)

# What does this PR do?

PR ogx-ai#6072 added `_reset_client()` to clear stale VertexAI clients after
the temporary startup event loop, but the reset loop in
`StackApp.__init__` only iterated `self.stack.impls.values()` — which
are routing tables (`CommonRoutingTableImpl`), not the actual provider
adapters.

The VertexAI adapter with `_reset_client` lives inside
`routing_table.impls_by_provider_id`, so it was never reached. This
walks into `impls_by_provider_id` to reset nested providers.

Closes ogx-ai#6057

## Test Plan

1. Configure VertexAI provider with WIF credentials
2. Start OGX server
3. Send inference request to
`vertexai/publishers/google/models/gemini-2.5-flash`

**Before:** `RuntimeError: Event loop is closed` on every request
**After:** Request completes successfully

Signed-off-by: Artemy Hladenko <[email protected]>
Signed-off-by: Artemy <[email protected]>
Co-authored-by: Charlie Doern <[email protected]>
Co-authored-by: Sébastien Han <[email protected]>
cdoern added a commit that referenced this pull request Jun 24, 2026
…ckport #6148) (#6177)

# What does this PR do?

PR #6072 added `_reset_client()` to clear stale VertexAI clients after
the temporary startup event loop, but the reset loop in
`StackApp.__init__` only iterated `self.stack.impls.values()` — which
are routing tables (`CommonRoutingTableImpl`), not the actual provider
adapters.

The VertexAI adapter with `_reset_client` lives inside
`routing_table.impls_by_provider_id`, so it was never reached. This
walks into `impls_by_provider_id` to reset nested providers.

Closes #6057

## Test Plan

1. Configure VertexAI provider with WIF credentials
2. Start OGX server
3. Send inference request to
`vertexai/publishers/google/models/gemini-2.5-flash`

**Before:** `RuntimeError: Event loop is closed` on every request
**After:** Request completes successfully<hr>This is an automatic
backport of pull request #6148 done by [Mergify](https://mergify.com).

Signed-off-by: Artemy Hladenko <[email protected]>
Signed-off-by: Artemy <[email protected]>
Co-authored-by: Artemy <[email protected]>
Co-authored-by: Charlie Doern <[email protected]>
Co-authored-by: Sébastien Han <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants