Repository navigation
Gemini model calls return 401 about 24h after start on Agent Runtime (Agent Identity) #7697
Description
🔴 Required Information
Describe the Bug:
An agent deployed to Vertex AI Agent Runtime with identity_type: AGENT_IDENTITY starts failing model calls with 401 UNAUTHENTICATED roughly 24 hours after the container starts. It keeps failing until the process is restarted. Under Agent Identity, access tokens are bound to a client certificate that rotates, and the long-lived genai.Client that Gemini.api_client caches appears to keep using the old one. Gemini offers no supported way to bound the client's age or rebuild it after an auth failure.
The one workaround, a Gemini subclass that evicts the cached client on a timer, was valid up to 2.9.x because the docstring recommended overriding api_client. In 2.10.0 api_client became a PerLoopCachedProperty stored under _per_loop_api_client, so evicting __dict__["api_client"] now silently does nothing. We have to maintain a private-attribute hack, which breaks when we update to the newer versions of the package.
Steps to Reproduce:
- Deploy any ADK agent using
Gemini(defaultapi_client) to Agent Runtime withidentity_type: AGENT_IDENTITY. - Send requests normally, then leave the engine running.
- About 24 hours after start, send another request.
- Model call fails with 401.
Expected Behavior:
Model calls keep working across certificate rotation. Failing that, Gemini should offer a supported way to bound client age (e.g. client_max_age) or rebuild on auth failure (e.g. an invalidate() that clears the per-loop cache).
Observed Behavior:
401 UNAUTHENTICATED from the model call about 24h in, until restart. With a subclass that rebuilds the client on a timer, the failures stop (the private attribute hack).
Environment Details:
- ADK Library Version: 2.9.x
- Desktop OS: Linux container on Vertex AI Agent Runt
- Python Version: 3.14
Model Information:
- Are you using LiteLLM: No
- Which model is being used: gemini-3.5-flash
🟡 Optional Information
Regression:
Partly. The 401 itself is not tied to an ADK version as far as I know. We did have a workaround, which no longer works in the newer versions of ADK.
Logs:
N/A
Additional Context:
- Certificate rotation is the working explanation; I have not captured certificate
notBeforedates, so it is not confirmed. client_kwargsconfigures the client but does not bound its lifetime, so it is not a substitute.
Minimal Reproduction Code:
The 401 needs a deployed engine and about a day, it has no minimal repro code.
How often has this issue occurred?:
- Always (100%) on a long-running engine.
Activity
surajksharma07 commentedon Oct 26, 2026
Confirmed the eviction part locally @JCorpe26. Popping dict["api_client"] rebuilds the client on 2.9.2 but is a no-op on 2.10+ since the cache moved to _per_loop_api_client.
As a workaround you could try a small Gemini subclass that clears both keys (plus the live client ones) on a client_max_age timer and rebuilds + retries once on a 401. It works on both 2.9.2 and 2.13.0 with a mocked call.
Could you share the exact ADK version, pip freeze (mainly google-genai/google-auth/httpx), the full 401 response or stack trace and whether other Google API calls in the same process also fail at that point?
If you’re able to grab the client cert notBefore around the failure that would really help confirm the rotation theory. Please test the workaround on a long-running engine before we look at a PR for a public invalidate API.
inamdarmihir commentedon Oct 29, 2026
Opened #7797 for this. It adds two optional controls on
Gemini:client_max_age(seconds, defaultNonekeeps today's behavior) which rebuilds the api and live clients once they reach that age, andinvalidate_clients()to drop them on demand for every event loop. Old clients are dropped, not closed, so in-flight requests keep working. I left out an automatic rebuild-and-retry on 401 because it can happen mid-stream and a retry could replay non-idempotent calls. I couldn't reproduce the 24h Agent Identity failure itself, so the tests cover the cache behavior (expiry boundary, invalidate, per-loop isolation, concurrency). Happy to rename or reshape the API if you'd prefer something else.surajksharma07 commentedon Nov 2, 2026
Went through #7797 @inamdarmihir and it looks good. It applies cleanly on current main (2.13.0), the generation counter covers a clear() that lands mid-build, and I agree with skipping the auto retry on 401 since it could replay a call mid-stream.
One thing on scope: VertexAiSessionService and VertexAiMemoryBankService also cache their clients per loop now. If it really is cert rotation they’d probably start failing at the same 24h mark, so it’d be good to know if those need the same treatment here or in a follow-up.
@JCorpe26 could you try the PR branch on a long-running engine with something like client_max_age=3600? You can also call model.invalidate_clients() from an on_model_error_callback when you get a 401 so the next call builds a fresh client.
Also if you still have them, the ADK/google-genai/google-auth versions, the 401 trace and whether session or memory calls fail around the same time would help confirm the root cause before this gets reviewed.