-
-
Notifications
You must be signed in to change notification settings - Fork 78.9k
[Bug]: [Bug]: google-vertex models fail with model_not_found at runtime on 2026.5.28 and 2026.6.1 — direct Vertex API calls succeed with same credentials #90506
Copy link
Copy link
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.ClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.ClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Give feedbackNo fields configured for issues without a type.
Bug type
Regression (worked before, now fails)
Beta release blocker
No
Summary
After upgrading from 2026.5.26 to 2026.5.28 (confirmed still broken on 2026.6.1), all google-vertex/* models fail at runtime with model_not_found, while direct Vertex AI REST calls using the exact same authorized_user ADC credentials succeed. Model appears correctly in openclaw models list and gateway startup logs — failure only occurs when a request is made.
Steps to reproduce
Expected behavior
google-vertex/gemini-3-flash-preview responds normally, as observed on 2026.5.26 with identical config and credentials. Direct Vertex AI REST calls with the same authorized_user credentials succeed on both broken versions, confirming the credentials and model are valid.
Actual behavior
FallbackSummaryError: All models failed (2): google-vertex/gemini-3-flash-preview: Unknown model: google-vertex/gemini-3-flash-preview (model_not_found) | google-vertex/gemini-2.5-flash: Unknown model: google-vertex/gemini-2.5-flash (model_not_found)
Gateway log: [model-fallback/decision] decision=candidate_failed reason=model_not_found
Auth state shows no google-vertex entry. openclaw models auth login --provider google-vertex returns "No provider plugins found" despite google plugin being enabled and loaded.
OpenClaw version
2026.5.28 (first broken) | 2026.6.1 (2e08f0f) also broken | 2026.5.26 (10ad3aa) last known good
Operating system
Ubuntu 24.04
Install method
No response
Model
npm global
Provider / routing chain
openclaw -> google-vertex (ADC/authorized_user) -> Vertex AI global endpoint
Additional provider/model setup details
Logs, screenshots, and evidence
Impact and severity
Affected: Any user running google-vertex/* models with authorized_user ADC credentials (gcloud auth application-default login) on Linux, via systemd gateway
Severity: High — blocks all agent requests, forces downgrade or expensive fallback to paid Anthropic API
Frequency: Always — 100% reproducible, every request fails on affected versions
Consequence: All agents go offline on google-vertex primary + fallback; forced fallback to Anthropic claude-sonnet-4-6 incurs unexpected API costs
Additional information
Last known good: 2026.5.26 (10ad3aa)
First known bad: 2026.5.28
Also broken on: 2026.6.1 (2e08f0f) — 2026.6.1 release notes mention "Google/Vertex catalog fixes" (#88480, #88512, #88851, #88860) but did not fix this specific case.
Temporary workaround: downgrade to 2026.5.26 — zero config changes needed, Vertex works immediately.
Hypothesis: similar root cause to #65715 (anthropic-vertex ADC detection fails when env is spread copy of process.env). The google-vertex provider's catalog resolution may share the same env !== process.env identity check that prevents ADC credential discovery at runtime.
Note: authorized_user credentials (not service account) may be a distinct code path not covered by existing fixes.