Skip to content

[Bug]: [Bug]: google-vertex models fail with model_not_found at runtime on 2026.5.28 and 2026.6.1 — direct Vertex API calls succeed with same credentials #90506

@paulogogs

Description

@paulogogs

Bug type

Regression (worked before, now fails)

Beta release blocker

No

Summary

After upgrading from 2026.5.26 to 2026.5.28 (confirmed still broken on 2026.6.1), all google-vertex/* models fail at runtime with model_not_found, while direct Vertex AI REST calls using the exact same authorized_user ADC credentials succeed. Model appears correctly in openclaw models list and gateway startup logs — failure only occurs when a request is made.

Steps to reproduce

  1. Configure openclaw.json with google-vertex/gemini-3-flash-preview as primary and google-vertex/gemini-2.5-flash as fallback
  2. Set GOOGLE_APPLICATION_CREDENTIALS=/path/to/vertex-key.json (authorized_user type), GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION=global in systemd drop-in env.conf
  3. Ensure plugins.entries.google.enabled = true
  4. Run on OpenClaw 2026.5.26 — works correctly
  5. Upgrade to 2026.5.28 or 2026.6.1 without any config changes
  6. Send any message to an agent
  7. Observe FallbackSummaryError: All models failed — model_not_found for both primary and fallback

Expected behavior

google-vertex/gemini-3-flash-preview responds normally, as observed on 2026.5.26 with identical config and credentials. Direct Vertex AI REST calls with the same authorized_user credentials succeed on both broken versions, confirming the credentials and model are valid.

Actual behavior

FallbackSummaryError: All models failed (2): google-vertex/gemini-3-flash-preview: Unknown model: google-vertex/gemini-3-flash-preview (model_not_found) | google-vertex/gemini-2.5-flash: Unknown model: google-vertex/gemini-2.5-flash (model_not_found)

Gateway log: [model-fallback/decision] decision=candidate_failed reason=model_not_found
Auth state shows no google-vertex entry. openclaw models auth login --provider google-vertex returns "No provider plugins found" despite google plugin being enabled and loaded.

OpenClaw version

2026.5.28 (first broken) | 2026.6.1 (2e08f0f) also broken | 2026.5.26 (10ad3aa) last known good

Operating system

Ubuntu 24.04

Install method

No response

Model

npm global

Provider / routing chain

openclaw -> google-vertex (ADC/authorized_user) -> Vertex AI global endpoint

Additional provider/model setup details

  • Credential type: authorized_user (gcloud auth application-default login), NOT service account
  • GOOGLE_APPLICATION_CREDENTIALS=/root/vertex-key.json
  • GOOGLE_CLOUD_PROJECT=project-4d1b0250-40a7-450a-a1f
  • GOOGLE_CLOUD_LOCATION=global
  • plugins.entries.google.enabled = true
  • No google-vertex entry in auth-profiles.json (Vertex works via ADC env vars, not auth profiles)
  • openclaw models auth login --provider google-vertex returns "No provider plugins found"
  • openclaw models list correctly shows google-vertex models with Auth: yes
  • Gateway startup log confirms: [gateway] agent model: google-vertex/gemini-3-flash-preview

Logs, screenshots, and evidence

# Gateway log (runtime failure):
[diagnostic] lane task error: error="FailoverError: Unknown model: google-vertex/gemini-3-flash-preview"
[model-fallback/decision] decision=candidate_failed reason=model_not_found next=google-vertex/gemini-2.5-flash
[model-fallback/decision] decision=candidate_failed reason=model_not_found next=none

# Direct Vertex API call with same credentials (succeeds on both broken versions):
python3 -c "
import google.auth, google.auth.transport.requests, urllib.request, json
creds, project = google.auth.default()
creds.refresh(google.auth.transport.requests.Request())
url = f'https://aiplatform.googleapis.com/v1/projects/{project}/locations/global/publishers/google/models/gemini-3-flash-preview:generateContent'
# Returns HTTP 200 — model and credentials valid
"

# Related issues: #65715 (anthropic-vertex ADC env spread copy), #56253 (google-vertex ADC ignored)

Impact and severity

Affected: Any user running google-vertex/* models with authorized_user ADC credentials (gcloud auth application-default login) on Linux, via systemd gateway
Severity: High — blocks all agent requests, forces downgrade or expensive fallback to paid Anthropic API
Frequency: Always — 100% reproducible, every request fails on affected versions
Consequence: All agents go offline on google-vertex primary + fallback; forced fallback to Anthropic claude-sonnet-4-6 incurs unexpected API costs

Additional information

Last known good: 2026.5.26 (10ad3aa)
First known bad: 2026.5.28
Also broken on: 2026.6.1 (2e08f0f) — 2026.6.1 release notes mention "Google/Vertex catalog fixes" (#88480, #88512, #88851, #88860) but did not fix this specific case.
Temporary workaround: downgrade to 2026.5.26 — zero config changes needed, Vertex works immediately.
Hypothesis: similar root cause to #65715 (anthropic-vertex ADC detection fails when env is spread copy of process.env). The google-vertex provider's catalog resolution may share the same env !== process.env identity check that prevents ADC credential discovery at runtime.
Note: authorized_user credentials (not service account) may be a distinct code path not covered by existing fixes.

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions