Skip to content

[Bug]: vertex_count_tokens_location ignored in handler.py — Claude count_tokens always routes to us-central1 #23872

@VivaciousDesire

Description

@VivaciousDesire

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When using Claude models on Vertex AI with a non-us-central1 location (e.g., europe-west1), the count_tokens endpoint always routes to us-central1 regardless of
vertex_location or vertex_count_tokens_location config. This causes 500 errors if the project doesn't have Claude count-tokens available in us-central1.

A workaround was added in common_utils.py (line 1064) to support vertex_count_tokens_location, but it is silently nullified by handler.py further down the call chain.


Root Cause

The fix in common_utils.py correctly resolves vertex_count_tokens_location:

# common_utils.py:1064
vertex_location = (
    count_tokens_params_request.get("vertex_count_tokens_location")
    or vertex_location
)
# → correctly resolves to "europe-west1" ✅

This value is passed to VertexAIPartnerModels.count_tokens() as vertex_location="europe-west1", which copies it into _litellm_params["vertex_location"] and calls
handle_count_tokens_request().

However, inside handler.py, the location is re-read from litellm_params and then unconditionally overridden because "claude" is always present in the model name:

# handler.py:108-113 — BROKEN
vertex_location = self.get_vertex_ai_location(litellm_params)
# → reads "europe-west1"

if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"
# → "claude" is ALWAYS in Claude model names → always overrides to "us-central1" ❌
# → vertex_count_tokens_location is never consulted here

---
Expected Behavior

vertex_count_tokens_location should take precedence over the us-central1 default. The default override should only apply when no explicit count-tokens location is configured.

---
Proposed Fix

In litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py, change lines 108113:

# Current (broken):
vertex_location = self.get_vertex_ai_location(litellm_params)
if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"

# Fixed:
vertex_location = (
    litellm_params.get("vertex_count_tokens_location")
    or self.get_vertex_ai_location(litellm_params)
)
if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"

This ensures vertex_count_tokens_location is checked first. If set, the us-central1 override is bypassed. If not set and the model is Claude, it falls back to us-central1 as before.

---
Steps to Reproduce

1. Configure a Claude model on Vertex AI with vertex_location: europe-west1 and vertex_count_tokens_location: europe-west1
2. Make a request that triggers token counting
3. Observe: request routes to us-central1 instead of europe-west1500 error

LiteLLM config:

model_list:
- model_name: claude-sonnet-4-6
litellm_params:
model: vertex_ai/claude-sonnet-4-6
vertex_project: my-gcp-project
vertex_location: europe-west1
vertex_count_tokens_location: europe-west1 # ignored due to bug


  ---
  Call Chain

common_utils.py:1064 → resolves vertex_count_tokens_location = "europe-west1" ✅
main.py:323-327 → copies into _litellm_params["vertex_location"] = "europe-west1" ✅
handler.py:108 → re-reads vertex_location from litellm_params = "europe-west1"
handler.py:112 → "claude" in model → overrides to "us-central1" ❌
handler.py:123-127 → builds URL with us-central1 → 500


### Relevant log output

```shell
litellm.proxy.anthropic_endpoints.count_tokens(): Exception occurred -
  Server error '500 Internal Server Error' for url
  'https://us-central1-aiplatform.googleapis.com/v1/projects/getmoss-ai/locations/us-central1/publishers/anthropic/models/count-tokens:rawPredict'

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

1.81.14

Twitter / LinkedIn details

https://www.linkedin.com/in/akanksha-singh-2989aa164/

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions