@markmcd Is there any way we could make (at least some of) the same values work for both GLA and Vertex?

cc @ewjoachim

I'll wait for the Vertex opinion on this one. The GLA values align with what other major providers do, so I'd prefer to remap the Vertex values back (if that's even feasible?)

These are the relevant docs:

• https://docs.cloud.google.com/vertex-ai/generative-ai/docs/provisioned-throughput/use-provisioned-throughput
• https://docs.cloud.google.com/vertex-ai/generative-ai/docs/flex-paygo

They control the following headers:

We could definitly use standard and flex though it's ambiguous if they should map to equivalent with or without PT. That said, by default, PT is used unless we send X-Vertex-AI-LLM-Request-Type: shared so it could make sense to:

• Replace pt_then_on_demand -> standard and pt_then_flex -> flex, but then we would need to rename at least on_demand to somthing like on_demand_only or pay_as_you_go_only (to match flex_only`)
• Or we could treat standard and flex as aliases for pt_then_on_demand and pt_then_flex but keep the non-ambiguous names around, it would mean a bit of extra documentation work to convey this in a very unambiguous manner, but this lets folks write unambiguous code.

priority doesn't match to anything on vertex and pt doesn't match to anything on Google (as far as I can tell), so it will be hard to get anything perfect. Not sure exactly if that's 100% helpful, but if you think I can help further, feel free :)

My vote is for unambiguous options, and then provide the additional aliases.

The GLA impl aligns with, e.g. OpenAI, and while that doesn't happen transparently in this case, it's a nice portability feature. I think providing similar ergonomics for the Vertex values would be positive, as long as that's a reasonable mental model for Vertex customers (hopefully you can make that call @ewjoachim! I don't know that stack well)

If we agree on this, I'll update the PR to make it clear that standard and flex are accepted for vertex as a shim only.

• Since we support service tiers now for OpenAI, Bedrock, GLA, and Vertex, it'd be nice to add a new top-level service_tier ModelSetting with a narrow set of values (most likely the OpenAI ones) that we then try to map to providers (i.e. interpret in the model classes) as best we can, with clear documentation (in the docstring) of how each provider interprets them.
• If we have a narrower set there, we could then rename the google_service_tier field to google_vertex_service_tier (and deprecate the original). Then we may either not need a separate google_gla_service_tier (if the top level service_tier covers all the values), or we can add a new google_gla_service_tier in case granular control is needed.

That way we get the convenience of a single set of values across providers, with the ability to override per-provider values as needed.

I don't have an opinion yet on what the exact top-level service tier values, or mapping to Vertex, should be, but if the above approach makes sense to you I trust either/both of you to be able to come up with some reasonable 😄

OK I've updated to take this into account using OpenAI's values as the "default". Provider-specific values take precedence, and supported providers have been updated, including adding mappings from generic to specific where it makes sense to.

Another PR has appeared that also addresses this, #5158, I haven't looked at it, but I'm not precious about keeping mine if that's better.

@markmcd Thanks Mark, I've been working with an agent on consolidating these related PRs so I'll tell it to look at your new changes.

The agent did have a question for you (to pass on to the Vertex team). In its own words:

@Mawox has been picking up the design direction in #5158 (top-level service_tier with per-provider fallbacks as we discussed), and @anatolec
added Priority PayGo values in #5094. I'm folding both into one PR and want to extend the cross-provider mapping to Vertex, rather than keeping the "Vertex ignored" carve-out — with
pt_then_priority now in scope, the mapping looks clean:

• flex → X-Vertex-AI-LLM-Shared-Request-Type: flex
• priority → X-Vertex-AI-LLM-Shared-Request-Type: priority
• default / auto → no headers (PT-then-on-demand default)

Before I commit to that, there's one thing the public docs don't quite spell out: if a project has zero PT quota on the target model/region and we send only the single shared-request-type
header, does the request fall through safely to Flex/Priority PayGo, or does it 429?

@ewjoachim's original writeup describes it as "Uses PT first. Excess traffic spills over to Flex PayGo," and @anatolec saw traffic_type: ON_DEMAND_PRIORITY in a live test — so empirically
it looks safe. But before defaulting every cross-provider service_tier='priority' user through this on Vertex, could you check with the DeepMind / Vertex team?

Specifically:

1. Zero-PT project + only X-Vertex-AI-LLM-Shared-Request-Type: flex → 429, or Flex PayGo?
2. Same with priority → 429, or Priority PayGo?
3. For a project with PT quota, is the spillover destination when these single-header requests exceed PT actually Flex/Priority (not standard on-demand)?

If 1+2 fall through safely we'll go single-header (respects PT for customers who have it, safe for everyone else). If not, we'll also send X-Vertex-AI-LLM-Request-Type: shared to guarantee
no PT dependency at the cost of bypassing PT entirely. I'd rather the former if it's actually safe.

Thanks!

Thanks @DouweM — happy to fold this into #5158.

Reproduced Devin's Q1 on a separate zero-PT Vertex project (gemini-3-flash-preview, location='global' — Flex PayGo is preview-only):

pt_then_flex    traffic_type='ON_DEMAND_FLEX'      ← Q1: single Shared-Request-Type: flex on zero-PT
flex_only       traffic_type='ON_DEMAND_FLEX'
pt_only         429, PT quota exceeded             ← zero-PT control ✓

Q2 (priority) isn't on this branch — @anatolec's #5094 already shows the same pattern: pt_then_priority → ON_DEMAND_PRIORITY on zero-PT.

Q3 (PT-quota spillover destination) still needs the Vertex team.

If Q3 spills to Flex/Priority: drop the carve-out in #5158, flex/priority → single Shared-Request-Type header, keep google_vertex_service_tier as the escape hatch (needs #5094 folded in first for the priority mapping). If it spills to plain on-demand: keep the carve-out — silent downgrade from priority is worse than requiring the explicit field on Vertex.

(Just clarifying that I don't feel I'm sufficiently knowledgeable on the subject to add anything meaningful to what has already been said)

Minor: when service_tier is not set at all (neither anthropic_service_tier nor service_tier), the or chain evaluates to None, and then None not in ('auto', 'standard_only') is True, so service_tier gets set to OMIT. This works but is subtle — it would be cleaner to guard with an early if service_tier is None: service_tier = OMIT before the mapping logic, for readability.

🚩 Bedrock unified service_tier='default' maps to {'type': 'default'} — verify this is valid

At bedrock.py:696-697, the unified service_tier='default' is wrapped as {'type': 'default'}. The ServiceTierTypeDef accepts Literal['default', 'flex', 'priority', 'reserved'], so 'default' should be valid. However, the Bedrock docs page linked from the docstring should be checked to confirm that 'default' is actually a meaningful tier value (as opposed to just being the absence of a tier selection). The test at test_bedrock.py:678-718 mocks the Bedrock client and verifies the dict structure but doesn't validate against the real API.

Was this helpful? React with 👍 or 👎 to provide feedback.

I might be completely off, but it seems strange that we mention "Gemini" here (I believe for GLA?), and google_vertex_service_tier above. I wonder it this bullet list should also mention Vertex (especially since Vertex can be used for non-gemini models)

(also, it looks like this docstring is duplicated from the TypeAlias above, which might lead to either getting out of sync)

I'm afraid conflating the provider-specific service-tier and the model_settings service tier is bound to create headaches: at some point, some provider is going to call their default mode "flex" or something like that.

Rather than putting either value in a variable and then handle a GoogleVertexServiceTier | ServiceTier, I think it's much much saner to:

1. See if a provider-specific (in this case GoogleVertexServiceTier) value is defined. If so, use it.
2. If not, map the ServiceTier to a GoogleVertexServiceTier
3. Always handle a GoogleVertexServiceTier

(here I'm saying this for Vertex but that would be the way we handle it for every other provider)

As the zen of python says: "In the face of ambiguity, refuse the temptation to guess."

🚩 Other OpenAI-compatible providers (Groq, xAI, OpenRouter) will silently pass service_tier through to the API

Only Cerebras explicitly marks service_tier and openai_service_tier as unsupported via openai_unsupported_model_settings. Other OpenAI-compatible providers (Groq, xAI, OpenRouter) don't declare these as unsupported, so the _resolve_openai_service_tier function will resolve the unified service_tier and pass it through to those APIs. Whether this is a bug depends on whether those providers support or ignore the service_tier parameter — OpenAI-compatible APIs generally ignore unknown parameters, so this is likely harmless, but it's worth verifying.

(Refers to lines 64-71)

Was this helpful? React with 👍 or 👎 to provide feedback.

OK to leave as is, I believe.