Skip to content

Commit 6438ded

Browse files
atishpatelclaude
andauthored
feat(agent): route Claude/GPT model families to their native gateway wire (#3538)
## Summary Databricks v2 chooses the gateway wire format — OpenAI Responses, Anthropic Messages, or MLflow chat — purely from substrings in the endpoint name. There is no family field on the endpoint to key off, so the substring set *is* the routing contract. The matcher only recognised `gpt-5`/`gpt5` and `claude`, which makes correct billing depend on every Claude endpoint happening to be named with the literal string "claude". ## Why this matters Getting a Claude model onto the Anthropic Messages route is exactly what lets buzz attach the `cache_control` breakpoint (the fix in #3463). If a Claude endpoint's catalog name omits "claude" — an alias, a bare `opus-5`, a `goose-opus-5` — it silently falls through to the MLflow (OpenAI-wire) path, where Anthropic prompt caching is **structurally impossible**. The result is the same failure #3463 fixed: 0% cache reads, the full ~10x read discount lost, and no error — a naming convention quietly holding up a billing-correctness invariant. ## What changed `databricks_v2_route_for_model` (`crates/buzz-agent/src/llm.rs`) now matches broader, case-insensitive marker sets: - **Claude → Anthropic Messages:** `claude`, `opus`, `sonnet`, `haiku`, `mythos`, `fable` — the Claude family names and release code names, so a Claude endpoint reaches the cache-capable route regardless of how it's named. - **GPT → OpenAI Responses:** the `gpt` family (now `gpt` on its own, not just `gpt-5`) plus the GPT-5 launch code names `sol`, `luna`, `terra`. OpenAI markers are evaluated first, preserving the prior `gpt-5`-first precedence for any name that could carry both. Names matching neither set still fall through to the MLflow chat route. ## Testing - `cargo fmt`, `cargo clippy -p buzz-agent --all-targets -- -D warnings` — clean. - `cargo test -p buzz-agent` — all green (299 lib + integration suites, 0 failures). The `databricks_v2_routes_by_model_family` test was expanded to cover each new marker, the GPT-5 code names, case-insensitivity, and the unchanged MLflow fallback (including `gemini`). ## Relationship to #3463 #3463 taught the Anthropic path to request caching; this makes sure Claude models actually land on that path. Follow-up still open: surfacing `cache_creation_input_tokens` end-to-end so a persistent `reads == 0 && writes == 0` reveals a disabled cache regardless of which wire a model takes — happy to do that next. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Atish Patel <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 24d90d1 commit 6438ded

1 file changed

Lines changed: 133 additions & 8 deletions

File tree

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 133 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1019,13 +1019,53 @@ fn is_responses_required_error(body: &str) -> bool {
10191019
|| b.contains("use the responses api")
10201020
}
10211021

1022+
/// OpenAI-family code names that appear as their own segment in a Databricks v2
1023+
/// endpoint name (the GPT-5 launch aliases). The `gpt` family itself is matched
1024+
/// separately by segment prefix so `gpt`, `gpt5`, and the `gpt` of a split
1025+
/// `gpt-5` all qualify.
1026+
const DATABRICKS_V2_OPENAI_CODE_NAMES: &[&str] = &["sol", "luna", "terra"];
1027+
1028+
/// Anthropic (Claude) family and release code names that appear as their own
1029+
/// segment in a Databricks v2 endpoint name — the `claude` prefix, the family
1030+
/// names (`opus`, `sonnet`, `haiku`), and the release code names (`mythos`,
1031+
/// `fable`). Getting a Claude model onto the Anthropic Messages route is what
1032+
/// lets it carry a `cache_control` breakpoint; an endpoint that matches none of
1033+
/// these falls through to the MLflow (OpenAI-wire) path, where Anthropic prompt
1034+
/// caching is structurally impossible and the discount is silently lost.
1035+
const DATABRICKS_V2_CLAUDE_NAMES: &[&str] =
1036+
&["claude", "opus", "sonnet", "haiku", "mythos", "fable"];
1037+
1038+
/// Split a Databricks v2 endpoint name into its lowercase alphanumeric segments,
1039+
/// breaking on any non-alphanumeric delimiter (`-`, `_`, `.`, `/`, …). E.g.
1040+
/// `Databricks-Claude-Opus-5` -> `["databricks", "claude", "opus", "5"]`.
1041+
fn model_name_segments(model: &str) -> Vec<String> {
1042+
model
1043+
.split(|c: char| !c.is_ascii_alphanumeric())
1044+
.filter(|s| !s.is_empty())
1045+
.map(str::to_ascii_lowercase)
1046+
.collect()
1047+
}
1048+
10221049
fn databricks_v2_route_for_model(model: &str) -> DatabricksV2Route {
1023-
// Databricks v2 catalog names currently identify OpenAI-shaped GPT-5
1024-
// models and Anthropic-shaped Claude models by these substrings.
1025-
let lower = model.to_ascii_lowercase();
1026-
if lower.contains("gpt-5") || lower.contains("gpt5") {
1050+
// The v2 catalog exposes no family field, so the wire format is inferred
1051+
// from the endpoint name. Discovery deliberately keeps arbitrary custom
1052+
// aliases, so we match whole name *segments* rather than raw substrings: a
1053+
// substring test would misroute unrelated names — `consolidated-llama`
1054+
// (`sol`), `terraform-coder` (`terra`), `corpus-reranker`/`octopus-model`
1055+
// (`opus`) — onto a wire whose request shape their backend can't parse,
1056+
// turning a caching optimization into a hard request/parse failure. Segment
1057+
// matching still accepts real prefixed names like `goose-opus-5`.
1058+
let segments = model_name_segments(model);
1059+
let has_named_segment =
1060+
|names: &[&str]| segments.iter().any(|seg| names.contains(&seg.as_str()));
1061+
// `gpt` family: any segment beginning with `gpt` — covers `gpt`, `gpt5`, and
1062+
// the `gpt` segment of a split `gpt-5`, without matching mid-word.
1063+
let is_gpt_family = segments.iter().any(|seg| seg.starts_with("gpt"));
1064+
// OpenAI is checked before Claude so a name carrying both markers resolves
1065+
// to the OpenAI wire (preserving the prior `gpt-5`-first precedence).
1066+
if is_gpt_family || has_named_segment(DATABRICKS_V2_OPENAI_CODE_NAMES) {
10271067
DatabricksV2Route::OpenAiResponses
1028-
} else if lower.contains("claude") {
1068+
} else if has_named_segment(DATABRICKS_V2_CLAUDE_NAMES) {
10291069
DatabricksV2Route::AnthropicMessages
10301070
} else {
10311071
DatabricksV2Route::MlflowChatCompletions
@@ -2579,20 +2619,105 @@ mod tests {
25792619

25802620
#[test]
25812621
fn databricks_v2_routes_by_model_family() {
2622+
use DatabricksV2Route::{AnthropicMessages, MlflowChatCompletions, OpenAiResponses};
25822623
for (model, route, path) in [
2624+
// OpenAI-shaped: the gpt family plus the GPT-5 code names.
25832625
(
25842626
"databricks-gpt-5-5",
2585-
DatabricksV2Route::OpenAiResponses,
2627+
OpenAiResponses,
2628+
"/ai-gateway/openai/v1/responses",
2629+
),
2630+
("gpt-4o", OpenAiResponses, "/ai-gateway/openai/v1/responses"),
2631+
// The intentional dashless `gpt5` spelling still routes to OpenAI.
2632+
("gpt5", OpenAiResponses, "/ai-gateway/openai/v1/responses"),
2633+
(
2634+
"databricks-gpt-5-6-luna",
2635+
OpenAiResponses,
2636+
"/ai-gateway/openai/v1/responses",
2637+
),
2638+
(
2639+
"databricks-gpt-5-6-sol",
2640+
OpenAiResponses,
2641+
"/ai-gateway/openai/v1/responses",
2642+
),
2643+
(
2644+
"databricks-terra",
2645+
OpenAiResponses,
25862646
"/ai-gateway/openai/v1/responses",
25872647
),
2648+
// Anthropic-shaped: the claude prefix, the family names, and the
2649+
// release code names — each must reach the cache-capable route even
2650+
// when the endpoint name omits the literal "claude".
25882651
(
25892652
"databricks-claude-opus-4-7",
2590-
DatabricksV2Route::AnthropicMessages,
2653+
AnthropicMessages,
25912654
"/ai-gateway/anthropic/v1/messages",
25922655
),
2656+
(
2657+
"goose-opus-5",
2658+
AnthropicMessages,
2659+
"/ai-gateway/anthropic/v1/messages",
2660+
),
2661+
(
2662+
"databricks-sonnet-5",
2663+
AnthropicMessages,
2664+
"/ai-gateway/anthropic/v1/messages",
2665+
),
2666+
(
2667+
"databricks-haiku-4-5",
2668+
AnthropicMessages,
2669+
"/ai-gateway/anthropic/v1/messages",
2670+
),
2671+
(
2672+
"databricks-mythos-5",
2673+
AnthropicMessages,
2674+
"/ai-gateway/anthropic/v1/messages",
2675+
),
2676+
(
2677+
"databricks-fable-5",
2678+
AnthropicMessages,
2679+
"/ai-gateway/anthropic/v1/messages",
2680+
),
2681+
// Case-insensitive.
2682+
(
2683+
"Databricks-Claude-Opus-5",
2684+
AnthropicMessages,
2685+
"/ai-gateway/anthropic/v1/messages",
2686+
),
2687+
// Unrecognised names still fall through to the MLflow chat route.
25932688
(
25942689
"custom-tool-model",
2595-
DatabricksV2Route::MlflowChatCompletions,
2690+
MlflowChatCompletions,
2691+
"/ai-gateway/mlflow/v1/chat/completions",
2692+
),
2693+
(
2694+
"databricks-gemini-3-pro",
2695+
MlflowChatCompletions,
2696+
"/ai-gateway/mlflow/v1/chat/completions",
2697+
),
2698+
// Collision guard: short code names must match only as whole
2699+
// segments, never as substrings of an unrelated custom alias.
2700+
// Each of these embeds a marker (`sol`, `terra`, `opus`) mid-word
2701+
// and must stay on the MLflow fallback, not adopt a wire its
2702+
// backend can't parse.
2703+
(
2704+
"consolidated-llama",
2705+
MlflowChatCompletions,
2706+
"/ai-gateway/mlflow/v1/chat/completions",
2707+
),
2708+
(
2709+
"terraform-coder",
2710+
MlflowChatCompletions,
2711+
"/ai-gateway/mlflow/v1/chat/completions",
2712+
),
2713+
(
2714+
"corpus-reranker",
2715+
MlflowChatCompletions,
2716+
"/ai-gateway/mlflow/v1/chat/completions",
2717+
),
2718+
(
2719+
"octopus-model",
2720+
MlflowChatCompletions,
25962721
"/ai-gateway/mlflow/v1/chat/completions",
25972722
),
25982723
] {

0 commit comments

Comments
 (0)