Skip to content

Commit cd54e26

Browse files
baxenFizz
andauthored
fix(agent): route GPT-5+ model-service FQNs to Responses (#7358)
## Summary Route Databricks Unity Catalog model services to OpenAI Responses when their service name matches GPT-5 or newer. These models can reject tools plus reasoning on Chat Completions. Match only the service component, using the existing family-token boundaries and a numeric major version. Catalog and schema names cannot select the protocol. Keep neutral effort capabilities and the full model ID unchanged; other services still use MLflow Chat Completions. Keep the Rust and desktop resolvers in sync, add shared boundary cases and a captured-HTTP regression for completion and summarization, and update the documented FQN rule. ### Related issue No duplicate found in searches for “FQN responses” PRs or “astra” issues. Related: #6918 introduced Unity Catalog discovery. Originating conversation: buzz://message?channel=0b881928-a3a6-4c01-b981-8e64268f01ce&id=770949343bc96a9ed88acd90a1b37d358a0efc52c79237d0fdb491ce02b8d4ed ### Testing No live Databricks inference test. The gateway must accept the full model-service ID on its OpenAI Responses route; this remains the integration risk. The local `just ci` attempt exceeded its five-minute deadline during `mobile-check`, so the full repository gate was not completed. All push-hook checks passed. Generated with Codex Signed-off-by: Fizz <400e8babadcee6a7f420103f10a2849d84c4a9c71d5bd04f3948c814216648a3@buzz.block.builderlab.xyz> Co-authored-by: Fizz <400e8babadcee6a7f420103f10a2849d84c4a9c71d5bd04f3948c814216648a3@buzz.block.builderlab.xyz>
1 parent 86c189e commit cd54e26

9 files changed

Lines changed: 559 additions & 16 deletions

File tree

‎crates/buzz-agent/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,7 @@ lifecycle hook — see [MCP_DRIVEN_HOOKS.md](../../docs/MCP_DRIVEN_HOOKS.md).
242242
| Block Gateway | `openai` | `POST {base}/chat/completions` | gpt-5, claude |
243243
| OpenRouter | `openrouter` | `POST {base}/chat/completions` | anything they route (extended-thinking replay, provider-agnostic tool calling) |
244244
| Databricks | `databricks` | `POST {host}/serving-endpoints/{model}/invocations` | goose-claude-4-6-sonnet |
245-
| Databricks AI Gateway v2 | `databricks_v2` | `POST {host}/ai-gateway/{provider}/v1/...` | workspace endpoints and Unity Catalog model-service FQNs; UC FQNs use MLflow Chat Completions |
245+
| Databricks AI Gateway v2 | `databricks_v2` | `POST {host}/ai-gateway/{provider}/v1/...` | workspace endpoints and Unity Catalog model-service FQNs; UC GPT-5+ services use OpenAI Responses; other UC FQNs use MLflow Chat Completions |
246246

247247
The optional `DATABRICKS_MODEL_FILTER` applies only to model discovery. Each comma-separated entry is trimmed and matched against the complete raw ID with case-sensitive `*` (zero or more characters) and `?` (one Unicode character) semantics; patterns are OR-ed. Unset or blank preserves the full authenticated catalog. A nonblank value containing no usable patterns is rejected. This controls picker visibility only; Databricks and Unity Catalog permissions remain the authorization boundary. A filtered-empty result is authoritative and does not restore the built-in fallback models.
248248

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2570,6 +2570,7 @@ fn apply_anthropic_cache_control(body: &mut serde_json::Map<String, Value>) {
25702570

25712571
#[cfg(test)]
25722572
mod tests {
2573+
include!("llm_fqn_tests.rs");
25732574
use super::*;
25742575
use crate::config::{Config, HookServers, OpenAiApi, Provider, ThinkingSummary};
25752576
use crate::types::{HistoryItem, ToolCall, ToolResult, ToolResultContent};
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
// Included in llm::tests to reuse the production-path HTTP capture fixture.
2+
#[tokio::test]
3+
async fn gpt_fqn_completion_and_summary_use_responses() {
4+
let response = json!({"status":"completed", "output":[{
5+
"type":"message", "content":[{"type":"output_text", "text":"ok"}]
6+
}]});
7+
let (base_url, captured) = spawn_sequence_stub(vec![
8+
StubHttpResponse::ok(response.clone()),
9+
StubHttpResponse::ok(response),
10+
])
11+
.await;
12+
let mut config = cfg(Provider::DatabricksV2);
13+
config.base_url = base_url;
14+
config.thinking_effort = Some(ThinkingEffort::High);
15+
let model = "catalog.schema.goose-gpt-6-astra";
16+
let llm = Llm::new(&config).unwrap();
17+
let tools = vec![ToolDef {
18+
name: "test_tool".into(),
19+
description: "Test".into(),
20+
input_schema: json!({"type":"object", "properties":{}}),
21+
}];
22+
let result = llm
23+
.complete(
24+
&config,
25+
"system",
26+
&[HistoryItem::User("hello".into())],
27+
&tools,
28+
model,
29+
)
30+
.await
31+
.unwrap();
32+
assert_eq!(result.text, "ok");
33+
assert_eq!(
34+
llm.summarize(&config, "system", "history", 128, model)
35+
.await
36+
.unwrap(),
37+
"ok"
38+
);
39+
let requests = captured.lock().await;
40+
let posts: Vec<_> = requests.iter().filter(|r| r.method == "POST").collect();
41+
assert_eq!(posts.len(), 2);
42+
for request in &posts {
43+
assert_eq!(request.path, "/v1/ai-gateway/openai/v1/responses");
44+
let body = request.body.as_ref().unwrap();
45+
assert_eq!(body["model"], model);
46+
assert!(body.get("input").is_some());
47+
assert!(body.get("messages").is_none());
48+
assert!(body.get("reasoning_effort").is_none());
49+
}
50+
let completion = posts[0].body.as_ref().unwrap();
51+
assert!(completion["input"].is_array());
52+
assert_eq!(completion["reasoning"]["effort"], "high");
53+
assert_eq!(completion["tools"][0]["type"], "function");
54+
assert_eq!(completion["tools"][0]["name"], "test_tool");
55+
assert_eq!(posts[1].body.as_ref().unwrap()["max_output_tokens"], 128);
56+
}

‎crates/buzz-agent/src/model_capabilities.rs‎

Lines changed: 51 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -292,8 +292,8 @@ fn prefix_matches(token: &str, s: &str) -> bool {
292292
///
293293
/// Databricks Unity Catalog model-service names are catalog data, not model
294294
/// family hints. Both capability interpreters use this shape check before
295-
/// family matching so suffixes such as `kimi-k3` cannot inherit endpoint
296-
/// capabilities accidentally.
295+
/// family matching so services cannot inherit endpoint capabilities accidentally.
296+
/// GPT-5+ services have a route-only Responses exception.
297297
pub(crate) fn is_databricks_model_service_fqn(model: &str) -> bool {
298298
let mut components = model.split('.');
299299
let (Some(catalog), Some(schema), Some(service)) =
@@ -308,16 +308,35 @@ pub(crate) fn is_databricks_model_service_fqn(model: &str) -> bool {
308308
}) && components.next().is_none()
309309
}
310310

311+
/// Route GPT-5+ UC services to Responses without borrowing endpoint effort facts.
312+
/// Match the first family token in the service only, preserving the existing
313+
/// boundary semantics (e.g. `claude-gpt-5` is not a GPT service).
314+
fn fqn_requires_responses(model: &str) -> bool {
315+
let Some(service) = model.rsplit('.').next() else {
316+
return false;
317+
};
318+
let lower = service.to_ascii_lowercase();
319+
let stripped = strip_catalog_prefix(&lower, &manifest().family_tokens);
320+
let Some(version) = stripped.strip_prefix("gpt-") else {
321+
return false;
322+
};
323+
let digits = version.bytes().take_while(u8::is_ascii_digit).count();
324+
let (major, suffix) = version.split_at(digits);
325+
if suffix.starts_with(|c: char| c.is_ascii_alphanumeric()) {
326+
return false;
327+
}
328+
major.parse::<u32>().is_ok_and(|major| major >= 5)
329+
}
330+
311331
/// Resolve the capability profile for a `(provider, raw_model_id)` pair.
312332
pub fn resolve(provider: &str, raw_model_id: &str) -> CapabilityResult {
313333
let m = manifest();
314334
let canon = canonical_provider(provider);
315335
let blank = raw_model_id.trim().is_empty();
316336

317-
// Unity Catalog FQNs are neutral model-service identities. Resolve them
318-
// through the concrete-unknown fallback before any suffix can match a
319-
// provider family rule. Routing and effort normalization then share this
320-
// one answer in Rust and TypeScript.
337+
// FQNs keep neutral effort capabilities, but GPT-5+ service names need
338+
// Responses for tools with reasoning. Only inspect the service component:
339+
// catalog/schema names must never choose a model protocol.
321340
let model_service_fqn =
322341
canon == "databricks_v2" && is_databricks_model_service_fqn(raw_model_id);
323342

@@ -394,7 +413,11 @@ pub fn resolve(provider: &str, raw_model_id: &str) -> CapabilityResult {
394413
thinking_mode: state.thinking_mode,
395414
supported_efforts: &state.supported_efforts,
396415
default_effort: state.default_effort,
397-
databricks_v2_wire_route: state.databricks_v2_wire_route,
416+
databricks_v2_wire_route: if model_service_fqn && fqn_requires_responses(raw_model_id) {
417+
DatabricksV2Route::OpenaiResponses
418+
} else {
419+
state.databricks_v2_wire_route
420+
},
398421
normalization_policy: state.normalization_policy,
399422
registry_label: None,
400423
}
@@ -724,6 +747,25 @@ mod tests {
724747
Q::Vector { id: "boundary-claude-3-digit-run-anthropic-probe", provider: "anthropic", raw_model_id: "claude-35", note: Some("Probes whether the claude-3 prefix binds a longer digit run ('35').") },
725748
Q::Vector { id: "boundary-claude-opus-4-70-anthropic-probe", provider: "anthropic", raw_model_id: "claude-opus-4-70", note: Some("Probes whether the claude-opus-4-7 prefix binds a longer digit run ('70').") },
726749
Q::Vector { id: "boundary-gpt-5-1234-openai-probe", provider: "openai", raw_model_id: "gpt-5-1234", note: Some("Probes a 4-digit run after the gpt-5 stem.") },
750+
Q::Section { group: "Databricks FQN GPT-5+ Responses routing", note: Some("Only the service component selects Responses; effort capabilities remain neutral.") },
751+
Q::Vector { id: "dbv2-fqn-responses-0", provider: "databricks_v2", raw_model_id: "catalog.schema.goose-gpt-6-astra", note: None },
752+
Q::Vector { id: "dbv2-fqn-responses-1", provider: "databricks_v2", raw_model_id: "catalog.schema.goose-gpt-5", note: None },
753+
Q::Vector { id: "dbv2-fqn-responses-2", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-5-5", note: None },
754+
Q::Vector { id: "dbv2-fqn-responses-3", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-10", note: None },
755+
Q::Vector { id: "dbv2-fqn-responses-4", provider: "databricks_v2", raw_model_id: "catalog.schema.GOOSE-GPT-6-ASTRA", note: None },
756+
Q::Vector { id: "dbv2-fqn-responses-5", provider: "databricks_v2", raw_model_id: "gpt-6.schema.other", note: None },
757+
Q::Vector { id: "dbv2-fqn-responses-6", provider: "databricks_v2", raw_model_id: "catalog.gpt-5.other", note: None },
758+
Q::Vector { id: "dbv2-fqn-responses-7", provider: "databricks_v2", raw_model_id: "catalog.schema.claude-gpt-6", note: None },
759+
Q::Vector { id: "dbv2-fqn-responses-8", provider: "databricks_v2", raw_model_id: "catalog.schema.my-gpt-6-astra", note: None },
760+
Q::Vector { id: "dbv2-fqn-responses-9", provider: "databricks_v2", raw_model_id: "catalog.schema.mygpt-6-astra", note: None },
761+
Q::Vector { id: "dbv2-fqn-responses-10", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-4", note: None },
762+
Q::Vector { id: "dbv2-fqn-responses-11", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-4o", note: None },
763+
Q::Vector { id: "dbv2-fqn-responses-12", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-6x", note: None },
764+
Q::Vector { id: "dbv2-fqn-responses-13", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-oss-120b", note: None },
765+
Q::Vector { id: "dbv2-fqn-responses-14", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-", note: None },
766+
Q::Vector { id: "dbv2-fqn-responses-15", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-4294967296", note: None },
767+
Q::Vector { id: "dbv2-fqn-responses-16", provider: "databricks_v2", raw_model_id: "catalog.schema.gpt-6", note: None },
768+
Q::Vector { id: "dbv2-fqn-responses-17", provider: "databricks_v2", raw_model_id: "catalog.schema.kimi-k3", note: None },
727769
Q::Section { group: "Databricks UC model-family humanization probes (#6918 follow-up)", note: Some("Exact-record and UC-FQN strip probes for the Gemini/DeepSeek/GLM/Grok/Llama/Qwen/Gemma/Inkling families surfaced by UC discovery.") },
728770
Q::Vector { id: "dbv2-gemini-3-1-flash-image-exact-record-probe", provider: "databricks_v2", raw_model_id: "databricks-gemini-3-1-flash-image", note: Some("Probes the Gemini 3.1 Flash Image endpoint record and label.") },
729771
Q::Vector { id: "dbv2-gemini-3-5-flash-exact-record-probe", provider: "databricks_v2", raw_model_id: "databricks-gemini-3-5-flash", note: Some("Probes the Gemini 3.5 Flash endpoint record and label.") },
@@ -844,7 +886,7 @@ mod tests {
844886
}
845887

846888
#[test]
847-
fn corpus_has_exactly_140_executable_vectors() {
889+
fn corpus_has_exactly_158_executable_vectors() {
848890
// Locks the vector count so a silent INPUTS edit can't quietly drop
849891
// coverage; must equal the gate in the TS harness
850892
// (modelCapabilitiesCorpus.test.mjs).
@@ -853,7 +895,7 @@ mod tests {
853895
.filter(|q| matches!(q, Q::Vector { .. }))
854896
.count();
855897
assert_eq!(
856-
vectors, 140,
898+
vectors, 158,
857899
"corpus executable-vector count changed; update this gate deliberately"
858900
);
859901
}
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
//! UC route selection must change only the route, never effort or model identity.
2+
use buzz_agent::model_capabilities::{resolve, DatabricksV2Route};
3+
4+
#[test]
5+
fn gpt_fqn_route_preserves_neutral_capabilities() {
6+
let fallback = resolve("databricks_v2", "catalog.schema.unknown");
7+
for model in [
8+
"catalog.schema.goose-gpt-6-astra",
9+
"catalog.schema.gpt-5",
10+
"catalog.schema.goose-gpt-5-5",
11+
"catalog.schema.gpt-10",
12+
"catalog.schema.GOOSE-GPT-6-ASTRA",
13+
"catalog.schema.team-gpt-6-astra",
14+
] {
15+
let got = resolve(" DATABRICKS-V2 ", model);
16+
assert_eq!(
17+
got.databricks_v2_wire_route,
18+
DatabricksV2Route::OpenaiResponses,
19+
"{model}"
20+
);
21+
assert_eq!(got.thinking_mode, fallback.thinking_mode, "{model}");
22+
assert_eq!(got.supported_efforts, fallback.supported_efforts, "{model}");
23+
assert_eq!(got.default_effort, fallback.default_effort, "{model}");
24+
assert_eq!(
25+
got.normalization_policy, fallback.normalization_policy,
26+
"{model}"
27+
);
28+
assert_eq!(got.registry_label, None, "{model}");
29+
}
30+
}
31+
32+
#[test]
33+
fn unrelated_fqns_do_not_select_responses() {
34+
for model in [
35+
"gpt-6.schema.other",
36+
"catalog.gpt-5.other",
37+
"catalog.schema.claude-gpt-6",
38+
"catalog.schema.mygpt-6-astra",
39+
"catalog.schema.gpt-4",
40+
"catalog.schema.gpt-4o",
41+
"catalog.schema.gpt-6x",
42+
"catalog.schema.gpt-oss-120b",
43+
"catalog.schema.gpt-",
44+
"catalog.schema.gpt-6",
45+
"catalog.schema.gpt-4294967296",
46+
"catalog.schema.kimi-k3",
47+
] {
48+
assert_eq!(
49+
resolve("databricks_v2", model).databricks_v2_wire_route,
50+
DatabricksV2Route::MlflowChat,
51+
"{model}"
52+
);
53+
}
54+
// The new rule is not an endpoint-family or cross-provider capability change.
55+
assert_eq!(
56+
resolve("databricks_v2", "goose-gpt-6-astra").databricks_v2_wire_route,
57+
DatabricksV2Route::MlflowChat
58+
);
59+
assert_eq!(
60+
resolve("openai", "catalog.schema.goose-gpt-6-astra").databricks_v2_wire_route,
61+
DatabricksV2Route::NotApplicable
62+
);
63+
}

‎desktop/src/features/agents/AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -306,7 +306,7 @@ with a TypeScript lookup table or an id comparison in a component.
306306
refresh only local persona/team/managed-agent caches; they must never
307307
invalidate the remote relay directory.
308308

309-
17. **Databricks model discovery has one shared catalog authority.** Desktop and ACP call the shared `buzz-agent` discovery library; Desktop passes the effective merged `DATABRICKS_MODEL_FILTER` explicitly, and the library applies it to raw workspace endpoint IDs and Unity Catalog model-service FQNs after the additive union. A successful filtered-empty catalog is authoritative: it stays empty, disables switching, and never falls through to configured or known-model fallback. UC FQNs are catalog data and always use the MLflow Chat Completions route, regardless of family-looking text in their components. Global Defaults preserves the discovered model ID as the selected value while its closed trigger renders the provider-scoped display label; do not force the raw persisted ID over that label.
309+
17. **Databricks model discovery has one shared catalog authority.** Desktop and ACP call the shared `buzz-agent` discovery library; Desktop passes the effective merged `DATABRICKS_MODEL_FILTER` explicitly, and the library applies it to raw workspace endpoint IDs and Unity Catalog model-service FQNs after the additive union. A successful filtered-empty catalog is authoritative: it stays empty, disables switching, and never falls through to configured or known-model fallback. UC FQNs retain neutral effort capabilities. A boundary-matched GPT-5-or-newer family in the service-name component selects OpenAI Responses so tools can coexist with reasoning; other FQNs use MLflow Chat Completions. Catalog/schema components never influence routing. Keep this route-only rule identical in the Rust and TypeScript capability interpreters. Global Defaults preserves the discovered model ID as the selected value while its closed trigger renders the provider-scoped display label; do not force the raw persisted ID over that label.
310310

311311
## Channel-only runtime controls
312312

‎desktop/src/features/agents/ui/modelCapabilities.ts‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,19 @@ function isDatabricksModelServiceFqn(model: string): boolean {
302302
);
303303
}
304304

305+
// Mirror fqn_requires_responses: routing is the only inferred FQN capability.
306+
function fqnRequiresResponses(model: string): boolean {
307+
const service = model.split(".").at(-1) ?? "";
308+
const stripped = stripCatalogPrefix(
309+
service.toLowerCase(),
310+
MANIFEST.family_tokens,
311+
);
312+
const match = /^gpt-([0-9]+)(?:$|[^a-z0-9])/.exec(stripped);
313+
if (!match) return false;
314+
const major = Number(match[1]);
315+
return major >= 5 && major <= 0xffffffff;
316+
}
317+
305318
/**
306319
* Resolve the capability profile for a `(provider, rawModelId)` pair.
307320
*
@@ -315,8 +328,8 @@ export function resolveModelCapabilities(
315328
): CapabilityResult {
316329
const canon = canonicalizeProvider(provider);
317330
const blank = rawModelId.trim().length === 0;
318-
// Unity Catalog FQNs are neutral model-service identities. Resolve them
319-
// through the concrete-unknown fallback before suffix family matching.
331+
// FQNs keep neutral effort capabilities; only GPT-5+ service names
332+
// select Responses. Catalog/schema names never choose the protocol.
320333
const modelServiceFqn =
321334
canon === "databricks_v2" && isDatabricksModelServiceFqn(rawModelId);
322335

@@ -370,7 +383,11 @@ export function resolveModelCapabilities(
370383
// 3. Provider fallback (blank vs. concrete-unknown); never carries a label.
371384
const pair = fallbackPair(canon);
372385
const state = blank ? pair.blank : pair.concrete_unknown;
373-
return toResult(state, state.databricks_v2_wire_route, null);
386+
const route =
387+
modelServiceFqn && fqnRequiresResponses(rawModelId)
388+
? "openai-responses"
389+
: state.databricks_v2_wire_route;
390+
return toResult(state, route, null);
374391
}
375392

376393
/** Authoritative list of known Databricks v2 model ids, sourced from the manifest. */

‎desktop/src/features/agents/ui/modelCapabilitiesCorpus.test.mjs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,10 @@ const corpus = JSON.parse(readFileSync(fileURLToPath(corpusUrl), "utf8"));
2525
// (`_group`) are skipped. Mirrors the Rust corpus filter.
2626
const executable = corpus.filter((entry) => entry.expect != null);
2727

28-
test("corpus has exactly 140 executable vectors", () => {
28+
test("corpus has exactly 158 executable vectors", () => {
2929
// Locks the vector count so a silent corpus edit can't quietly drop coverage;
3030
// must equal the gate in the Rust suite (model_capabilities.rs).
31-
assert.equal(executable.length, 140);
31+
assert.equal(executable.length, 158);
3232
});
3333

3434
test("registry label aliases refuse an unprefixed query", () => {

0 commit comments

Comments
 (0)