Skip to content

Commit d01e5f8

Browse files
wpfleger96Duncan
andauthored
feat(agents): humanize uncurated Databricks model ids with a label grammar (#7844)
New Databricks endpoints (Claude Opus 5.5, GPT-6 Astra/Sol, etc.) showed as raw FQNs in the model picker until someone added a curated corpus record for each. This adds a strict token-class grammar as a third label tier inside the Databricks label helpers (Rust `model_capabilities.rs` + TS `modelCapabilities.ts`): exact record → unique alias → **generated label** → raw id. In TS it fires only for an explicit `databricks_v2` provider; providerless ids stay raw. ## Grammar Takes the last component of a UC FQN (or strips one wrapper: `databricks-`, `goose-`, `kgoose-`, `builderbot-`; an id that is neither stays raw), splits on `-`, and classifies each token as version, letter+digit version, size, trailing date (dropped), or word. Labels are built only from the id's own text, and the grammar never reinterprets a version. When in doubt it returns raw: - **Known families only.** The family (leading letters of the first token) must be in the manifest's existing `label_family_tokens`. Custom endpoints such as `prod.ai.rag-agent-3` or `builderbot-pr-reviews-v2` stay raw. Supporting a new vendor means adding one entry to that list, not a record per model. - **Attached digits stay whole** (`gpt10` → `GPT10`). If the family already carries digits and the next token is a version (`goose-qwen3-5-122b-a10b`, `system.ai.llama3-1-8b`, `system.ai.claude3-5-sonnet`), the id stays raw; this is checked before any reorder. - A third consecutive number (`gemini-3-7-1-flash`) returns raw rather than splitting off a patch version. - Any non-ASCII input returns raw, checked before trimming or case folding. - Any unclassifiable token, or no version-bearing token (`builderbot-pr-reviews`), returns raw. `(latest)`/`Preview` never appear unless an exact record supplies them. Special cases, each justified by an unseen-id shape rather than the corpus score: - `meta-llama-` → `Llama`: a namespace convention. - Claude numeric-first reorder: Goose names Claude endpoints `claude-4-7-opus`. Only 1–2 version tokens followed by `opus`/`sonnet`/`haiku` reorder (`claude-5-6-opus` → `Claude Opus 5.6`); anything else keeps its order (`claude-5-6-thinking` → `Claude 5.6 Thinking`). Both current numeric-first ids also have exact records. - Qwen compact decimal: Qwen's own naming `qwen35` → `Qwen3.5` when the second digit is nonzero. ## Colliding labels in the picker The same model is often served from several sources, so humanized labels collide (four rows are all `GPT-5.5`). `disambiguateModelLabels` in `formatAgentModelLabel.ts` suffixes only colliding Databricks labels with their source: - UC FQN: its `catalog.schema`, e.g. `GPT-6 Astra (system.ai)` / `GPT-6 Astra (sandbox.public)`. - Workspace endpoint: its wrapper, e.g. `GPT-5.5 (databricks)` / `GPT-5.5 (goose)` / `GPT-5.5 (kgoose)`. - Still colliding after that (e.g. a date variant in the same schema): the full id, e.g. `Gemini 3.7 Flash (system.ai.gemini-3-7-flash-0925)`. Unique labels, raw-id rows, the default row, and other providers are unchanged. It runs in `getDiscoveredPersonaModelOptions` (covering `AgentDropdownSelect`, `PersonaModelCombobox`, `PersonaDropdownField`, whose closed triggers read the option label) and in `ModelPicker`. Model search in `AgentDropdownSelect` and `PersonaModelCombobox` also matches the raw id, so searching `system.ai` finds those rows. Rust `ModelEntry.name` is unchanged. ## What labels drive Picker values, persistence, routing, catalog dedup, `resolve()` and capabilities all stay keyed on the raw id; two ids sharing a label both remain selectable. Labels do drive picker search (alongside the raw id), the workspace catalog's equal-time sort, and `TeamIdentityCard`'s "Mixed models" summary, which compares lowercased base labels, so `goose-` and `system.ai` GPT-6 Astra read as one model there (correct, since they are the same model). ## Tests Shared fixtures in `scripts/databricks-label-fixtures.json` are replayed by both Rust and TS. Each row asserts its tier: `exact` rows match a record, `alias` rows resolve with generation off, `generated` rows return nothing with generation off (so a future alias or record cannot mask them), and `raw` rows return nothing either way. A curator guard requires that across all 57 exact records, the ids the grammar does not reproduce equal exactly the 10 curator-only ids. `disambiguateModelLabels` has unit tests for namespace, wrapper, and full-id-fallback collisions, untouched rows, other providers, and case-insensitive grouping. The defaults-picker e2e checks the suffixed rows and trigger, raw-id search, and that choosing the second uncurated FQN by its visible name persists its raw id and restores its label. An edit-agent e2e covers the existing-agent model dropdown (`#edit-agent-model`): both suffixed rows render, the `system.ai` row is chosen by name, and after save and reopen the agent stores the `system.ai` id and shows `GPT-6 Astra (system.ai)`. --------- Signed-off-by: Will Pfleger <[email protected]> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
1 parent 4888484 commit d01e5f8

16 files changed

Lines changed: 998 additions & 60 deletions

‎crates/buzz-agent/src/catalog.rs‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,9 @@ use crate::{
2727
/// A discovered model entry: `id` is the picker value (the raw endpoint id or
2828
/// Unity Catalog model-service FQN, and the wire/config value), `name` is the
2929
/// display label. Databricks catalog APIs do not provide a consistently useful
30-
/// picker label, so discovery curates names from the capability manifest when
31-
/// an exact known id exists and otherwise uses the raw id.
30+
/// picker label, so discovery derives `name` as exact record → unique alias →
31+
/// generated label grammar (known `label_family_tokens` families only) → raw
32+
/// id. Adding a vendor means adding one `label_family_tokens` entry.
3233
#[derive(Debug, Clone, PartialEq, Eq)]
3334
pub struct ModelEntry {
3435
pub id: String,
@@ -80,12 +81,10 @@ const UNITY_CATALOG_DESCRIPTOR: CatalogDescriptor<ModelEntry> = CatalogDescripto
8081
parse_page: parse_uc_model_services_page,
8182
};
8283

83-
/// Curated display label for a discovered Databricks endpoint or model-service
84-
/// id. Unknown ids deliberately pass through unchanged.
84+
/// Display label for a discovered Databricks endpoint or model-service id. Ids
85+
/// the label grammar cannot fully parse deliberately pass through unchanged.
8586
fn curated_model_name(id: &str) -> String {
86-
crate::model_capabilities::databricks_registry_label(id)
87-
.unwrap_or(id)
88-
.to_string()
87+
crate::model_capabilities::databricks_registry_label(id).unwrap_or_else(|| id.to_string())
8988
}
9089

9190
/// Fallback catalog used only when both authenticated Databricks v2 catalogs
@@ -1903,7 +1902,7 @@ mod tests {
19031902
// `name` is the curated label + provenance suffix, not the raw id.
19041903
assert!(models.iter().all(|model| {
19051904
let label = crate::model_capabilities::databricks_registry_label(&model.id)
1906-
.unwrap_or(model.id.as_str());
1905+
.unwrap_or_else(|| model.id.clone());
19071906
model.name == format!("{label}{AUTHENTICATED_EMPTY_CATALOG_SUFFIX}")
19081907
}));
19091908
}
Lines changed: 281 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,281 @@
1+
//! Deterministic display-label grammar for Databricks endpoint ids.
2+
//!
3+
//! The last tier of [`crate::model_capabilities::databricks_registry_label`],
4+
//! reached only after an exact-record miss and a unique-alias miss. The result
5+
//! is either a label built solely from the id's own tokens or `None`, in which
6+
//! case callers show the raw id. It is presentation-only: capabilities, wire
7+
//! routing, and the saved model id never depend on it. Ids from families not
8+
//! in the manifest's `label_family_tokens` stay raw; supporting a new vendor
9+
//! means adding one entry to that list, not a record per model.
10+
//!
11+
//! Mirrored by `desktop/src/features/agents/ui/modelCapabilities.ts`; both
12+
//! replay `scripts/databricks-label-fixtures.json`.
13+
14+
use crate::model_capabilities::is_databricks_model_service_fqn;
15+
16+
/// Endpoint-name wrappers that precede the model name in Databricks ids.
17+
const WRAPPERS: [&str; 4] = ["databricks-", "goose-", "kgoose-", "builderbot-"];
18+
19+
enum Part {
20+
/// Numeric version (`5`, `5.5`) — the one part GPT/GLM join with a hyphen.
21+
Version(String),
22+
Word(String),
23+
}
24+
25+
/// Parse a Databricks endpoint id into a display label, or `None` when any part
26+
/// of the id falls outside the grammar. Only families listed in the manifest's
27+
/// `label_family_tokens` are named, so custom endpoints (`rag-agent-3`) stay raw.
28+
pub(crate) fn generate_databricks_label(
29+
raw_model_id: &str,
30+
family_tokens: &[String],
31+
) -> Option<String> {
32+
// Refuse non-ASCII before trimming or folding, so no Unicode case or
33+
// whitespace rule can turn an unsupported id into an ASCII-looking one.
34+
if !raw_model_id.is_ascii() {
35+
return None;
36+
}
37+
let id = raw_model_id.trim().to_ascii_lowercase();
38+
let is_fqn = is_databricks_model_service_fqn(&id);
39+
let service = if is_fqn {
40+
id.rsplit('.').next()?
41+
} else {
42+
id.as_str()
43+
};
44+
// Only a UC FQN or a wrapped endpoint name is attributable to Databricks; a
45+
// bare id such as `gpt-5` stays raw.
46+
let body = match WRAPPERS.iter().find_map(|w| service.strip_prefix(w)) {
47+
Some(body) => body,
48+
None if is_fqn => service,
49+
None => return None,
50+
};
51+
let body = body
52+
.strip_prefix("meta-")
53+
.filter(|rest| rest.starts_with("llama-"))
54+
.unwrap_or(body);
55+
56+
let mut tokens = body.split('-');
57+
let (family, stem) = split_family(tokens.next()?)?;
58+
if !family_tokens
59+
.iter()
60+
.any(|t| t.strip_suffix('-').unwrap_or(t) == family)
61+
{
62+
return None;
63+
}
64+
let mut rest: Vec<&str> = tokens.collect();
65+
// Attached digits followed by a version (`qwen3-5`, `llama3-1`) cannot be
66+
// read without guessing; checked before any reorder.
67+
if stem.is_some() && rest.first().is_some_and(|t| is_version(t)) {
68+
return None;
69+
}
70+
if family == "claude" {
71+
// Goose's numeric-first Claude ids (`claude-4-7-opus`) name the tier
72+
// after the version. Reorder only that exact shape; any other token
73+
// after the version keeps its position.
74+
let digits = rest.iter().take_while(|t| is_version(t)).count();
75+
if (1..=2).contains(&digits)
76+
&& rest
77+
.get(digits)
78+
.is_some_and(|t| matches!(*t, "opus" | "sonnet" | "haiku"))
79+
{
80+
rest[..=digits].rotate_right(1);
81+
}
82+
}
83+
84+
let mut parts = Vec::new();
85+
let mut has_number = stem.is_some();
86+
let mut after_minor = false;
87+
let mut rest = rest.into_iter().peekable();
88+
while let Some(tok) = rest.next() {
89+
if is_date(tok) && rest.peek().is_none() {
90+
break;
91+
}
92+
let part = if is_version(tok) {
93+
let minor = rest.next_if(|t| is_version(t));
94+
after_minor = minor.is_some();
95+
if after_minor && rest.peek().is_some_and(|t| is_version(t)) {
96+
// A third number (`3-7-1`) would read as a separate version.
97+
return None;
98+
}
99+
Part::Version(minor.map_or_else(|| tok.to_string(), |m| format!("{tok}.{m}")))
100+
} else if is_letter_version(tok) {
101+
let minor = rest.next_if(|t| is_version(t));
102+
if minor.is_some() && rest.peek().is_some_and(|t| is_version(t)) {
103+
return None;
104+
}
105+
let major = capitalize(tok);
106+
Part::Word(minor.map_or_else(|| major.clone(), |m| format!("{major}.{m}")))
107+
} else if is_size(tok) {
108+
Part::Word(tok.to_ascii_uppercase())
109+
} else if !tok.is_empty() && tok.bytes().all(|b| b.is_ascii_lowercase()) {
110+
Part::Word(match tok {
111+
"oss" => "OSS".to_string(),
112+
"mini" | "nano" if family == "gpt" && after_minor => tok.to_string(),
113+
_ => capitalize(tok),
114+
})
115+
} else {
116+
return None;
117+
};
118+
has_number |= match &part {
119+
Part::Version(_) => true,
120+
Part::Word(word) => word.bytes().any(|b| b.is_ascii_digit()),
121+
};
122+
parts.push(part);
123+
}
124+
// A versionless id (`builderbot-pr-reviews`) is a named endpoint, not a model.
125+
if !has_number {
126+
return None;
127+
}
128+
129+
let (brand, hyphenated) = match family {
130+
"gpt" => ("GPT".to_string(), true),
131+
"glm" => ("GLM".to_string(), true),
132+
"deepseek" => ("DeepSeek".to_string(), false),
133+
_ => (capitalize(family), false),
134+
};
135+
let mut label = brand + &stem.unwrap_or_default();
136+
for (index, part) in parts.iter().enumerate() {
137+
let (sep, text) = match part {
138+
Part::Version(v) if index == 0 && hyphenated => ('-', v),
139+
Part::Version(text) | Part::Word(text) => (' ', text),
140+
};
141+
label.push(sep);
142+
label.push_str(text);
143+
}
144+
Some(label)
145+
}
146+
147+
/// Split the leading token into its family name and an optional in-name
148+
/// version, kept whole (`nova12` → `12`). The one exception is Qwen's own
149+
/// compact-decimal naming: `qwen35` → `3.5` when the second digit is nonzero.
150+
fn split_family(tok: &str) -> Option<(&str, Option<String>)> {
151+
let alpha = tok.bytes().take_while(u8::is_ascii_lowercase).count();
152+
let (family, digits) = tok.split_at(alpha);
153+
if family.is_empty() || digits.len() > 2 || !digits.bytes().all(|b| b.is_ascii_digit()) {
154+
return None;
155+
}
156+
let stem = match digits.as_bytes() {
157+
[] => None,
158+
[major, minor] if family == "qwen" && *minor != b'0' => {
159+
Some(format!("{}.{}", *major as char, *minor as char))
160+
}
161+
_ => Some(digits.to_string()),
162+
};
163+
Some((family, stem))
164+
}
165+
166+
/// `0` or a 1–2-digit number without a leading zero.
167+
fn is_version(tok: &str) -> bool {
168+
match tok.as_bytes() {
169+
[d] => d.is_ascii_digit(),
170+
[a, b] => (b'1'..=b'9').contains(a) && b.is_ascii_digit(),
171+
_ => false,
172+
}
173+
}
174+
175+
/// A 4- or 8-digit date stamp (`0731`, `20260101`).
176+
fn is_date(tok: &str) -> bool {
177+
matches!(tok.len(), 4 | 8) && tok.bytes().all(|b| b.is_ascii_digit())
178+
}
179+
180+
/// A letter-prefixed version such as `v4` or `k2`.
181+
fn is_letter_version(tok: &str) -> bool {
182+
matches!(tok.as_bytes(), [l, digits @ ..] if l.is_ascii_lowercase()
183+
&& !digits.is_empty()
184+
&& digits.iter().all(u8::is_ascii_digit))
185+
}
186+
187+
/// A parameter-size token: `120b`, or letter-prefixed `a3b`.
188+
fn is_size(tok: &str) -> bool {
189+
let Some(body) = tok.strip_suffix('b') else {
190+
return false;
191+
};
192+
let digits = body
193+
.strip_prefix(|c: char| c.is_ascii_lowercase())
194+
.unwrap_or(body);
195+
!digits.is_empty() && digits.bytes().all(|b| b.is_ascii_digit())
196+
}
197+
198+
fn capitalize(word: &str) -> String {
199+
let mut chars = word.chars();
200+
chars.next().map_or_else(String::new, |first| {
201+
first.to_ascii_uppercase().to_string() + chars.as_str()
202+
})
203+
}
204+
205+
#[cfg(test)]
206+
mod tests {
207+
use crate::model_capabilities::{databricks_curated_label, databricks_registry_label};
208+
use serde_json::Value;
209+
210+
const FIXTURES_JSON: &str = include_str!("../../../scripts/databricks-label-fixtures.json");
211+
const MANIFEST_JSON: &str = include_str!("../../../scripts/model-capabilities.json");
212+
213+
fn generate(id: &str) -> Option<String> {
214+
let family_tokens: Vec<String> =
215+
serde_json::from_value(json(MANIFEST_JSON)["label_family_tokens"].clone())
216+
.expect("label_family_tokens");
217+
super::generate_databricks_label(id, &family_tokens)
218+
}
219+
220+
fn json(source: &str) -> Value {
221+
serde_json::from_str(source).expect("fixture JSON must parse")
222+
}
223+
224+
/// `(raw_model_id, registry_label)` for every `databricks_v2` exact record.
225+
fn exact_records() -> Vec<(String, String)> {
226+
json(MANIFEST_JSON)["exact_records"]
227+
.as_array()
228+
.expect("exact_records")
229+
.iter()
230+
.filter(|rec| rec["provider"] == "databricks_v2")
231+
.map(|rec| {
232+
let field = |key: &str| rec[key].as_str().expect(key).to_string();
233+
(field("raw_model_id"), field("registry_label"))
234+
})
235+
.collect()
236+
}
237+
238+
#[test]
239+
fn shared_fixtures_replay_through_the_label_helper() {
240+
let fixtures = json(FIXTURES_JSON);
241+
let records = exact_records();
242+
for row in fixtures["lookup"].as_array().expect("lookup") {
243+
let id = row["id"].as_str().expect("id");
244+
let label = row["label"].as_str();
245+
assert_eq!(databricks_registry_label(id).as_deref(), label, "id={id:?}");
246+
let has_exact = records.iter().any(|(raw, _)| raw.eq_ignore_ascii_case(id));
247+
let curated = databricks_curated_label(id);
248+
let tier = row["tier"].as_str().expect("tier");
249+
assert_eq!(has_exact, tier == "exact", "tier={tier} id={id:?}");
250+
match tier {
251+
"exact" | "alias" => assert_eq!(curated.as_deref(), label, "id={id:?}"),
252+
"generated" => {
253+
assert_eq!(curated, None, "generated fixture is masked: {id:?}");
254+
assert_eq!(generate(id).as_deref(), label, "id={id:?}");
255+
}
256+
"raw" => {
257+
assert_eq!(label, None, "raw fixture has a label: {id:?}");
258+
assert_eq!(curated, None, "raw fixture is curated: {id:?}");
259+
}
260+
tier => panic!("unknown tier {tier:?} for {id}"),
261+
}
262+
}
263+
}
264+
265+
#[test]
266+
fn grammar_reproduces_every_curated_label_except_curator_only_ones() {
267+
let fixtures = json(FIXTURES_JSON);
268+
let curator_only: Vec<&str> = fixtures["curator_only_labels"]
269+
.as_array()
270+
.expect("curator_only_labels")
271+
.iter()
272+
.map(|id| id.as_str().expect("id"))
273+
.collect();
274+
let misses: Vec<String> = exact_records()
275+
.into_iter()
276+
.filter(|(raw, label)| generate(raw).as_deref() != Some(label))
277+
.map(|(raw, _)| raw)
278+
.collect();
279+
assert_eq!(misses, curator_only);
280+
}
281+
}

‎crates/buzz-agent/src/lib.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ mod builtin;
66
pub mod catalog;
77
pub mod config;
88
pub mod databricks;
9+
mod databricks_label_grammar;
910
mod handoff;
1011
mod hints;
1112
mod llm;
@@ -413,8 +414,7 @@ async fn resolve_models_catalog(
413414
fn configured_model_fallback(model: &str) -> Vec<ModelEntry> {
414415
let model = model.trim().to_string();
415416
let name = crate::model_capabilities::databricks_registry_label(&model)
416-
.unwrap_or(&model)
417-
.to_string();
417+
.unwrap_or_else(|| model.clone());
418418
vec![ModelEntry { id: model, name }]
419419
}
420420

0 commit comments

Comments
 (0)