Skip to content

Commit 0cc63fe

Browse files
wesbillmanCarl
andauthored
fix(agent): preserve Databricks Opus UC reasoning and tool continuation (#7840)
## Summary Fix `reasoning_effort: Extra inputs are not permitted` for Databricks v2 model `data_workflow_tools.goose.goose-claude-opus-5-5` without dropping explicit High. - Add one provider-qualified exact capability record: Anthropic Messages, adaptive thinking, `output_config.effort`, medium upstream default. Mirror exact-record precedence in Rust/TS; keep uncurated UC services neutral and existing GPT routing unchanged. - Preserve ordered native content, including signed/empty thinking and redacted blocks, across tool results. Do not cache-stamp thinking blocks or leak native state into Chat reasoning details. Discard truncated native state; reject more than 64 native calls before any tool executes instead of altering signed content. - Add production-dispatch/ACP regressions and six shared capability vectors. No relay/ACP protocol, schema, settings, inheritance, retries, or other harness changes. ## Validation At final head `82f43f8d07bfcc3f4af492afc0a58a02f8e549c4`, clean tree: - Rebuilt real `buzz-agent`, drove ACP with a harmless MCP fixture against the actual service. Two POSTs to `/ai-gateway/anthropic/v1/messages`, both HTTP 200, exact FQN, High via `output_config`, adaptive/summarized thinking, no `reasoning_effort`. First response contained signed thinking + tool use; second request preserved signature/content order and paired result; final ACP stop `end_turn`. No credentials, signature contents, or conversation data included here. - Mandatory pre-commit/pre-push checks passed: formatting/lint, file-size gate, Rust test lanes, desktop typechecking and full JS tests, Tauri clippy/tests. - Full `cargo test -p buzz-agent` passed on the pre-commit tree (only hook formatting followed), including the new native truncation/65-call ACP regressions, ordered interleaved signed/redacted two-tool capture, nearby-ID negatives, and existing provider regressions. - Independent API/replay, regression-boundary, and Rust/TS capability reviews cleared. Full `just ci`, packaging and manual desktop UI testing were not run locally; repository CI remains the merge gate. Cross-harness internals and other UC service contracts are outside scope. ## References - [Databricks model services](https://docs.databricks.com/aws/en/ai-gateway/query-model-services) - [Anthropic thinking preservation](https://platform.claude.com/docs/en/build-with-claude/thinking#thinking-block-preservation-by-model) - [Anthropic effort](https://platform.claude.com/docs/en/build-with-claude/effort) - Originating Buzz channel: `fccdea96-0f31-4343-ba6a-324214dc8d73` - Originating thread: buzz://message?channel=fccdea96-0f31-4343-ba6a-324214dc8d73&id=33f242923a317e269c1ff7b960cacd6ec1f3371eff2f97592857804c50fb100c Signed-off-by: Wes <[email protected]> Signed-off-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz>
1 parent b6a2655 commit 0cc63fe

10 files changed

Lines changed: 444 additions & 19 deletions

File tree

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ use crate::auth::{PkceOAuthConfig, PkceOAuthTokenSource, StaticTokenSource, Toke
1010
use crate::config::{
1111
is_openai_host, normalize_effort_for_anthropic_route, normalize_effort_for_databricks_v2,
1212
normalize_effort_for_provider, Config, OpenAiApi, Provider, ThinkingEffort,
13+
MAX_TOOL_CALLS_PER_TURN,
1314
};
1415
use crate::types::{
1516
AgentError, HistoryItem, LlmResponse, ProviderStop, ToolCall, ToolDef, ToolResultContent,
@@ -588,9 +589,21 @@ fn anthropic_body(
588589
HistoryItem::Assistant {
589590
text,
590591
tool_calls,
591-
reasoning_details: _,
592+
reasoning_details,
592593
} => {
593594
flush(&mut messages, &mut pending);
595+
// Native Anthropic blocks carry signed, potentially interleaved
596+
// thinking. Replay the original order, not flattened text/calls.
597+
if let Some(blocks) = reasoning_details
598+
.as_ref()
599+
.and_then(|v| v.get("anthropic_content"))
600+
.and_then(Value::as_array)
601+
{
602+
if !blocks.is_empty() {
603+
messages.push(json!({ "role": "assistant", "content": blocks }));
604+
}
605+
continue;
606+
}
594607
let mut content: Vec<Value> = Vec::new();
595608
if !text.is_empty() {
596609
content.push(json!({ "type": "text", "text": text }));
@@ -688,6 +701,9 @@ fn stamp_rolling_cache_breakpoint(messages: &mut [Value]) {
688701
.get_mut("content")
689702
.and_then(Value::as_array_mut)
690703
.and_then(|c| c.last_mut())
704+
// Thinking blocks must remain byte-for-byte unchanged, and cannot
705+
// carry an explicit cache breakpoint.
706+
.filter(|b| !matches!(b["type"].as_str(), Some("thinking" | "redacted_thinking")))
691707
.and_then(Value::as_object_mut)
692708
{
693709
block.insert("cache_control".into(), json!({ "type": "ephemeral" }));
@@ -746,7 +762,9 @@ fn openai_body(
746762
let mut msg = serde_json::Map::new();
747763
msg.insert("role".into(), json!("assistant"));
748764
msg.insert("content".into(), json!(text.as_str()));
749-
if let Some(details) = reasoning_details {
765+
// OpenRouter owns the array shape; native Anthropic state must
766+
// not leak into Chat requests after a session model switch.
767+
if let Some(details) = reasoning_details.as_ref().filter(|v| v.is_array()) {
750768
msg.insert("reasoning_details".into(), details.clone());
751769
}
752770
if !tool_calls.is_empty() {
@@ -1386,6 +1404,14 @@ fn parse_anthropic(v: Value) -> Result<LlmResponse, AgentError> {
13861404
}
13871405
}
13881406
}
1407+
// The run loop caps other providers' calls by truncating them. Native
1408+
// content must be replayed intact, so reject oversized turns before any
1409+
// tools execute rather than silently altering signed/interleaved state.
1410+
if tool_calls.len() > MAX_TOOL_CALLS_PER_TURN {
1411+
return Err(AgentError::Llm(format!(
1412+
"Anthropic response exceeds {MAX_TOOL_CALLS_PER_TURN} tool calls; cannot truncate native content"
1413+
)));
1414+
}
13891415
// anthropic_input_tokens() returns Option<SumUsageResult> because it sums
13901416
// three fields that can collectively overflow u64. Propagate the overflow
13911417
// signal via `input_tokens_overflowed` so the run loop can poison the
@@ -1415,7 +1441,13 @@ fn parse_anthropic(v: Value) -> Result<LlmResponse, AgentError> {
14151441
// total from them. Always None for this provider.
14161442
total_tokens: None,
14171443
reasoning,
1418-
reasoning_details: None,
1444+
// Truncated content is not a valid signed assistant turn; the run loop
1445+
// keeps only its text and never executes its tool calls.
1446+
reasoning_details: v
1447+
.get("content")
1448+
.filter(|v| v.is_array())
1449+
.filter(|_| stop != ProviderStop::MaxTokens)
1450+
.map(|blocks| json!({ "anthropic_content": blocks })),
14191451
// Stamped by the dispatch layer (complete) after parse.
14201452
request_model: None,
14211453
})

‎crates/buzz-agent/src/llm_fqn_tests.rs‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,112 @@ async fn gpt_fqn_completion_and_summary_use_responses() {
5454
assert_eq!(completion["tools"][0]["name"], "test_tool");
5555
assert_eq!(posts[1].body.as_ref().unwrap()["max_output_tokens"], 128);
5656
}
57+
58+
const OPUS_UC: &str = "data_workflow_tools.goose.goose-claude-opus-5-5";
59+
60+
#[tokio::test]
61+
async fn opus_uc_replays_ordered_signed_content_with_high_effort() {
62+
// Empty thinking can still carry a signature; redacted data is opaque.
63+
let blocks = json!([
64+
{"type":"thinking", "thinking":"", "signature":"signed-empty"},
65+
{"type":"tool_use", "id":"a", "name":"test_tool", "input":{"n":1}},
66+
{"type":"redacted_thinking", "data":"opaque-redacted"},
67+
{"type":"tool_use", "id":"b", "name":"test_tool", "input":{"n":2}},
68+
{"type":"thinking", "thinking":"checking", "signature":"signed-check"},
69+
{"type":"text", "text":"running both"}
70+
]);
71+
let done = json!({"stop_reason":"end_turn", "content":[{"type":"text","text":"done"}]});
72+
let (base_url, captured) = spawn_sequence_stub(vec![
73+
StubHttpResponse::ok(json!({"stop_reason":"tool_use", "content":blocks})),
74+
StubHttpResponse::ok(done.clone()),
75+
StubHttpResponse::ok(done),
76+
]).await;
77+
let mut config = cfg(Provider::DatabricksV2);
78+
config.base_url = base_url;
79+
config.thinking_effort = Some(ThinkingEffort::High);
80+
config.prompt_caching = true;
81+
let llm = Llm::new(&config).unwrap();
82+
let tools = vec![ToolDef { name:"test_tool".into(), description:"Test".into(),
83+
input_schema:json!({"type":"object", "properties":{"n":{"type":"integer"}}}) }];
84+
let mut history = vec![HistoryItem::User("run both".into())];
85+
let first = llm.complete(&config, "system", &history, &tools, OPUS_UC).await.unwrap();
86+
assert_eq!(first.stop, ProviderStop::ToolUse);
87+
assert_eq!(first.reasoning, "checking");
88+
assert_eq!(first.tool_calls.len(), 2);
89+
assert_eq!(first.tool_calls[1].arguments, json!({"n":2}));
90+
history.push(HistoryItem::Assistant { text:first.text, tool_calls:first.tool_calls,
91+
reasoning_details:first.reasoning_details });
92+
for id in ["a", "b"] {
93+
history.push(HistoryItem::ToolResult(crate::types::ToolResult {
94+
provider_id:id.into(), content:vec![ToolResultContent::Text(format!("result-{id}"))], is_error:false,
95+
}));
96+
}
97+
let second = llm.complete(&config, "system", &history, &tools, OPUS_UC).await.unwrap();
98+
assert_eq!(second.stop, ProviderStop::EndTurn);
99+
assert_eq!(second.text, "done");
100+
assert_eq!(llm.summarize(&config, "system", "history", 128, OPUS_UC).await.unwrap(), "done");
101+
let requests = captured.lock().await;
102+
let posts: Vec<_> = requests.iter().filter(|r| r.method == "POST").collect();
103+
assert_eq!(posts.len(), 3);
104+
for post in &posts {
105+
assert_eq!(post.path, "/v1/ai-gateway/anthropic/v1/messages");
106+
assert_eq!(post.body.as_ref().unwrap()["model"], OPUS_UC);
107+
assert!(post.body.as_ref().unwrap().get("reasoning_effort").is_none());
108+
}
109+
for post in &posts[..2] {
110+
let body = post.body.as_ref().unwrap();
111+
assert_eq!(body["output_config"], json!({"effort":"high"}));
112+
assert_eq!(body["thinking"], json!({"type":"adaptive", "display":"summarized"}));
113+
assert_eq!(body["tools"][0]["input_schema"], tools[0].input_schema);
114+
}
115+
let body = posts[1].body.as_ref().unwrap();
116+
let mut replay = body["messages"][1]["content"].clone();
117+
assert_eq!(replay[5]["cache_control"], json!({"type":"ephemeral"}));
118+
replay[5].as_object_mut().unwrap().remove("cache_control");
119+
assert_eq!(replay, blocks);
120+
let results = body["messages"][2]["content"].as_array().unwrap();
121+
assert_eq!(results.len(), 2);
122+
for (result, id) in results.iter().zip(["a", "b"]) {
123+
assert_eq!(result["type"], "tool_result");
124+
assert_eq!(result["tool_use_id"], id);
125+
assert_eq!(result["content"][0]["text"], format!("result-{id}"));
126+
}
127+
}
128+
129+
#[tokio::test]
130+
async fn neighboring_opus_uc_services_stay_on_mlflow_chat() {
131+
for model in ["system.ai.claude-opus-5-5", "other.goose.goose-claude-opus-5-5",
132+
"data_workflow_tools.goose.goose-claude-opus-5-5-preview"] {
133+
let (base_url, captured) = spawn_sequence_stub(vec![StubHttpResponse::ok(json!({
134+
"choices":[{"finish_reason":"stop", "message":{"content":"ok"}}]
135+
}))]).await;
136+
let mut config = cfg(Provider::DatabricksV2);
137+
config.base_url = base_url;
138+
config.thinking_effort = Some(ThinkingEffort::High);
139+
Llm::new(&config).unwrap().complete(&config, "system", &[HistoryItem::User("hi".into())], &[], model).await.unwrap();
140+
let requests = captured.lock().await;
141+
let post = requests.iter().find(|r| r.method == "POST").unwrap();
142+
assert_eq!(post.path, "/v1/ai-gateway/mlflow/v1/chat/completions");
143+
let body = post.body.as_ref().unwrap();
144+
assert_eq!(body["model"], model);
145+
assert_eq!(body["reasoning_effort"], "high");
146+
assert!(body.get("thinking").is_none());
147+
}
148+
}
149+
150+
#[test]
151+
fn native_thinking_tail_is_not_cache_stamped_or_leaked_to_chat() {
152+
for block in [json!({"type":"thinking", "thinking":"", "signature":"signed"}),
153+
json!({"type":"redacted_thinking", "data":"opaque"})] {
154+
let response = parse_anthropic(json!({"stop_reason":"end_turn", "content":[block]})).unwrap();
155+
let history = vec![HistoryItem::Assistant { text:response.text, tool_calls:response.tool_calls,
156+
reasoning_details:response.reasoning_details }];
157+
let mut config = cfg(Provider::DatabricksV2);
158+
config.prompt_caching = true;
159+
let native = anthropic_body(&config, "system", &history, &[], OPUS_UC, Some(ThinkingEffort::High), "databricks_v2");
160+
assert_eq!(native["messages"][0]["content"], json!([block]));
161+
let chat = openai_body(&config, "system", &history, &[], "other", None);
162+
assert!(chat["messages"][1].get("reasoning_details").is_none());
163+
assert!(!responses_body(&config, "system", &history, &[], "other", None).to_string().contains("anthropic_content"));
164+
}
165+
}

‎crates/buzz-agent/src/model_capabilities.rs‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,8 @@ fn prefix_matches(token: &str, s: &str) -> bool {
293293
/// Databricks Unity Catalog model-service names are catalog data, not model
294294
/// family hints. Both capability interpreters use this shape check before
295295
/// family matching so services cannot inherit endpoint capabilities accidentally.
296-
/// GPT-5+ services have a route-only Responses exception.
296+
/// Verified exact records may supply capabilities; other GPT-5+ services have
297+
/// a route-only Responses exception.
297298
pub(crate) fn is_databricks_model_service_fqn(model: &str) -> bool {
298299
let mut components = model.split('.');
299300
let (Some(catalog), Some(schema), Some(service)) =
@@ -334,14 +335,14 @@ pub fn resolve(provider: &str, raw_model_id: &str) -> CapabilityResult {
334335
let canon = canonical_provider(provider);
335336
let blank = raw_model_id.trim().is_empty();
336337

337-
// FQNs keep neutral effort capabilities, but GPT-5+ service names need
338-
// Responses for tools with reasoning. Only inspect the service component:
339-
// catalog/schema names must never choose a model protocol.
338+
// Uncurated FQNs keep neutral effort capabilities. Exact records are verified
339+
// service contracts, not family-name inference. The GPT-5+ route-only fallback
340+
// inspects just the service component, never catalog/schema names.
340341
let model_service_fqn =
341342
canon == "databricks_v2" && is_databricks_model_service_fqn(raw_model_id);
342343

343344
// 1. Provider-qualified exact-record lookup (case-insensitive on the id).
344-
if !blank && !model_service_fqn {
345+
if !blank {
345346
for rec in &m.exact_records {
346347
if rec.provider == canon && rec.raw_model_id.eq_ignore_ascii_case(raw_model_id) {
347348
return CapabilityResult {
@@ -747,6 +748,13 @@ mod tests {
747748
Q::Vector { id: "boundary-claude-3-digit-run-anthropic-probe", provider: "anthropic", raw_model_id: "claude-35", note: Some("Probes whether the claude-3 prefix binds a longer digit run ('35').") },
748749
Q::Vector { id: "boundary-claude-opus-4-70-anthropic-probe", provider: "anthropic", raw_model_id: "claude-opus-4-70", note: Some("Probes whether the claude-opus-4-7 prefix binds a longer digit run ('70').") },
749750
Q::Vector { id: "boundary-gpt-5-1234-openai-probe", provider: "openai", raw_model_id: "gpt-5-1234", note: Some("Probes a 4-digit run after the gpt-5 stem.") },
751+
Q::Section { group: "Verified Databricks Opus UC service and exact-record boundaries", note: None },
752+
Q::Vector { id: "dbv2-opus-uc-exact", provider: "databricks_v2", raw_model_id: "data_workflow_tools.goose.goose-claude-opus-5-5", note: None },
753+
Q::Vector { id: "dbv2-opus-uc-case", provider: "databricks_v2", raw_model_id: "DATA_WORKFLOW_TOOLS.GOOSE.GOOSE-CLAUDE-OPUS-5-5", note: None },
754+
Q::Vector { id: "dbv2-opus-uc-namespace", provider: "databricks_v2", raw_model_id: "other.goose.goose-claude-opus-5-5", note: None },
755+
Q::Vector { id: "dbv2-opus-uc-service", provider: "databricks_v2", raw_model_id: "data_workflow_tools.goose.goose-claude-opus-5-5-preview", note: None },
756+
Q::Vector { id: "dbv2-opus-uc-system", provider: "databricks_v2", raw_model_id: "system.ai.claude-opus-5-5", note: None },
757+
Q::Vector { id: "dbv2-opus-uc-provider", provider: "openai", raw_model_id: "data_workflow_tools.goose.goose-claude-opus-5-5", note: None },
750758
Q::Section { group: "Databricks FQN GPT-5+ Responses routing", note: Some("Only the service component selects Responses; effort capabilities remain neutral.") },
751759
Q::Vector { id: "dbv2-fqn-responses-0", provider: "databricks_v2", raw_model_id: "catalog.schema.goose-gpt-6-astra", note: None },
752760
Q::Vector { id: "dbv2-fqn-responses-1", provider: "databricks_v2", raw_model_id: "catalog.schema.goose-gpt-5", note: None },
@@ -886,7 +894,7 @@ mod tests {
886894
}
887895

888896
#[test]
889-
fn corpus_has_exactly_158_executable_vectors() {
897+
fn corpus_has_exactly_164_executable_vectors() {
890898
// Locks the vector count so a silent INPUTS edit can't quietly drop
891899
// coverage; must equal the gate in the TS harness
892900
// (modelCapabilitiesCorpus.test.mjs).
@@ -895,7 +903,7 @@ mod tests {
895903
.filter(|q| matches!(q, Q::Vector { .. }))
896904
.count();
897905
assert_eq!(
898-
vectors, 158,
906+
vectors, 164,
899907
"corpus executable-vector count changed; update this gate deliberately"
900908
);
901909
}

‎crates/buzz-agent/src/types.rs‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -215,9 +215,10 @@ pub struct LlmResponse {
215215
///
216216
/// Empty string when the provider returned no reasoning content.
217217
pub reasoning: String,
218-
/// Raw `reasoning_details` array from an OpenRouter response, if present.
219-
/// Replayed on subsequent turns so the model can continue its chain-of-thought.
220-
/// `None` for all non-OpenRouter providers.
218+
/// Provider-owned replay state: OpenRouter's raw `reasoning_details` array,
219+
/// or an `anthropic_content` object holding ordered native content blocks.
220+
/// Kept opaque so signed thinking survives tool-result continuation. Each
221+
/// serializer consumes only its own shape, including after model switches.
221222
pub reasoning_details: Option<Value>,
222223
/// The model id that was actually sent in the request body — the
223224
/// actually-requested model after any auto/mesh resolution. Populated by

0 commit comments

Comments
 (0)