Skip to content

Commit cc0c47c

Browse files
authored
Test provider connections before model selection (#500)
Signed-off-by: Salman Mohammed <[email protected]>
1 parent e5a7046 commit cc0c47c

10 files changed

Lines changed: 670 additions & 137 deletions

File tree

‎docs/agent-control.md‎

Lines changed: 37 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -507,22 +507,43 @@ Settings says **Shell setup not verified**; Buzz does not check it before Start.
507507
model browsing. A Goose catalog entry does not establish caller EXECUTE permission
508508
or successful inference. Advanced arguments remain a literal JSON array. Old native hosts
509509
without this metadata fall back to custom entry.
510-
- With a provider and model selected, **Test connection** runs one small Goose
511-
request in the agent workspace using the effective draft provider, model, and
512-
write-only environment. Buzz requests a ten-token output limit and turns
513-
thinking effort off. Bundled `goose-acp` uses a hidden temporary session in
514-
chat mode with extensions disabled. Buzz checks its recorded conversation
515-
for a nonempty assistant reply and rejects error content: ACP can render
516-
provider errors and empty token-limit fallbacks as ordinary text notifications.
517-
Buzz then deletes only that temporary session before returning, including on
518-
inference failure or timeout. Cancellation schedules bounded cleanup, also
519-
handling an in-flight session creation. If deletion fails, Buzz reports that
520-
the session may remain in Goose history; a crash or unresponsive sidecar can
521-
interrupt cleanup. The test consumes a small amount of provider quota.
522-
An external full Goose CLI pin retains `goose run --text --no-session
523-
--no-profile` and checks its JSON result without saving a session. Model
524-
browsing does not create a session or fall back to the CLI. Neither check
525-
verifies Buzz relay readiness, launches the agent, or changes the draft.
510+
- For Goose and Pi, **Test connection** appears below the provider/API key and
511+
above Model, including before a model is chosen. The shared form and native
512+
request lane own cancellation and result display; tests never fill Model.
513+
- For Goose, a blank Model resolves the selected provider's `defaultModel`
514+
through Goose ACP provider metadata. Buzz keeps no default-model mapping.
515+
A nonblank explicit model or effective `GOOSE_MODEL` override is tested as entered.
516+
**Test connection** sends one small request in the agent workspace using the
517+
effective draft provider, model, and write-only environment. Goose keeps its
518+
own output and thinking defaults.
519+
Bundled `goose-acp` uses a hidden temporary session in chat mode with extensions
520+
disabled. Buzz checks its recorded conversation for a nonempty assistant reply
521+
and rejects error content: ACP can render provider errors and empty token-limit
522+
fallbacks as ordinary text notifications. Buzz then deletes only that temporary
523+
session before returning, including on inference failure or timeout. Cancellation
524+
schedules bounded cleanup, also handling an in-flight session creation. If deletion
525+
fails, Buzz reports that the session may remain in Goose history; a crash or
526+
unresponsive sidecar can interrupt cleanup. The test consumes a small amount of
527+
provider quota.
528+
An external full Goose CLI pin retains `goose run --text --no-session --no-profile`
529+
without saving a session. Success requires nonempty assistant text and reported
530+
token usage in its JSON output; Goose can exit successfully with synthetic text
531+
after a provider error. Providers that omit usage cannot confirm success through
532+
this CLI check. Model browsing does not create a session or fall back to the CLI.
533+
Neither check verifies Buzz relay readiness, launches the agent, or changes the
534+
draft. Success reports the canonical provider/model tested, except that write-only
535+
overrides keep their values hidden. A provider without a default asks for a model;
536+
Buzz never substitutes another provider.
537+
- For Pi, with Model blank, Pi chooses from
538+
the selected provider's `--models provider/*` scope. Buzz checks Pi's actual
539+
selection through RPC before sending a short prompt; if Pi falls back to
540+
another provider, Buzz stops and asks for credentials. With a model selected,
541+
Buzz tests that exact provider/model. Success requires an assistant text reply
542+
and reports its canonical model ID without changing the draft. Listing models
543+
alone does not verify the API key or inference access. Tests use the agent's
544+
effective environment and existing Pi sign-in, create no saved Pi session,
545+
and stop on cancellation or timeout.
546+
526547
- Environment values never arrive in snapshots. Inputs are masked write-only
527548
patches: missing key preserves; string replaces (including empty); null removes.
528549
Undo omits a patch again. Successful save clears entered values from UI state.

‎src-tauri/src/agent_models.rs‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ pub(crate) struct Catalog {
5959
models: Vec<Model>,
6060
model_overridden: bool,
6161
disconnected: bool,
62+
#[serde(skip_serializing_if = "Option::is_none")]
63+
tested_model: Option<String>,
6264
}
6365
#[derive(Serialize)]
6466
struct Model {
@@ -282,12 +284,14 @@ pub(crate) async fn agent_models_run<R: tauri::Runtime>(
282284
let context = crate::pi_models::verify(prepared?).await?.into_context();
283285
if request.action == Operation::Test {
284286
let harness = &edit.harness;
285-
crate::pi_models::test(context, &harness.provider, &harness.model).await?;
287+
let tested_model =
288+
crate::pi_models::test(context, &harness.provider, &harness.model).await?;
286289
return Ok(Catalog {
287290
host: String::new(),
288291
models: vec![],
289292
model_overridden: false,
290293
disconnected: false,
294+
tested_model: Some(tested_model),
291295
});
292296
}
293297
let models = crate::pi_models::fetch(context)
@@ -303,6 +307,7 @@ pub(crate) async fn agent_models_run<R: tauri::Runtime>(
303307
models,
304308
model_overridden: false,
305309
disconnected: false,
310+
tested_model: None,
306311
})
307312
})
308313
.await;
@@ -335,12 +340,17 @@ pub(crate) async fn agent_models_run<R: tauri::Runtime>(
335340
.run(ticket, async move {
336341
let context = prepared?;
337342
if request.action == Operation::Test {
338-
crate::goose_models::test(context).await?;
343+
// Saved environment overrides are write-only. Testing them
344+
// must not return their hidden provider/model values to IPC.
345+
let selection_overridden = context.model_overridden
346+
|| context.environment.contains_key("GOOSE_PROVIDER");
347+
let tested_model = crate::goose_models::test(context).await?;
339348
return Ok(Catalog {
340349
host: String::new(),
341350
models: vec![],
342351
model_overridden: false,
343352
disconnected: false,
353+
tested_model: (!selection_overridden).then_some(tested_model),
344354
});
345355
}
346356
let model_overridden = context.model_overridden;
@@ -357,6 +367,7 @@ pub(crate) async fn agent_models_run<R: tauri::Runtime>(
357367
models,
358368
model_overridden,
359369
disconnected: false,
370+
tested_model: None,
360371
})
361372
})
362373
.await;
@@ -426,6 +437,7 @@ pub(crate) async fn agent_models_run<R: tauri::Runtime>(
426437
models: vec![],
427438
model_overridden,
428439
disconnected: true,
440+
tested_model: None,
429441
});
430442
}
431443
execute(
@@ -606,6 +618,7 @@ async fn execute(
606618
models,
607619
model_overridden,
608620
disconnected: false,
621+
tested_model: None,
609622
})
610623
}
611624

‎src-tauri/src/agent_models/tests.rs‎

Lines changed: 96 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -143,18 +143,30 @@ fn goose_connection_test_uses_the_draft_model_and_environment() {
143143
let (dir, _, _app, view) = fixture();
144144
let goose = dir.path().join("goose");
145145
let script = r#"#!/bin/sh
146+
if [ "$1" = acp ]; then
147+
read request
148+
case "$request" in *providers/list*) ;; *) exit 1;; esac
149+
case "$TEST_DEFAULT" in
150+
missing) printf '%s\n' '{"jsonrpc":"2.0","id":1,"result":{"entries":[]}}';;
151+
mismatch) printf '%s\n' '{"jsonrpc":"2.0","id":1,"result":{"entries":[{"providerId":"anthropic","defaultModel":"other-model"}]}}';;
152+
invalid) printf '%s\n' '{"jsonrpc":"2.0","id":1,"result":{"entries":[{"providerId":"openai","defaultModel":"bad\nmodel"}]}}';;
153+
*) printf '%s\n' '{"jsonrpc":"2.0","id":1,"result":{"entries":[{"providerId":"openai","defaultModel":"provider-default"}]}}';;
154+
esac
155+
exit
156+
fi
146157
[ "$1 $2 $3" = 'run --text Reply OK.' ] || exit 1
147158
[ "$4 $5 $6 $7 $8 $9" = '--no-session --no-profile --max-turns 1 --quiet --output-format' ] || exit 1
148159
[ "${10}" = 'json' ] || exit 1
149160
[ "$GOOSE_PROVIDER" = 'openai' ] || exit 1
150-
[ "$GOOSE_MODEL" = 'effective-model' ] || exit 1
151-
[ "$GOOSE_MAX_TOKENS" = '10' ] || exit 1
152-
[ "$GOOSE_THINKING_EFFORT" = 'off' ] || exit 1
161+
case "$GOOSE_MODEL" in effective-model|provider-default) ;; *) exit 1;; esac
153162
[ "$(pwd)" = '__WORKSPACE__' ] || exit 1
163+
printf '%s\n' 'prompt' >> prompts
154164
if [ "$OPENAI_API_KEY" = 'draft-key' ]; then
155-
printf '%s\n' '{"metadata":{"status":"completed"},"messages":[{"role":"assistant","content":[{"type":"text","text":"OK"}]}]}'
165+
printf '%s\n' '{"metadata":{"status":"completed","total_tokens":4},"messages":[{"role":"assistant","content":[{"type":"text","text":"OK"}]}]}'
166+
elif [ "$BAD_STYLE" = text ]; then
167+
printf '%s\n' '{"metadata":{"status":"completed","total_tokens":0},"messages":[{"role":"assistant","content":[{"type":"text","text":"Ran into this error: API key not valid."}]}]}'
156168
else
157-
printf '%s\n' '{"metadata":{"status":"completed"},"messages":[{"role":"assistant","content":[{"type":"error","error":"authentication failed"}]}]}'
169+
printf '%s\n' '{"metadata":{"status":"completed","total_tokens":4},"messages":[{"role":"assistant","content":[{"type":"error","error":"authentication failed"}]}]}'
158170
fi
159171
"#
160172
.replace(
@@ -174,29 +186,80 @@ fi
174186
"host":"","filter":"","action":"test","edit":edit
175187
}}),
176188
);
177-
assert_eq!(result.unwrap()["models"], json!([]));
178-
let mut bad = edit;
179-
bad["environment"]["OPENAI_API_KEY"] = json!("bad-key");
189+
let result = result.unwrap();
190+
assert_eq!(result["models"], json!([]));
191+
assert!(result.get("testedModel").is_none());
192+
assert!(!result.to_string().contains("effective-model"));
193+
let mut automatic = edit.clone();
194+
automatic["harness"]["model"] = json!("");
195+
automatic["environment"]
196+
.as_object_mut()
197+
.unwrap()
198+
.remove("GOOSE_MODEL");
180199
let ticket = invoke(&view, "agent_models_begin", json!({})).unwrap();
181-
let error = invoke(
200+
let result = invoke(
182201
&view,
183202
"agent_models_run",
184203
json!({"ticket":ticket,"request":{
185-
"host":"","filter":"","action":"test","edit":bad
204+
"host":"","filter":"","action":"test","edit":automatic
186205
}}),
187206
)
188-
.unwrap_err();
189-
assert!(error.to_string().contains("could not complete a request"));
207+
.unwrap();
208+
assert_eq!(result["testedModel"], "openai/provider-default");
209+
let prompts = std::fs::read_to_string(dir.path().join("prompts")).unwrap();
210+
for case in ["missing", "mismatch", "invalid"] {
211+
let mut unavailable = automatic.clone();
212+
unavailable["environment"]["TEST_DEFAULT"] = json!(case);
213+
let ticket = invoke(&view, "agent_models_begin", json!({})).unwrap();
214+
assert!(
215+
invoke(
216+
&view,
217+
"agent_models_run",
218+
json!({"ticket":ticket,"request":{
219+
"host":"","filter":"","action":"test","edit":unavailable
220+
}})
221+
)
222+
.is_err(),
223+
"{case}"
224+
);
225+
assert_eq!(
226+
std::fs::read_to_string(dir.path().join("prompts")).unwrap(),
227+
prompts
228+
);
229+
}
230+
let mut bad = automatic;
231+
bad["environment"]["OPENAI_API_KEY"] = json!("bad-key");
232+
for style in ["error-content", "text"] {
233+
bad["environment"]["BAD_STYLE"] = json!(style);
234+
let ticket = invoke(&view, "agent_models_begin", json!({})).unwrap();
235+
let error = invoke(
236+
&view,
237+
"agent_models_run",
238+
json!({"ticket":ticket,"request":{
239+
"host":"","filter":"","action":"test","edit":bad
240+
}}),
241+
)
242+
.unwrap_err();
243+
assert!(
244+
error.to_string().contains("could not complete a request"),
245+
"{style}"
246+
);
247+
}
190248

249+
let mut bad = edit.clone();
191250
let sidecar = dir.path().join("goose-acp");
192251
let temporary = dir.path().join("temporary-session");
193252
std::fs::write(
194253
&sidecar,
195254
r#"#!/bin/sh
196255
[ "$#" -eq 0 ] || exit 1
197-
[ "$GOOSE_PROVIDER $GOOSE_MODEL $GOOSE_MAX_TOKENS $GOOSE_THINKING_EFFORT $GOOSE_MODE" = 'openai effective-model 10 off chat' ] || exit 1
198-
[ "$(pwd)" = '__WORKSPACE__' ] || exit 1
199256
read request
257+
case "$request" in *'"method":"_goose/unstable/providers/list"'*)
258+
printf '%s\n' '{"jsonrpc":"2.0","id":1,"result":{"entries":[{"providerId":"openai","defaultModel":"provider-default"}]}}'
259+
exit 0;; esac
260+
[ "$(pwd)" = '__WORKSPACE__' ] || exit 1
261+
[ "$GOOSE_PROVIDER $GOOSE_MODE" = 'openai chat' ] || exit 1
262+
case "$GOOSE_MODEL" in effective-model|provider-default) ;; *) exit 1;; esac
200263
printf '%s\n' '{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"sessionCapabilities":{"delete":{}}}}}'
201264
read request
202265
case "$request" in *'"hidden":true'*) ;; *) exit 1 ;; esac
@@ -226,7 +289,18 @@ printf '%s\n' '{"jsonrpc":"2.0","id":4,"result":{}}'
226289
std::fs::set_permissions(&sidecar, std::fs::Permissions::from_mode(0o700)).unwrap();
227290
bad["harness"]["command"] = json!(sidecar);
228291
bad["harness"]["args"] = json!([]);
229-
for key in ["draft-key", "bad-key"] {
292+
for (key, default_model) in [
293+
("draft-key", false),
294+
("bad-key", false),
295+
("draft-key", true),
296+
] {
297+
if default_model {
298+
bad["harness"]["model"] = json!("");
299+
bad["environment"]
300+
.as_object_mut()
301+
.unwrap()
302+
.remove("GOOSE_MODEL");
303+
}
230304
bad["environment"]["OPENAI_API_KEY"] = json!(key);
231305
let ticket = invoke(&view, "agent_models_begin", json!({})).unwrap();
232306
let result = invoke(
@@ -237,7 +311,13 @@ printf '%s\n' '{"jsonrpc":"2.0","id":4,"result":{}}'
237311
}}),
238312
);
239313
if key == "draft-key" {
240-
assert_eq!(result.unwrap()["models"], json!([]));
314+
let result = result.unwrap();
315+
assert_eq!(result["models"], json!([]));
316+
if default_model {
317+
assert_eq!(result["testedModel"], "openai/provider-default");
318+
} else {
319+
assert!(result.get("testedModel").is_none());
320+
}
241321
} else {
242322
let error = result.unwrap_err().to_string();
243323
assert!(error.contains("could not complete a request"));

‎src-tauri/src/agents/tests.rs‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2226,7 +2226,23 @@ fn pi_connection_test_prompts_the_draft_selection() {
22262226
std::fs::create_dir(&tools).unwrap();
22272227
for tool in ["pi", "node", "buzz-pi-acp"] {
22282228
let file = tools.join(tool);
2229-
std::fs::write(&file, "#!/bin/sh\nif [ \"$1\" = --version ]; then printf '0.99.1\\n'; exit 0; fi\nread request\ncase \"$*\" in *'--model model-a'*) stop=stop;; *) stop=error;; esac\nprintf '{\"type\":\"message_end\",\"message\":{\"role\":\"assistant\",\"stopReason\":\"%s\",\"errorMessage\":\"401\"}}\\n' \"$stop\"\n").unwrap();
2229+
std::fs::write(
2230+
&file,
2231+
r#"#!/bin/sh
2232+
if [ "$1" = --version ]; then printf '0.99.1\n'; exit 0; fi
2233+
case "$*" in
2234+
*'--model databricks/model-a'*) model=model-a; stop=stop; error='';;
2235+
*'--model databricks/model-b'*) model=model-b; stop=error; error=401;;
2236+
*) exit 1;;
2237+
esac
2238+
read request
2239+
case "$request" in *get_state*) ;; *) exit 1;; esac
2240+
printf '{"id":"selection","type":"response","command":"get_state","success":true,"data":{"model":{"provider":"databricks","id":"%s"}}}\n' "$model"
2241+
read request
2242+
case "$request" in *prompt*) ;; *) exit 1;; esac
2243+
printf '{"type":"message_end","message":{"role":"assistant","provider":"databricks","model":"%s","content":[{"type":"text","text":"OK"}],"stopReason":"%s","errorMessage":"%s"}}\n' "$model" "$stop" "$error"
2244+
"#,
2245+
).unwrap();
22302246
std::fs::set_permissions(file, std::fs::Permissions::from_mode(0o700)).unwrap();
22312247
}
22322248
let test = |model: &str| {
@@ -2243,7 +2259,9 @@ fn pi_connection_test_prompts_the_draft_selection() {
22432259
}}),
22442260
)
22452261
};
2246-
assert_eq!(test("model-a").unwrap()["models"], json!([]));
2262+
let result = test("model-a").unwrap();
2263+
assert_eq!(result["models"], json!([]));
2264+
assert_eq!(result["testedModel"], "databricks/model-a");
22472265
let error = test("model-b").unwrap_err();
22482266
assert!(
22492267
error.as_str().unwrap().contains("rejected the API key"),

0 commit comments

Comments
 (0)