Skip to content

Commit aa51dab

Browse files
micspiralBartok9michaelneale
authored
fix(desktop): supervise and re-arm relay-mesh runtime (#2823)
Refs #2062 This carries forward the relay-mesh recovery work from #2304 by @Bartok9 (cherry-picked with original authorship/sign-offs) and adds the startup/readiness supervision and packaging fixes found while validating it against a real two-machine Buzz setup. ## From #2304 - Watch the local OpenAI ingress (`:9337`) after launch and re-arm a stale relay-mesh runtime. - Require consecutive failed probes before eviction so a transient inference stall does not cause a cold restart. - Bound stale-runtime shutdown, preserve runtime identity across the asynchronous probe, and never evict a concurrent replacement. - Re-arm only for agents that are actually running; deliberately stopped agents stay stopped. - Persist an actionable sentinel error under the managed-agent store lock and clear only that error after recovery. - Treat serve-to-client fallback as an intentional fail-safe; configured serve restoration remains on its existing path. ## Added here - Use the inference ingress (`:9337`), rather than management port `:3131`, as Buzz's client-readiness boundary. A usable client no longer fails or holds agent-save open merely because management startup is still pending. - Supervise the embedded SDK startup asynchronously, publish a pending status while management is unavailable, and keep its mesh identity alive. - Avoid racing a replacement while SDK startup still owns the embedded runtime. If that pending startup later loses ingress while a running agent still needs it, request a controlled Buzz restart to reclaim the otherwise-unreachable SDK thread. - Defer roster-driven replacement while client management startup is pending. - Keep post-launch recovery in a dedicated module so the mesh entry point remains within the desktop file-size gate. - Explicitly mark generated Unix sidecars executable. On macOS, copying over an existing non-executable destination preserved its old mode, causing packaged `buzz-acp`, `buzz-agent`, and tool sidecars to be reported as missing. ## Validation Automated: - `just ci` — passed, including formatting, Clippy with warnings denied, desktop/web/mobile checks and tests, and builds. - Full Tauri mesh-feature suite — 1,702 passed, 0 failed, 15 ignored. - Mesh-feature Clippy with `-D warnings` — passed. - Release macOS app bundle with `mesh-llm` — built successfully; every bundled sidecar passed executable-mode and deep code-signature verification. Live two-machine E2E: - M5: released Buzz serving `unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M`. - Mac mini: this branch's packaged Buzz running a saved `buzz-agent`/relay-mesh agent. - Confirmed `:9337` accepted inference and the saved ACP harness started with no error while `:3131` was still unavailable. - Exact inference succeeded before restart (`CORRECTED-MINI-E2E-OK`). - Bespoke Buzz shut down cleanly in 2.3s, then restored ingress and the saved harness in 18.8s while `:3131` was still unavailable. - Exact inference succeeded after restart (`AFTER-RESTART-E2E-OK`). - A real Buzz `@C55` message traversed desktop → `buzz-acp` → `buzz-agent` → mini `:9337` → M5 compute and published the requested reply successfully. --------- Signed-off-by: Bartok9 <[email protected]> Signed-off-by: Michael Neale <[email protected]> Co-authored-by: Bartok9 <[email protected]> Co-authored-by: Michael Neale <[email protected]>
1 parent 8eb6e3e commit aa51dab

8 files changed

Lines changed: 924 additions & 26 deletions

File tree

‎desktop/scripts/check-file-sizes.mjs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -400,7 +400,10 @@ const overrides = new Map([
400400
// transition lock doc broadened to cover all protected-PID transitions, and
401401
// clear_agent_session_caches (per-pubkey retain) added alongside the
402402
// per-key clear. Load-bearing identity-contract change; queued to split.
403-
["src-tauri/src/app_state.rs", 1081],
403+
// +4 (1081 -> 1085): mesh recovery keeps one app-scoped state object beside
404+
// the embedded runtime and coordinator. Probe/re-arm logic lives in
405+
// mesh_llm/recovery.rs rather than growing AppState or command modules.
406+
["src-tauri/src/app_state.rs", 1085],
404407
// multi-slot splitting + no-op suppression (#1309): the ReadStateManager
405408
// class grew from ~700 lines to ~1019 with the addition of
406409
// splitContextsIntoBudgetedSlots (pure fn + 5 tests), publishSplitSlots,

‎desktop/src-tauri/src/app_state.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ pub struct AppState {
108108
/// In-process mesh-llm node started by Buzz Desktop.
109109
#[cfg(feature = "mesh-llm")]
110110
pub mesh_llm_runtime: AsyncMutex<Option<crate::mesh_llm::DesktopMeshRuntime>>,
111+
#[cfg(feature = "mesh-llm")]
112+
pub mesh_recovery: crate::mesh_llm::MeshRecoveryState,
111113
/// Runtime-owned shared-compute coordinator. It publishes member-signed
112114
/// discovery status and reconciles MeshLLM's admission roster; MeshLLM
113115
/// itself owns direct QUIC/iroh connection establishment.
@@ -223,6 +225,8 @@ pub fn build_app_state() -> AppState {
223225
#[cfg(feature = "mesh-llm")]
224226
mesh_llm_runtime: AsyncMutex::new(None),
225227
#[cfg(feature = "mesh-llm")]
228+
mesh_recovery: crate::mesh_llm::MeshRecoveryState::default(),
229+
#[cfg(feature = "mesh-llm")]
226230
mesh_coordinator: AsyncMutex::new(None),
227231
pending_owned_channels: Mutex::new(std::collections::HashSet::new()),
228232
}

‎desktop/src-tauri/src/commands/mesh_llm.rs‎

Lines changed: 54 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ pub(crate) async fn restore_mesh_sharing(app: &AppHandle, state: &AppState) -> C
157157
};
158158
let started = mesh_llm::DesktopMeshRuntime::start(request)
159159
.await
160-
.map_err(|error| format!("failed to restore Share Compute: {error}"))?;
160+
.map_err(|error| format!("failed to restore Share Compute: {error:#}"))?;
161161
*runtime = Some(started);
162162
drop(runtime);
163163
mesh_llm::publish_current_status_once(app, "restore").await;
@@ -183,11 +183,11 @@ pub async fn mesh_start_node(
183183
let saved_request = request.clone();
184184
let started = mesh_llm::DesktopMeshRuntime::start(request)
185185
.await
186-
.map_err(|error| error.to_string())?;
186+
.map_err(|error| format!("{error:#}"))?;
187187
let status = started
188188
.status()
189189
.await
190-
.map_err(|error| format!("mesh node started but status probe failed: {error}"))?;
190+
.map_err(|error| format!("mesh node started but status probe failed: {error:#}"))?;
191191
*runtime = Some(started);
192192
drop(runtime);
193193
if saved_request.mode == mesh_llm::MeshNodeMode::Serve {
@@ -407,11 +407,11 @@ pub(crate) async fn ensure_client_node_for_model(
407407
}
408408
let started = mesh_llm::DesktopMeshRuntime::start(start)
409409
.await
410-
.map_err(|error| format!("mesh client failed to start: {error}"))?;
410+
.map_err(|error| format!("mesh client failed to start: {error:#}"))?;
411411
let status = started
412412
.status()
413413
.await
414-
.map_err(|error| format!("mesh client started but status probe failed: {error}"))?;
414+
.map_err(|error| format!("mesh client started but status probe failed: {error:#}"))?;
415415
*runtime = Some(started);
416416
Ok(status)
417417
}
@@ -490,9 +490,47 @@ pub(crate) async fn ensure_relay_mesh_for_record(
490490
};
491491
// A local serve/client runtime already owns the OpenAI ingress and its
492492
// router can resolve both `auto` and explicit remote models. Do not require
493-
// a separate relay-advertised target in that case.
493+
// a separate relay-advertised target in that case — BUT only trust it when
494+
// the ingress is actually alive. A runtime that exited/wedged after launch
495+
// leaves `mesh_llm_runtime = Some` pointing at a dead `:9337` ingress, so a
496+
// blind `wait_for_mesh_inference` would just time out and the agent would
497+
// stay silent (#2062). Probe first; if the ingress is dead, drop the stale
498+
// runtime and fall through to re-arm it. The mesh coordinator watchdog also
499+
// calls this path after eviction so recovery is not start-only (Brad #2304).
494500
if state.mesh_llm_runtime.lock().await.is_some() {
495-
return wait_for_mesh_inference(&model_id).await;
501+
match mesh_llm::recover_stale_mesh_runtime(
502+
&state,
503+
mesh_llm::MeshRecoveryUrgency::Foreground,
504+
)
505+
.await
506+
{
507+
mesh_llm::MeshRuntimeRecovery::Live => {
508+
return wait_for_mesh_inference(&model_id).await;
509+
}
510+
mesh_llm::MeshRuntimeRecovery::Evicted | mesh_llm::MeshRuntimeRecovery::Absent => {}
511+
mesh_llm::MeshRuntimeRecovery::Debouncing => {
512+
return Err(
513+
"Buzz shared compute ingress is temporarily unresponsive; recovery is already scheduled. Try again shortly."
514+
.to_string(),
515+
);
516+
}
517+
mesh_llm::MeshRuntimeRecovery::ReleasePending => {
518+
return Err(
519+
"Buzz shared compute is still shutting down its previous local ingress. Try again shortly."
520+
.to_string(),
521+
);
522+
}
523+
mesh_llm::MeshRuntimeRecovery::Replaced => {
524+
return wait_for_mesh_inference(&model_id).await;
525+
}
526+
mesh_llm::MeshRuntimeRecovery::RestartRequired => {
527+
app.request_restart();
528+
return Err(
529+
"Buzz shared compute startup lost its local ingress before shutdown control became available. Buzz is restarting to recover it."
530+
.to_string(),
531+
);
532+
}
533+
}
496534
}
497535
let target = match resolve_mesh_bootstrap_target(&state, &model_id).await {
498536
Ok(Some(target)) => target,
@@ -509,6 +547,15 @@ pub(crate) async fn ensure_relay_mesh_for_record(
509547
}
510548
};
511549

550+
// Serve→Client re-arm transition (micspiral review #3, intentional-by-design):
551+
// if the dead ingress belonged to a *serve* node with running consumer
552+
// agents, this re-arms it as a Client (`MeshNodeMode::Client`). That is the
553+
// correct/safe recovery here — config-backed serve restoration is
554+
// `restore_mesh_sharing`'s job (`MeshNodeMode::Serve`), and
555+
// `ensure_client_node_for_model` reuses any live runtime of *either* mode
556+
// (the router resolves per-request), so it only cold-starts a Client when
557+
// there is genuinely no runtime. Falling back to Client if a serve node
558+
// crashed under local pressure is a desirable fail-safe, not a regression.
512559
ensure_client_node_for_model(&state, &model_id, Some(target.endpoint_addr)).await?;
513560
wait_for_mesh_inference(&model_id).await
514561
}

‎desktop/src-tauri/src/mesh_llm/coordinator.rs‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,16 @@ const STATUS_D_TAG_PREFIX: &str = "buzz-mesh-member-status";
2222
const ROSTER_POLL_INTERVAL: Duration = Duration::from_secs(60);
2323
const STATUS_PUBLISH_INTERVAL: Duration = Duration::from_secs(45);
2424
const STATUS_PUBLISH_TIMEOUT: Duration = Duration::from_secs(10);
25+
/// Post-launch ingress liveness / re-arm for #2062. Bounded backoff: base 15s,
26+
/// doubles after consecutive failures up to 120s so a sticky offline peer does
27+
/// not hammer discovery every tick, but a recovered peer is noticed quickly.
28+
const INGRESS_WATCHDOG_BASE: Duration = Duration::from_secs(15);
29+
const INGRESS_WATCHDOG_MAX: Duration = Duration::from_secs(120);
2530

2631
pub struct MeshCoordinator {
2732
_status_publisher: tokio::task::JoinHandle<()>,
2833
_roster_watcher: tokio::task::JoinHandle<()>,
34+
_ingress_watchdog: tokio::task::JoinHandle<()>,
2935
}
3036

3137
/// Start the runtime-owned status publisher and admission-roster watcher.
@@ -63,16 +69,40 @@ pub async fn start_coordinator(app: AppHandle) {
6369
}
6470
});
6571

72+
// Brad #2304 / #2062: ensure_relay_mesh_for_record only runs on explicit
73+
// start + launch restore. After launch, local buzz-agent processes talk
74+
// directly to :9337; there is no desktop "turn dispatch" hook. This
75+
// watchdog is the post-launch seam: probe ingress, drop a zombie handle,
76+
// re-arm via ensure_relay_mesh_for_record, surface last_error on failure.
77+
let ingress_app = app.clone();
78+
let ingress_watchdog = tokio::spawn(async move {
79+
let mut sleep_for = INGRESS_WATCHDOG_BASE;
80+
loop {
81+
tokio::time::sleep(sleep_for).await;
82+
match crate::mesh_llm::rearm_relay_mesh_for_running_agents(&ingress_app).await {
83+
Ok(()) => {
84+
sleep_for = INGRESS_WATCHDOG_BASE;
85+
}
86+
Err(error) => {
87+
eprintln!("buzz-mesh: ingress re-arm watchdog: {error}");
88+
sleep_for = (sleep_for * 2).min(INGRESS_WATCHDOG_MAX);
89+
}
90+
}
91+
}
92+
});
93+
6694
let state = app.state::<AppState>();
6795
let mut guard = state.mesh_coordinator.lock().await;
6896
if guard.is_none() {
6997
*guard = Some(MeshCoordinator {
7098
_status_publisher: status_publisher,
7199
_roster_watcher: roster_watcher,
100+
_ingress_watchdog: ingress_watchdog,
72101
});
73102
} else {
74103
status_publisher.abort();
75104
roster_watcher.abort();
105+
ingress_watchdog.abort();
76106
}
77107
}
78108

@@ -182,6 +212,16 @@ async fn reconcile_roster(
182212
let mut request = current_request;
183213
request.trusted_owner_ids = Some(fresh);
184214
let mut guard = state.mesh_llm_runtime.lock().await;
215+
let startup_pending = match guard.as_ref() {
216+
Some(runtime) => runtime.is_starting().await,
217+
None => false,
218+
};
219+
if startup_pending {
220+
eprintln!(
221+
"buzz-mesh: membership roster changed while client management startup is pending; deferring restart"
222+
);
223+
return Ok(());
224+
}
185225
let Some(running) = guard.take() else {
186226
return Ok(());
187227
};
@@ -191,7 +231,7 @@ async fn reconcile_roster(
191231
}
192232
let replacement = crate::mesh_llm::DesktopMeshRuntime::start(request)
193233
.await
194-
.map_err(|error| format!("mesh node restart after roster change failed: {error}"))?;
234+
.map_err(|error| format!("mesh node restart after roster change failed: {error:#}"))?;
195235
*guard = Some(replacement);
196236
Ok(())
197237
}

0 commit comments

Comments
 (0)