Phase 1 -- investigation findings

Gap confirmed. EmbeddingWarmupHostedService.WarmUpAsync/WarmUpRelevanceGateAsync catch every provisioning/load failure for both ONNX models (embedder: snowflake-arctic-embed-m-int8 or configured ModelId; relevance: ms-marco-minilm-l-6-v2), set the holder to an Unavailable* stub, and logger.LogError(...). Nothing operator-facing fired -- discovery required netclaw doctor, netclaw status, or reading daemon.log, all pull-based.

Mechanism found and reused: IOperationalNotificationSink/OperationalAlert (src/Netclaw.Configuration/OperationalAlert.cs, IOperationalNotificationSink.cs). This is the generic push-to-operator seam already used by:

• McpReconnectionService (mcp.server.reconnected)
• McpClientManager (mcp.server.disconnected, mcp.auth.expired)
• ReminderManagerActor (reminder.execution.failed, reminder.auto_disabled) -- the PR feat(reminders): operator visibility into reminder failures and skips (#1494) #1503 "reminder failure visibility" mechanism turned out to be built on this same generic sink, not a bespoke reminder-only path (the reminder-specific IReminderChannelNotifier is a separate, narrower mechanism for posting into a specific reminder's destination channel -- not reusable here since embedding warmup runs at startup with no session/channel context)
• RoutingChatClient (provider.failover, provider.unreachable)
• ProviderOAuthTokenRefreshService, McpOAuthService, BinaryUpdateCheckService, DaemonLifecycleNotifier

WebhookNotificationService implements the sink and delivers to configured Slack/generic webhook targets (or NullNotificationSink when none configured -- alerts are always logged at the emission site regardless). This is a required, non-nullable dependency at every existing call site, consistent with the constitution's "no silent fallbacks" / required-dependency rules.

The 0.24.3 "Degraded startup mode" banner (PR #1540, No-Op IChatClient fallback) is a reactive mechanism (a chat-turn response), not a push -- not applicable here.

Given a strong existing mechanism, proceeded to Phase 2 (no STOP).

What was implemented

• Two new AlertType values: MemoryEmbeddingModelUnavailable, MemoryRelevanceModelUnavailable (OperationalAlert.cs).
• EmbeddingWarmupHostedService takes a new required IOperationalNotificationSink dependency. On provisioning/load failure for either model (while Memory.Embeddings.Enabled=true), fires an alert carrying:
  • which model failed (modelId, alert Source)
  • why (reason = exception message)
  • consequence: "Memory recall/dedup is running lexical-only -- semantic features are degraded." (embedder) / "The relevance gate is disabled -- recall is unfiltered by the cross-encoder." (relevance model)
  • remediation: check network/disk, netclaw doctor, and (embedder only) netclaw memory backfill-embeddings re-provisions on next start -- wording mirrors the existing MemoryEmbeddingDoctorCheck/MemoryRelevanceGateDoctorCheck remediation text so operators see consistent guidance whether pulling netclaw doctor or reacting to a push.
• Latched per model via Interlocked.CompareExchange so each model alerts at most once per daemon run, never per retry.
• No alert when Memory.Embeddings.Enabled=false -- intentional, not degraded, state.
• Keep-warm mid-run failures deliberately not wired in. KeepWarmTickAsync's own doc comment already states a single tick miss is not user-visible degradation (transient ONNX hiccup, may self-heal next tick). Alerting on the first miss would false-positive on exactly that documented case. Doing this right needs a consecutive-failure threshold (mirroring ReminderManagerActor's auto-disable pattern) before promoting a keep-warm miss to alert-worthy -- a real design decision, not just plumbing. Left as a follow-up, documented in LogKeepWarmFailed's remarks.

Bonus fix (surfaced by testing the both-models-fail case): WarmUpAsync had a pre-existing return; in the embedder's catch block that made WarmUpRelevanceGateAsync unreachable whenever the embedder itself failed -- directly contradicting the method's own documented "runs regardless of whether the embedder itself just degraded" contract, and silently suppressing the relevance-model alert in the worst-case (both models down) scenario. Restructured so the embedder try/catch no longer early-returns; gap-repair only runs if the embedder loaded, and the relevance gate always gets its independent attempt.

Sample alert content

Category: MemoryEmbeddingModelUnavailable
Type: memory.embedding_model.unavailable
Severity: Warning
Source: snowflake-arctic-embed-m-int8
Summary: Memory embedding model 'snowflake-arctic-embed-m-int8' could not be
  provisioned or loaded: <exception message> Memory recall/dedup is running
  lexical-only -- semantic features are degraded.
Context:
  modelId: snowflake-arctic-embed-m-int8
  reason: <exception message>
  consequence: Memory recall/dedup is running lexical-only -- semantic
    features are degraded.
  remediation: Check network access and disk space, run `netclaw doctor`, or
    run `netclaw memory backfill-embeddings` -- the daemon re-provisions the
    model on its next start.

Tests

EmbeddingWarmupHostedServiceTests (25 tests total, new coverage added):

• embedder-only failure -> exactly one alert, correct category/content
• relevance-only failure -> exactly one alert, correct category/content (no backfill-embeddings mention)
• both models fail -> two distinct alerts, distinct alert ids
• success path -> zero alerts
• disabled config -> zero alerts
• repeated WarmUpAsync calls -> latch holds, still exactly one alert per model total (not per call)

Gates run locally:

• dotnet build Netclaw.slnx -- clean
• dotnet test -- Netclaw.Daemon.Tests (854), Netclaw.Actors.Tests (2671), Netclaw.Embeddings.Tests (48), Netclaw.Configuration.Tests (467) -- all green
• dotnet slopwatch analyze -- 0 issues
• Add-FileHeaders.ps1 -Verify -- all headers present
• No *Config properties added/changed, so no schema (netclaw-config.v1.schema.json) update needed
• Skill-content eval suite (./evals/run-evals.sh) not run here -- requires a live model endpoint/Docker not available in this environment (same caveat noted in PR feat(reminders): operator visibility into reminder failures and skips (#1494) #1503); the skill edits are additive documentation lines describing an already-implemented, tested code path, low risk

Skill sync

• netclaw-operations SKILL.md 2.25.0 -> 2.26.0; references/diagnostics.md documents the new alert types and a new "Memory embedding/relevance model unavailable" symptom row
• netclaw-memory SKILL.md 1.13.0 -> 1.14.0; Embeddings section now mentions the pushed operator alert alongside the existing log/status pull-based signals