Skip to content

fix(kernel): harden task lifecycle (panics, locks, races)#4122

Merged
houko merged 4 commits into
mainfrom
fix/kernel-task-lifecycle
Apr 30, 2026
Merged

fix(kernel): harden task lifecycle (panics, locks, races)#4122
houko merged 4 commits into
mainfrom
fix/kernel-task-lifecycle

Conversation

@houko

@houko houko commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Five kernel-side fixes around spawned-task lifecycle, lock recovery,
and per-session task tracking.

Test plan

  • CI green (clippy, workspace tests)
  • Trigger a panic inside one of the migrated background loops
    (e.g. inbox watcher) — verify error! shows up with
    task=inbox_watcher
  • Run two concurrent send_message_full calls for the same
    (agent, session) and verify the older AbortHandle is the one
    aborted, and that cleanup of the older task does not evict the
    new entry
  • Cause a panic inside a RwLock write critical section (in a
    test) and verify the next budget_config() call logs a warn
    and proceeds with the recovered state

@github-actions github-actions Bot added area/kernel Core kernel (scheduling, RBAC, workflows) size/L 250-999 lines changed has-conflicts PR has merge conflicts that need resolution labels Apr 29, 2026
houko and others added 4 commits May 1, 2026 07:25
Five related kernel-side fixes around spawned-task lifecycle and
shared-state safety, surfaced by #3753, #3740, #3739, #3445, #3447.

* persist_runs_async (workflow): outer JoinError from spawn_blocking
  was logged-and-swallowed. Now returns Result and the two execute
  callers propagate the persist failure into the run result instead
  of pretending success. Closes #3753.

* supervised_spawn module + spawn_logged consolidation: introduce
  spawn_supervised(name, future) that catch_unwinds the future and
  emits an error! with the task name on panic. Existing in-file
  spawn_logged becomes a thin alias. Migrated ~14 critical
  fire-and-forget sites (inbox watcher + dispatch, whatsapp gateway
  supervisor, session stream fan-out, auto_dream scheduler/dispatch
  /run, owner notify, auto-memorize, several daemon-loop tickers
  inside kernel boot, mcp connect/reconnect, a2a discover, approval
  resolution, memory event publish, cron agent turn, staggered
  background-agent start). Remaining raw tokio::spawn calls fall
  into three buckets that don't fit fire-and-forget: tests, handles
  collected for join (Vec<JoinHandle> in MCP key probing), and
  closures whose JoinHandle is the contract (background.start_agent).
  Partially addresses #3740.

* running_tasks insert overwrite: kept the existing atomic
  insert-then-abort-displaced pattern (correct under DashMap shard
  lock) and documented it. Closes #3739.

* running_tasks insert-after-spawn race: gave RunningTask a unique
  task_id (Uuid). The task's own cleanup uses remove_if(... task_id
  matches ...) so a slow predecessor can never delete a successor's
  entry, and the registration site skips insert when
  handle.is_finished() so a task that completed before we got to
  register doesn't leave a stale AbortHandle behind. Closes #3445.

* budget_config RwLock poison: read/write paths already used
  unwrap_or_else(|p| p.into_inner()) but did so silently. Added a
  tracing::warn on the recovery path so operators can see when a
  poisoned-then-recovered lock is masking an upstream panic.
  Closes #3447.
The module-level doc had a paragraph where one line started with
"+ \`catch_unwind\`". Markdown parsers (and rustdoc/clippy) read the
"+ " as a list-item bullet, then flag the next three continuation
lines as `doc_lazy_continuation` (list items need indent). Replacing
"+" with "plus" keeps the prose identical and lets the paragraph
parse as plain text again.
- kernel/mod.rs: document the residual is_finished() race window and
  explain why it is harmless (AbortHandle::abort on a finished task is
  no-op; next turn for the same session overwrites the entry).
- supervised_spawn.rs: add doc note that on panic the future is dropped
  and not retried — this wrapper is a diagnostic aid, not a restart
  mechanism.
@houko
houko force-pushed the fix/kernel-task-lifecycle branch from 2b9edd8 to ac8513b Compare April 30, 2026 22:26
@houko
houko merged commit 1ca5598 into main Apr 30, 2026
12 of 13 checks passed
@houko
houko deleted the fix/kernel-task-lifecycle branch April 30, 2026 22:26
@github-actions github-actions Bot added ready-for-review PR is ready for maintainer review and removed has-conflicts PR has merge conflicts that need resolution labels Apr 30, 2026
@houko houko mentioned this pull request May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment