fix(cron+subagent): use task-completion-routes as announce-delivery fallback#93899
fix(cron+subagent): use task-completion-routes as announce-delivery fallback#93899wangmiao0668000666 wants to merge 2 commits into
Conversation
Adds a new 'task_completion_routes' table to the shared state DB and a src/infra/task-completion-route.ts module exposing four idempotent operations: - registerTaskCompletionRoute(input) - called once at task prep / first delivery-target resolution - resolveTaskCompletionRoute(taskId) - called by the deliverer as a fallback when in-memory routing is missing - noteRouteDeliveryAttempt(taskId, status) - debug telemetry - retireTaskCompletionRoute(taskId) - called in finally blocks so unretired rows do not accumulate This is foundation only - no behavior change. No call site is wired to the registry yet. That is the next PR (which will close openclaw#92460, openclaw#92076, and openclaw#93323 by hooking the registry into the cron and subagent completion delivery paths). Schema: - New table task_completion_routes(task_id PK, source, channel, to_target, account_id, thread_id, registered_at, retired_at, delivery_attempts, last_delivery_status, last_delivery_at) - Two partial indexes: active lookup (retired_at IS NULL) and source+age for orphan pruning Module guarantees: - register is idempotent on duplicate task_id - resolve returns null for retired rows - note and retire are idempotent - pruneOrphanedRoutes deletes unretired rows older than threshold Tests: 9 unit tests covering register, resolve, note, retire, prune, durability (rows survive across DB connections), and validation (missing taskId/source throw).
…allback Wires the new task_completion_routes SQLite registry (from feat/task-completion-routes-module, PR 1) into the cron and subagent completion delivery paths. The registry is the authoritative fallback when the in-memory deliveryContext is lost between prep and delivery. Closes three P1 issues that all share the same root cause (no durable per-task record of the routable target resolved at task start): - openclaw#92460: cron completion announcer drops delivery.channel when the session entry's deliveryContext is never persisted. The cron hookup registers the resolved target at dispatchCronDelivery start. - openclaw#92076: subagent completion delivery fails when the requester is inactive and the handoff path is unavailable. The subagent hookup registers the target at first delivery-target resolution and adds a registry fallback between the active-requester-wake failure and the existing generated-media path. Long child output is bounded via capDirectTextContent (head 2600 + tail 1000, max 4000) to avoid lock-amplifying transcript writes. - openclaw#93323: subagent-announce-delivery drops at all spawn depths; the same registry works at all spawn depths, so this fix covers it. Cron hookup (src/cron/isolated-agent/delivery-dispatch.ts, +34): - registerTaskCompletionRoute at the top of dispatchCronDelivery when resolvedDelivery.ok (uses runSessionKey as taskId) - retireTaskCompletionRoute in finishSilentReplyDelivery and the deliverViaDirectAndCleanup finally block (both wrapped in try/catch to never mask the return value) Subagent hookup (src/agents/subagent-announce-delivery.ts, +112): - capDirectTextContent with MAX=4000, HEAD=2600, TAIL=1000 constants. Applied inside deliverTextCompletionDirect so long child output is bounded to head + truncated-marker + tail. - Route-registry fallback (line 1441) inserted after activeRequesterWakeFailed = true. When in-memory deliveryTarget.deliver is false but the registry has a routable target, consult the registry and dispatch via deliverTextCompletionDirect. Gated by isDirectMessageDeliveryTarget so we never accidentally route to a non-direct target picked up from a stale record. - retireTaskCompletionRoute in a finally block so the route is always retired, even on throw paths. Idempotent. Tests + live proof: - 4 cron integration tests in src/cron/isolated-agent.task-completion-route.test.ts (session entry wins, registry is the fallback, finally retires on both success and throw, orphan prune eligibility) - 80 cron dispatch tests (no regression) - 97 subagent deliverer tests (no regression) - 2 standalone live repros that exercise the full failure cascade against a real SQLite state DB (real channel, not import-test) Orphan self-lock (openclaw#92395) and ACP / media-understanding completion paths are explicitly out of scope; same module covers them with a 3-line register + retire pair each, follow-up PR per source value.
|
Heads up for reviewers: this PR's base is To see only the 6 hookup files added by this PR, change the base to |
|
Closing this PR together with #93898. The 2-PR split (foundation + hookups) did not deliver the intended benefit because cross-fork base change is not supported by GitHub PR API, so reviewers see all 11 files vs main regardless of which PR is opened. Closing this PR to start fresh: the cron + subagent hookup code (delivered here against the
Branch is preserved at |
Summary
Wires the
task_completion_routesSQLite registry module (added infeat/task-completion-routes-module, PR 1) into the cron and subagent completion delivery paths. The registry is the authoritative fallback when the in-memorydeliveryContextis lost between prep and delivery.Closes three P1 issues that all share the same root cause — no durable per-task record of the routable target resolved at task start:
delivery.channelon the final controller return because the session entry'sdeliveryContextis never persisted. The cron hookup registers the resolved target atdispatchCronDeliverystart, so the announce deliverer can always find a routable target even if the in-memory session entry is lost.capDirectTextContentto head 2600 + tail 1000 chars (max 4000) to avoid lock-amplifying transcript writes.subagent-announce-deliverydrops direct-announce notifications at all spawn depths (likely root of subagent spawn persists raw provider instead of CLI runtime — depth-2 cold spawns silently die with 'lost execution context' (two unpatched #57326 call sites, fix included) #92405). The same route-registry works at all spawn depths, so the fix covers this too.Depends on
feat/task-completion-routes-module(PR 1, same author) — adds thetask_completion_routestable andsrc/infra/task-completion-route.tsmodule. This PR assumes that module is in main or stacked.Changes
Cron hookup (
src/cron/isolated-agent/delivery-dispatch.ts, +34)registerTaskCompletionRoute(...)called at the top ofdispatchCronDeliverywhenparams.resolvedDelivery.ok. This is the earliest point in the cron flow where the delivery target is known. The same taskId (runSessionKey) is used for both register and retire.retireTaskCompletionRoute(params.runSessionKey)added in two places:finishSilentReplyDelivery(silent-reply path) aftercleanupDirectCronSessionIfNeededfinallyblock ofdeliverViaDirectAndCleanup(direct-delivery path) aftercleanupDirectCronSessionIfNeededtry { } catch {}so registry failures never mask the return value.Subagent hookup (
src/agents/subagent-announce-delivery.ts, +112)capDirectTextContent(content)(line 926) with constantsMAX=4000, HEAD=2600, TAIL=1000. Used insidedeliverTextCompletionDirectso long child output is bounded to head + truncated-marker + tail, eliminating the lock-amplification risk.activeRequesterWakeFailed = true. When the in-memorydeliveryTarget.deliverisfalsebut the registry has a routable target (registered earlier in the same call), consult the registry and dispatch viadeliverTextCompletionDirect. Gated byisDirectMessageDeliveryTargetso we never accidentally route to a non-direct target picked up from a stale record.try { retireTaskCompletionRoute(...) } catch {}added in afinallyblock so the route is always retired, even on throw paths. Idempotent — safe even if register was never called.Tests
src/cron/isolated-agent.task-completion-route.test.ts(+220, new) — 4 cron integration tests for the fallback priority (session entry wins, registry is the fallback, finally retires on both success and throw, orphan prune eligibility)src/agents/subagent-announce-delivery.test.ts(+10) — a small comment block documenting the design decision (unit tests for the deliverer text-fallback path are covered by the live repro below; the existing wrapper shape does not expose the conditions needed to force that path)Live environment proof
scripts/repro/issue-92460-task-completion-route.mts(+226, new) — 5-step proof against a real SQLite state DB. Walks the cron completion delivery flow with the session entry'sdeliveryContextdeliberately absent, proves the registry surfaces the target, retires on settle, survives a gateway restart, and supports orphan pruning.scripts/repro/issue-92076-subagent-direct-fallback.mts(+221, new) — 5-step proof against a real SQLite state DB. Drives the subagent completion deliverer throughno_active_runwake failure with no in-memorydeliveryTarget, proves the route-registry fallback surfaces a routable target and callssendMessagewith the route's channel/to, the long child result is bounded from 5001 → 3733 chars via the cap function, and the route is retired infinally.Real behavior proof
Behavior addressed: When the in-memory session entry loses its
deliveryContext(cron side, #92460) OR when the active requester wake fails and the in-memorydeliveryTargetis unavailable (subagent side, #92076), the route-registry fallback surfaces the target that was registered at task prep / first delivery-target resolution. Long child results are bounded viacapDirectTextContent(head 2600 + tail 1000, max 4000) so requester transcript writes are not lock-amplified.Real environment tested:
Linux 4.19.112-2.el8.x86_64, Nodev22.22.0,pnpm exec tsxagainst the production module + production code paths. Two standalone repro scripts run against a realnode:sqlitedatabase; no vitest mocks for the registry.Exact steps or command run after this patch:
Evidence after fix:
#92460live proof (5/5 steps pass):#92076live proof (5/5 steps pass):Observed result after fix:
deliveryContextis missing. The fallback path resolves the routable target from the durable registry, retires on settle, and survives a gateway restart (Step 4 proves the row is on disk in anode:sqliteconnection).no_active_run) and the in-memory target is unavailable. The route-registry fallback fires,sendMessageis called with the route's channel/to, the long child result is bounded from 5001 → 3733 chars via the cap function, and the route is retired infinally.What was not tested: A live
openclawcron run / subagent run was not executed; the repros use the production module + real SQLite but the cron and subagent orchestration is not invoked end-to-end. The repros also do not exercise the orphan self-lock path (#92395), which is explicitly out of scope for this PR (see below).Verification
Out of scope (explicitly tracked separately)
shouldRemoveContendedLockFilechange would expand the PR surface tosrc/agents/session-write-lock.tsand require its own test surface. ClawSweeper's review explicitly said "decide whether in scope or separately tracked" — we chose separately tracked and reference [Bug]: Session write lock held for entire turn duration causes queued user messages to silently fail #92395 in the comments.task-completion-routesmodule covers them with a 3-lineregister+retirepair each, but this PR is intentionally limited to cron + subagent to keep the review surface focused. Follow-up PR persourcevalue.Issues
Branch
fix/cron-subagent-announce-fallbackbased onfeat/task-completion-routes-module(PR 1).