Skip to content

fix(cron+subagent): use task-completion-routes as announce-delivery fallback#93899

Closed
wangmiao0668000666 wants to merge 2 commits into
openclaw:mainfrom
wangmiao0668000666:fix/cron-subagent-announce-fallback
Closed

fix(cron+subagent): use task-completion-routes as announce-delivery fallback#93899
wangmiao0668000666 wants to merge 2 commits into
openclaw:mainfrom
wangmiao0668000666:fix/cron-subagent-announce-fallback

Conversation

@wangmiao0668000666

Copy link
Copy Markdown
Contributor

Summary

Wires the task_completion_routes SQLite registry module (added in feat/task-completion-routes-module, PR 1) into the cron and subagent completion delivery paths. The registry is the authoritative fallback when the in-memory deliveryContext is lost between prep and delivery.

Closes three P1 issues that all share the same root cause — no durable per-task record of the routable target resolved at task start:

Depends on

  • feat/task-completion-routes-module (PR 1, same author) — adds the task_completion_routes table and src/infra/task-completion-route.ts module. This PR assumes that module is in main or stacked.

Changes

Cron hookup (src/cron/isolated-agent/delivery-dispatch.ts, +34)

  • registerTaskCompletionRoute(...) called at the top of dispatchCronDelivery when params.resolvedDelivery.ok. This is the earliest point in the cron flow where the delivery target is known. The same taskId (runSessionKey) is used for both register and retire.
  • retireTaskCompletionRoute(params.runSessionKey) added in two places:
    • inside finishSilentReplyDelivery (silent-reply path) after cleanupDirectCronSessionIfNeeded
    • in the finally block of deliverViaDirectAndCleanup (direct-delivery path) after cleanupDirectCronSessionIfNeeded
  • Both retires are wrapped in try { } catch {} so registry failures never mask the return value.

Subagent hookup (src/agents/subagent-announce-delivery.ts, +112)

  • New capDirectTextContent(content) (line 926) with constants MAX=4000, HEAD=2600, TAIL=1000. Used inside deliverTextCompletionDirect so long child output is bounded to head + truncated-marker + tail, eliminating the lock-amplification risk.
  • New route-registry fallback (line 1441) inserted after activeRequesterWakeFailed = true. When the in-memory deliveryTarget.deliver is false but the registry has a routable target (registered earlier in the same call), consult the registry and dispatch via deliverTextCompletionDirect. Gated by isDirectMessageDeliveryTarget so we never accidentally route to a non-direct target picked up from a stale record.
  • try { retireTaskCompletionRoute(...) } catch {} added in a finally block so the route is always retired, even on throw paths. Idempotent — safe even if register was never called.

Tests

  • src/cron/isolated-agent.task-completion-route.test.ts (+220, new) — 4 cron integration tests for the fallback priority (session entry wins, registry is the fallback, finally retires on both success and throw, orphan prune eligibility)
  • src/agents/subagent-announce-delivery.test.ts (+10) — a small comment block documenting the design decision (unit tests for the deliverer text-fallback path are covered by the live repro below; the existing wrapper shape does not expose the conditions needed to force that path)

Live environment proof

  • scripts/repro/issue-92460-task-completion-route.mts (+226, new) — 5-step proof against a real SQLite state DB. Walks the cron completion delivery flow with the session entry's deliveryContext deliberately absent, proves the registry surfaces the target, retires on settle, survives a gateway restart, and supports orphan pruning.
  • scripts/repro/issue-92076-subagent-direct-fallback.mts (+221, new) — 5-step proof against a real SQLite state DB. Drives the subagent completion deliverer through no_active_run wake failure with no in-memory deliveryTarget, proves the route-registry fallback surfaces a routable target and calls sendMessage with the route's channel/to, the long child result is bounded from 5001 → 3733 chars via the cap function, and the route is retired in finally.

Real behavior proof

Behavior addressed: When the in-memory session entry loses its deliveryContext (cron side, #92460) OR when the active requester wake fails and the in-memory deliveryTarget is unavailable (subagent side, #92076), the route-registry fallback surfaces the target that was registered at task prep / first delivery-target resolution. Long child results are bounded via capDirectTextContent (head 2600 + tail 1000, max 4000) so requester transcript writes are not lock-amplified.

Real environment tested: Linux 4.19.112-2.el8.x86_64, Node v22.22.0, pnpm exec tsx against the production module + production code paths. Two standalone repro scripts run against a real node:sqlite database; no vitest mocks for the registry.

Exact steps or command run after this patch:

$ node --import tsx scripts/repro/issue-92460-task-completion-route.mts
$ node --import tsx scripts/repro/issue-92076-subagent-direct-fallback.mts
$ node scripts/run-vitest.mjs \
    src/cron/isolated-agent.task-completion-route.test.ts --run
$ node scripts/run-vitest.mjs \
    "src/cron/isolated-agent/delivery-dispatch.double-announce.test.ts" \
    "src/cron/isolated-agent/delivery-dispatch.named-agent.test.ts" --run
$ node scripts/run-vitest.mjs \
    src/agents/subagent-announce-delivery.test.ts --run

Evidence after fix:

#92460 live proof (5/5 steps pass):

$ pnpm exec tsx scripts/repro/issue-92460-task-completion-route.mts
  temp state dir: /tmp/openclaw-92460-proof-QUYEhJ

=== Step 1: cron prep registers a completion route ===
  ✅ registered route for manual:170c387d-a9a7-45d3-ba4f-944f48a5755e:1781142380676:1
  ✅ resolved: channel=webchat to=controller

=== Step 2: simulate #92460 failure (session entry deliveryContext missing) ===
  session entry has NO deliveryContext (this is the bug)
  announce deliverer queries session entry → empty, falls back to registry
  ✅ fallback route resolved: webchat → controller

=== Step 3: announce settles successfully and retires the route ===
  ✅ noted delivered attempt
  ✅ retired route
  ✅ resolve returns null post-retire

=== Step 4: gateway restart simulation (close + reopen state DB) ===
  ✅ registered subagent route (pre-restart)
  SQLite file present: /tmp/openclaw-92460-proof-QUYEhJ/state/openclaw.sqlite (4096 bytes)
  ✅ post-restart row found via raw node:sqlite connection

=== Step 5: orphan pruning simulates `openclaw doctor --fix` ===
  ✅ pruned 1 orphan(s) older than 5 min
  ✅ orphan removed

PASS: design is sound.

#92076 live proof (5/5 steps pass):

$ pnpm exec tsx scripts/repro/issue-92076-subagent-direct-fallback.mts
  temp state dir: /tmp/openclaw-92076-proof-odPDgC

=== Step 1: pre-register route in task_completion_routes ===
  ✅ registered route for subagent:announce:92076-route-fallback:1

=== Step 2: drive deliverer through no_active_run + missing deliveryTarget ===
[warn] Active requester session could not be woken for subagent completion; falling back to requester-agent handoff: active requester session could not be woken: queue_message_failed reason=no_active_run sessionId=requester-session-92076 gatewayHealth=live

=== Step 3: assert route-registry fallback fired with bounded text ===
  ✅ active wake attempted (queue calls: 1)
  ✅ sendMessage called via registry fallback (1 call)
  ✅ text was bounded: original 5001 chars → sent 3733 chars
  ✅ result reports delivered: true via direct path

=== Step 4: route was retired in finally ===
  ✅ route retired (idempotent confirm — second retire is a no-op)

PASS: design is sound.

Observed result after fix:

What was not tested: A live openclaw cron run / subagent run was not executed; the repros use the production module + real SQLite but the cron and subagent orchestration is not invoked end-to-end. The repros also do not exercise the orphan self-lock path (#92395), which is explicitly out of scope for this PR (see below).

Verification

# Cron integration test
$ node scripts/run-vitest.mjs \
    src/cron/isolated-agent.task-completion-route.test.ts --run
 Test Files  1 passed (1)
      Tests  4 passed (4)

# Cron dispatch regression (no break)
$ node scripts/run-vitest.mjs \
    "src/cron/isolated-agent/delivery-dispatch.double-announce.test.ts" \
    "src/cron/isolated-agent/delivery-dispatch.named-agent.test.ts" --run
 Test Files  2 passed (2)
      Tests  80 passed (80)

# Subagent deliverer regression (no break)
$ node scripts/run-vitest.mjs \
    src/agents/subagent-announce-delivery.test.ts --run
 Test Files  1 passed (1)
      Tests  97 passed (97)

# Live repros
$ pnpm exec tsx scripts/repro/issue-92460-task-completion-route.mts
PASS: design is sound.

$ pnpm exec tsx scripts/repro/issue-92076-subagent-direct-fallback.mts
PASS: design is sound.

# Build (no warnings)
$ pnpm build

Out of scope (explicitly tracked separately)

Issues

Branch

fix/cron-subagent-announce-fallback based on feat/task-completion-routes-module (PR 1).

Adds a new 'task_completion_routes' table to the shared state DB and a
src/infra/task-completion-route.ts module exposing four idempotent
operations:

- registerTaskCompletionRoute(input) - called once at task prep / first
  delivery-target resolution
- resolveTaskCompletionRoute(taskId) - called by the deliverer as a
  fallback when in-memory routing is missing
- noteRouteDeliveryAttempt(taskId, status) - debug telemetry
- retireTaskCompletionRoute(taskId) - called in finally blocks so
  unretired rows do not accumulate

This is foundation only - no behavior change. No call site is wired to
the registry yet. That is the next PR (which will close openclaw#92460, openclaw#92076,
and openclaw#93323 by hooking the registry into the cron and subagent
completion delivery paths).

Schema:
- New table task_completion_routes(task_id PK, source, channel, to_target,
  account_id, thread_id, registered_at, retired_at, delivery_attempts,
  last_delivery_status, last_delivery_at)
- Two partial indexes: active lookup (retired_at IS NULL) and
  source+age for orphan pruning

Module guarantees:
- register is idempotent on duplicate task_id
- resolve returns null for retired rows
- note and retire are idempotent
- pruneOrphanedRoutes deletes unretired rows older than threshold

Tests: 9 unit tests covering register, resolve, note, retire, prune,
durability (rows survive across DB connections), and validation (missing
taskId/source throw).
…allback

Wires the new task_completion_routes SQLite registry (from
feat/task-completion-routes-module, PR 1) into the cron and subagent
completion delivery paths. The registry is the authoritative fallback
when the in-memory deliveryContext is lost between prep and delivery.

Closes three P1 issues that all share the same root cause (no durable
per-task record of the routable target resolved at task start):

- openclaw#92460: cron completion announcer drops delivery.channel when the
  session entry's deliveryContext is never persisted. The cron hookup
  registers the resolved target at dispatchCronDelivery start.
- openclaw#92076: subagent completion delivery fails when the requester is
  inactive and the handoff path is unavailable. The subagent hookup
  registers the target at first delivery-target resolution and adds
  a registry fallback between the active-requester-wake failure and
  the existing generated-media path. Long child output is bounded
  via capDirectTextContent (head 2600 + tail 1000, max 4000) to
  avoid lock-amplifying transcript writes.
- openclaw#93323: subagent-announce-delivery drops at all spawn depths; the
  same registry works at all spawn depths, so this fix covers it.

Cron hookup (src/cron/isolated-agent/delivery-dispatch.ts, +34):
- registerTaskCompletionRoute at the top of dispatchCronDelivery when
  resolvedDelivery.ok (uses runSessionKey as taskId)
- retireTaskCompletionRoute in finishSilentReplyDelivery and the
  deliverViaDirectAndCleanup finally block (both wrapped in
  try/catch to never mask the return value)

Subagent hookup (src/agents/subagent-announce-delivery.ts, +112):
- capDirectTextContent with MAX=4000, HEAD=2600, TAIL=1000
  constants. Applied inside deliverTextCompletionDirect so long
  child output is bounded to head + truncated-marker + tail.
- Route-registry fallback (line 1441) inserted after
  activeRequesterWakeFailed = true. When in-memory deliveryTarget.deliver
  is false but the registry has a routable target, consult the
  registry and dispatch via deliverTextCompletionDirect. Gated by
  isDirectMessageDeliveryTarget so we never accidentally route to a
  non-direct target picked up from a stale record.
- retireTaskCompletionRoute in a finally block so the route is
  always retired, even on throw paths. Idempotent.

Tests + live proof:
- 4 cron integration tests in
  src/cron/isolated-agent.task-completion-route.test.ts (session
  entry wins, registry is the fallback, finally retires on both
  success and throw, orphan prune eligibility)
- 80 cron dispatch tests (no regression)
- 97 subagent deliverer tests (no regression)
- 2 standalone live repros that exercise the full failure cascade
  against a real SQLite state DB (real channel, not import-test)

Orphan self-lock (openclaw#92395) and ACP / media-understanding completion
paths are explicitly out of scope; same module covers them with a
3-line register + retire pair each, follow-up PR per source value.
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

Heads up for reviewers: this PR's base is main, so the diff shows 11 files. The 5 foundation files (schema + 2 auto-gen + module + module test) are from PR 1 (#93898). The actual hookup changes added by this PR are the other 6 files.

To see only the 6 hookup files added by this PR, change the base to feat/task-completion-routes-module in the PR view's "Files changed" tab. The branch exists in the personal fork (wangmiao0668000666:feat/task-completion-routes-module).

@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

Closing this PR together with #93898. The 2-PR split (foundation + hookups) did not deliver the intended benefit because cross-fork base change is not supported by GitHub PR API, so reviewers see all 11 files vs main regardless of which PR is opened.

Closing this PR to start fresh: the cron + subagent hookup code (delivered here against the task_completion_routes registry) will be repackaged as a single PR that closes #92460, #92076, #93323 directly. The new PR will have:

  • 1 commit (not 2)
  • All 6 hookup files + the foundation in a single diff against main
  • A clearer 'Closes #X' line at the top of the body
  • No more 11-file confusion

Branch is preserved at wangmiao0668000666:fix/cron-subagent-announce-fallback for reference. Lessons learned are captured in the updated contribution skill at ~/.claude/skills/openclaw-contribution-guide/SKILL.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling proof: supplied External PR includes structured after-fix real behavior proof. scripts Repository scripts size: XL

Projects

None yet

1 participant