Summary

Track iMessage response latency instrumentation and follow-up performance investigation for source-reply/message-tool-only turns.

This issue captures the current evidence from a patched local worktree that included:

• PR perf(imessage): show typing sooner for slow replies #95621: iMessage early typing / cleanup patches
• PR Fix Codex message-tool-only source reply completion #95942: Codex message tool source-reply / incomplete-turn patches
• Additional local instrumentation for iMessage end-to-end phases and a Codex model-call-start marker

Observed behavior

With the Codex patch actually loaded from the worktree, iMessage ingestion and gateway routing are fast. The remaining latency is dominated by the Codex/model turn between model-call start and first tool call.

Latest live sample, redacted direct iMessage DM, model openai/gpt-5.5, launched via pnpm openclaw gateway run --verbose:

iMessage receipt -> context ready: 105 ms
gateway dispatch started: 220 ms
model selected: 724 ms
agent run started: 1.16 s
agent model call starting: 1.90 s
first before_tool_call hook: ~12.05 s
embedded run done: ~12.65 s
gateway dispatch finished: 12.75 s

Key interpretation:

• iMessage watch/debounce/process/routing/context is not the bottleneck: ~105 ms.
• Gateway dispatch is not the bottleneck: ~220 ms to dispatch start.
• Agent setup before model-call is modest: ~746 ms from agent-run-started to agent-model-call-starting.
• Main current bottleneck is the model/tool decision window: ~10.1 s from agent-model-call-starting to first before_tool_call.

Prior samples

Before verifying Codex loaded from the patched worktree, Codex was still loading from the installed npm plugin path under ~/.openclaw/npm/projects/.../@openclaw/codex. In that state, the incomplete-turn fix was not active and runs ended with:

incomplete turn detected ... tools=1 ... surfacing error to user
dispatch-from-config: final reply suppressed by sourceReplyDeliveryMode: message_tool_only ... error=true

After repointing Codex to the patched worktree build, the same flow completed cleanly with:

codex app-server turn released after terminal dynamic tool result
embedded run done ... aborted=false

No incomplete turn detected line appeared after the patched Codex path was active.

Instrumentation added locally

Repo-local files touched in the investigation branch:

• extensions/imessage/src/monitor/monitor-provider.ts
  • Adds verbose-gated imessage perf: phase logs.
  • Tracks watch receipt, debounce flush, processing, route, early typing, context, gateway dispatch, model selection, agent run start, model-call start, tool start, drops, and dispatch finish.
• src/auto-reply/get-reply-options.types.ts
  • Adds optional onAgentModelCallStart reply lifecycle callback.
• src/auto-reply/reply/agent-runner-execution.ts
  • Bridges Codex codex_app_server.lifecycle phase=turn_starting into onAgentModelCallStart.
• extensions/imessage/src/monitor.last-route.test.ts
  • Verifies the new iMessage perf phases are emitted without enabling onToolResult or changing source-reply delivery behavior.

Validation run locally

pnpm format -- src/auto-reply/get-reply-options.types.ts src/auto-reply/reply/agent-runner-execution.ts extensions/imessage/src/monitor/monitor-provider.ts extensions/imessage/src/monitor.last-route.test.ts
git diff --check
node scripts/run-vitest.mjs extensions/imessage/src/monitor.last-route.test.ts src/auto-reply/reply/agent-runner-execution.test.ts extensions/codex/src/app-server/dynamic-tools.test.ts extensions/codex/src/app-server/event-projector.test.ts src/agents/embedded-agent-message-tool-source-reply.test.ts

Result: all targeted tests passed.

Investigation notes

• The installed plugin ledger initially pinned codex to @openclaw/[email protected] under ~/.openclaw/npm/projects/..., so the local PR Fix Codex message-tool-only source reply completion #95942 commits were present in the branch but not active in the live gateway.
• Installing ./dist/extensions/codex copied a built plugin whose index.js still referenced repo-root chunks, causing a missing chunk error outside the repo dist tree.
• Symlinking ~/.openclaw/extensions/codex to the worktree dist/extensions/codex allowed the gateway to load the patched Codex plugin and resolve shared chunks.
• Current logs show a large session context/transcript, e.g. ~32k prompt tokens in the latest run and transcript bytes >2.5 MB. This may contribute to the ~10 s model decision window, but the current trace only proves the delay is after model-call start.

Open questions / next steps

• Decide whether this instrumentation should be kept, refined, or moved behind a more generic per-channel performance trace flag.
• Add a narrower Codex/provider trace point for first model output / first assistant delta, not just first tool hook.
• Measure the same iMessage prompt with smaller context or fresh session to isolate context-size contribution.
• Compare message_tool_only source-reply path with a normal final-text reply path.
• Consider an optimization for trivial iMessage replies that avoids a full tool-send decision when the expected response is simple text.
• Decide how to make local official-plugin source testing less error-prone so installed npm plugin copies do not silently shadow patched worktree code.

Impact

Affected surface: iMessage direct-message replies using Codex/OpenAI with sourceReplyDeliveryMode: message_tool_only.

Severity: performance/user experience issue. iMessage receipt and routing are fast, but user-visible response latency remains around 12-13 seconds for trivial messages in the observed session.

Frequency: reproduced across multiple live iMessage samples in this local environment.