[codex] add gateway stall diagnostics#77542
Conversation
|
Thanks for the context here. I swept through the related work, and this is now duplicate or superseded. Keep open: the protected Canonical path: Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review. So I’m closing this here because the remaining work is already tracked in the canonical issue. Review detailsBest possible solution: Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review. Do we have a high-confidence way to reproduce the issue? Not applicable: this is a diagnostic feature/default-change PR, not a bug report with a current-main failing reproduction path. The blockers come from source comparison, live PR state, Codex source inspection, and missing real behavior proof. Is this the best way to solve the issue? No, not as submitted. Current main already carries the central diagnostics while keeping gateway-watch tracing opt-in, so the maintainable path is a refresh or replacement that preserves that default unless maintainers explicitly approve changing it. Security review: Security review cleared: No concrete security or supply-chain concern found; the diff changes diagnostics, script-runner flags, docs, tests, and Slack log formatting without dependency, workflow, secret, permission, or download changes. AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against 010b61746379. |
1ae7805 to
cc1e702
Compare
|
Closing as stale and superseded by current diagnostics. Current |
Summary
This adds higher-signal gateway stall diagnostics for slow Telegram turns where the transport is healthy but the Gateway/embedded Codex run stops making useful progress.
Changes:
diagnostic.phase.completedfor Gateway and CLI startup spans.diagnostic.liveness.warningevents and logs.terminalProgressStale=truewhen the last bridge progress looked terminal, such asrawResponseItem/completed, but the Gateway still considers the embedded run active.OPENCLAW_TRACE_SYNC_IO=1support inscripts/run-node.mjsto pass Node--trace-sync-iothrough the normal source runner for startup I/O investigations.OPENCLAW_TRACE_SYNC_IO=1by default forpnpm gateway:watchGateway children, while preserving explicit overrides likeOPENCLAW_TRACE_SYNC_IO=0.unknown errorinstead ofundefined.Why
The Telegram path was not the bottleneck: transport probes were fast, but Gateway sessions could remain active after the Codex bridge had already reported terminal-looking progress. Existing liveness warnings only showed aggregate counts (
active,waiting,queued) plus event-loop/CPU numbers, so diagnosing the next stall required manual tmux/log archaeology.With this change, the next warning should say which phase was active, what recent startup/runtime spans completed, which session/work label is blocking, and whether the bridge emitted terminal progress without closing the embedded run. Gateway watch also records sync I/O traces by default so local dev stalls have file/module-loader evidence without needing to remember the env flag.
Operator impact
No behavior change to scheduling, Telegram delivery, recovery thresholds, or restart policy.
Manual one-off source-runner tracing remains available:
Gateway watch enables the same trace mode by default for the watched child. To suppress Node sync I/O trace output in watch mode:
Verification
pnpm test src/logging/diagnostic.test.tspnpm test src/infra/run-node.test.tspnpm test src/infra/watch-node.test.tspnpm test src/infra/gateway-watch-tmux.test.tspnpm tsgo:corepnpm tsgo:core:testpnpm format:docs:checkpnpm docs:check-mdxpnpm tsgo:extensions:testpnpm tsgo:extensionspnpm test extensions/slack/src/monitor/provider.reconnect.test.tsgit diff --check