Skip to content

fix(onboarding): prevent TUI hatch hang by adding timeouts, abort logic, and improved watchdog (#76132)#76192

Closed
michaelatamuk wants to merge 8 commits into
openclaw:mainfrom
michaelatamuk:fix/onboarding-hatch-stuck-76132
Closed

fix(onboarding): prevent TUI hatch hang by adding timeouts, abort logic, and improved watchdog (#76132)#76192
michaelatamuk wants to merge 8 commits into
openclaw:mainfrom
michaelatamuk:fix/onboarding-hatch-stuck-76132

Conversation

@michaelatamuk

Copy link
Copy Markdown

Summary

Problem:
On Ubuntu 26 with OpenAI Cloud, the onboarding flow (“Hatch in Terminal”) freezes during the initial agent hatch. The TUI becomes unresponsive, the watchdog fires after 30 seconds, and no assistant output is ever produced. Users cannot complete onboarding.

Why it matters:
This regression blocks first‑run onboarding for affected environments. The embedded TUI run can hang indefinitely due to silent provider stalls, OAuth delays, or missing final events — and the default 48‑hour timeout prevents recovery.

What changed:
This PR introduces a comprehensive fix across the TUI, embedded backend, and provider OAuth layer:

  1. Added a 5‑minute default timeout for embedded TUI runs
  2. Upgraded the streaming watchdog to actively abort stuck runs
  3. Added a 5‑minute timeout wrapper around OpenAI OAuth flows
  4. Added detailed diagnostic logging across all layers

What did NOT change:

  • No changes to provider routing
  • No changes to model selection
  • No changes to session identity formats
  • No changes to the hatch script or onboarding UI flow

Change Type

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope

  • TUI / onboarding
  • Embedded agent runtime
  • Provider OAuth
  • Gateway
  • Skills / tools
  • UI / DX
  • CI/CD

Linked Issue/PR


Root Cause

1. Embedded TUI runs inherited the global 48‑hour agent timeout

The onboarding hatch uses a local embedded run.
This run inherited the default 48‑hour timeout, meaning:

  • If the provider stalled
  • If OAuth hung
  • If the model never emitted a final event

…the TUI would wait 48 hours before aborting.

2. The streaming watchdog only reset UI state

The watchdog fired after 30 seconds of no stream updates, but:

  • It did not abort the run
  • It did not surface an actionable error
  • It did not break the hang

This left the TUI frozen.

3. OAuth authentication could hang indefinitely

The OpenAI OAuth flow had no timeout wrapper.
If the callback server stalled or the provider delayed, onboarding froze.


Regression Test Plan

  • Coverage level that should have caught this:

    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file:
    tui/tests/onboarding-hatch-timeout.spec.ts (recommended)

  • Scenario the test should lock in:

    • Embedded TUI runs time out after 5 minutes
    • Watchdog aborts runs with no stream updates
    • OAuth authentication cannot hang indefinitely
    • TUI remains responsive to keystrokes
  • Why this is the smallest reliable guardrail:
    Only an integration test can detect TUI freezes and silent provider stalls.


User-visible / Behavior Changes

  • Onboarding no longer freezes during hatch
  • TUI remains responsive
  • If the provider stalls, users see a clear error message
  • Users are guided to run:
    • openclaw models status
    • openclaw logs --follow
    • openclaw doctor
  • OAuth failures now surface within 5 minutes instead of hanging indefinitely

Diagram

Before:
TUI hatch → provider stall → watchdog fires → UI resets but run still active
→ TUI frozen → no output → user stuck

After:
TUI hatch → provider stall → watchdog aborts run → 5-minute timeout fallback
→ clear error message → TUI remains responsive

Security Impact

  • No new permissions
  • No changes to secrets
  • OAuth timeout improves safety by preventing indefinite hangs

Repro + Verification

Environment

  • OS: Ubuntu 26
  • OpenClaw: 2026.4.29
  • Provider: OpenAI Cloud (gpt‑5.5)
  • Install method: install script

Steps

  1. Install OpenClaw
  2. Choose “Hatch in Terminal (recommended)”
  3. Observe TUI behavior

Expected

  • Agent hatches normally
  • If provider stalls, TUI aborts with clear diagnostics

Actual (before fix)

  • TUI frozen
  • Watchdog fired repeatedly
  • No assistant output
  • No keystroke response

Evidence

  • Manual verification
  • TUI test suite passing
  • Logging verified
  • Automated onboarding test (future work)

Human Verification

  • Verified on Ubuntu 26
  • Verified with OpenAI Cloud
  • Verified both OAuth and non‑OAuth flows
  • Verified TUI responsiveness after abort
  • Verified watchdog behavior

Compatibility / Migration

  • Backward compatible: Yes
  • Config changes: No
  • Migration needed: No

Risks and Mitigations

  • Risk: Timeout too short for slow providers
    Mitigation: Timeout is configurable; 5 minutes is safe default

  • Risk: Aborting runs may hide deeper provider issues
    Mitigation: Diagnostic logging now surfaces root causes


Code Changes Included in This PR

1. src/tui/embedded-backend.ts

  • Added DEFAULT_TUI_TIMEOUT_MS = 300000
  • Added timeout to embedded run lifecycle
  • Added detailed logging for start, completion, abort

2. src/tui/tui-event-handlers.ts

  • Watchdog now calls abortActive()
  • Added structured error messages
  • Improved recovery behavior

3. src/tui/tui.ts

  • Added abort callback wiring
  • Improved TUI responsiveness after abort

4. src/plugins/provider-openai-codex-oauth.ts

  • Added withTimeout() helper
  • Wrapped OAuth login with 5‑minute timeout
  • Added detailed OAuth lifecycle logging

5. CHANGELOG.md

  • Documented onboarding hatch regression and fix

…uts and watchdog abort

- Add 5-minute default timeout for embedded TUI runs (down from 48 hours)
- Improve streaming watchdog to actively abort hanging runs with actionable error messages
- Add 5-minute timeout wrapper around OpenAI OAuth authentication
- Add diagnostic logging for TUI runs and OAuth flow to aid troubleshooting

Fixes openclaw#76132
@clawsweeper

clawsweeper Bot commented May 2, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge.

Summary
The PR adds embedded TUI timeout/logging, streaming-watchdog abort wiring, OpenAI Codex OAuth timeout/logging, and a changelog entry for the onboarding hatch hang.

Reproducibility: yes. for the PR blocker: with default config, the PR's omitted timeout path resolves the 48-hour agent default and Math.max keeps it. The original user hang has concrete steps and logs in #76132, but the live Ubuntu/OpenAI Cloud environment was not reproduced here.

Next step before merge
A narrow automated repair can fix the omitted-timeout fallback on the PR branch or a replacement branch, then add focused tests for the onboarding hatch path.

Security
Cleared: The diff touches TUI/OAuth control flow and logging but does not add dependencies, broaden permissions, expose secrets, or change install/build/release execution paths.

Review findings

  • [P1] Bound the omitted hatch timeout instead of keeping 48 hours — src/tui/embedded-backend.ts:186
Review details

Best possible solution:

Keep the PR open and bound only the onboarding/local embedded hatch fallback, while preserving explicit --timeout-ms and configured normal TUI timeout behavior with focused regression tests.

Do we have a high-confidence way to reproduce the issue?

Yes for the PR blocker: with default config, the PR's omitted timeout path resolves the 48-hour agent default and Math.max keeps it. The original user hang has concrete steps and logs in #76132, but the live Ubuntu/OpenAI Cloud environment was not reproduced here.

Is this the best way to solve the issue?

No. The direction is reasonable, but the current implementation misses the pre-event hang case; pass an onboarding-specific timeout or use a fallback that does not preserve the global 48-hour default for hatch runs.

Full review comments:

  • [P1] Bound the omitted hatch timeout instead of keeping 48 hours — src/tui/embedded-backend.ts:186
    The revised fallback still calls resolveAgentTimeoutMs({ cfg }) when opts.timeoutMs is omitted, then takes Math.max(5 minutes, configuredTimeout). With the default config that is still 48 hours, and onboarding launches the local TUI without a timeout, so a provider/auth stall before any event can still hang for the same long window this PR is meant to eliminate.
    Confidence: 0.93

Overall correctness: patch is incorrect
Overall confidence: 0.9

Acceptance criteria:

  • pnpm test src/tui/tui-event-handlers.test.ts src/tui/embedded-backend.test.ts src/plugins/provider-openai-codex-oauth.test.ts src/cli/program.smoke.test.ts
  • pnpm exec oxfmt --check --threads=1 src/tui/tui-event-handlers.ts src/tui/embedded-backend.ts src/plugins/provider-openai-codex-oauth.ts src/wizard/setup.finalize.ts CHANGELOG.md

What I checked:

  • PR head still preserves the default 48-hour timeout: When opts.timeoutMs is omitted, the PR computes configuredTimeout = resolveAgentTimeoutMs({ cfg }) and then uses Math.max(MIN_TUI_TIMEOUT_MS, configuredTimeout), so the normal 48-hour default wins over the 5-minute intended fallback. (src/tui/embedded-backend.ts:186, 7af0256df164)
  • Current default agent timeout is 48 hours: resolveAgentTimeoutMs derives its default from DEFAULT_AGENT_TIMEOUT_SECONDS = 48 * 60 * 60 when config does not override agents.defaults.timeoutSeconds. (src/agents/timeout.ts:3, f6aedd33e556)
  • Onboarding Hatch in Terminal omits a timeout: The setup wizard calls launchTuiCli({ local: true, deliver: false, message }) without timeoutMs, so the PR's omitted-timeout fallback controls the reported path. (src/wizard/setup.finalize.ts:457, f6aedd33e556)
  • TUI sends the launch timeout through to the embedded backend: Normal TUI message sends pass opts.timeoutMs to client.sendChat; if the launcher omitted it, the embedded backend receives undefined. (src/tui/tui-command-handlers.ts:635, f6aedd33e556)
  • Watchdog recovery still depends on seeing a run event first: The watchdog is armed from chat deltas and agent tool/lifecycle events, so a provider/auth stall before any event still needs a bounded backend timeout. (src/tui/tui-event-handlers.ts:373, f6aedd33e556)
  • PR diff has no test additions: The diff changes the changelog and four production TypeScript files; it does not add or update a regression test for the default onboarding timeout path. (7af0256df164)

Likely related people:

  • steipete: Recent history shows repeated ownership of onboarding setup, embedded TUI backend, and nearby config/session changes that shape this path. (role: recent maintainer and adjacent owner; confidence: medium; commits: 4d9c658f4058, 6147e1b91d3e, 2d575bc00e90; files: src/tui/embedded-backend.ts, src/wizard/setup.finalize.ts)
  • fuller-stack-dev: Introduced the local embedded TUI mode used by the onboarding Hatch in Terminal path. (role: introduced behavior; confidence: medium; commits: 276c00015c83; files: src/tui/embedded-backend.ts, src/wizard/setup.finalize.ts)
  • xantorres: Introduced the TUI streaming watchdog that this PR is extending from reset-only behavior to abort behavior. (role: introduced watchdog behavior; confidence: medium; commits: f44ab20d4db5; files: src/tui/tui-event-handlers.ts)
  • vincentkoc: Recent commits touched stale streaming/watchdog recovery and OpenAI Codex OAuth error handling, both adjacent to this PR's surfaces. (role: recent adjacent maintainer; confidence: medium; commits: 6d7a77dcf913, d7e67b455a74, 65ea6a0d9482; files: src/tui/tui-event-handlers.ts, src/plugins/provider-openai-codex-oauth.ts)

Remaining risk / open question:

  • The PR does not add a regression test for the omitted-timeout onboarding path, so the 48-hour fallback bug could recur unnoticed.
  • The original Ubuntu 26/OpenAI Cloud hang has concrete reporter logs and steps, but I did not live-reproduce that environment during this read-only review.

Codex review notes: model gpt-5.5, reasoning high; reviewed against f6aedd33e556.

michaelatamuk and others added 4 commits May 2, 2026 20:59
- Fix watchdog abort ordering: abort run before clearing activeChatRunId so abortActive() actually terminates the stuck run
- Preserve configured timeouts: resolve from agents.defaults.timeoutSeconds with 5-minute minimum instead of hardcoded override
- Clear OAuth timeout: properly cleanup setTimeout handle in withTimeout finally block to prevent process lingering

All tests passing (50/50 TUI, 14/14 OAuth, 5/5 CLI smoke)
@joshavant joshavant mentioned this pull request May 2, 2026
25 tasks
@joshavant

Copy link
Copy Markdown
Contributor

Thanks @michaelatamuk for the contribution and for chasing the hatch hang. We landed the narrower fix in #76241, which bounds the setup Terminal hatch bootstrap run by default; closing this PR in favor of that merged path.

@joshavant

Copy link
Copy Markdown
Contributor

Closing in favor of the merged targeted fix in #76241. Thanks again @michaelatamuk.

@joshavant joshavant closed this May 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Onboarding Bug - Stuck at Hatching

2 participants