Summary

• Problem: Gateway restarts had no restart-scoped trace that showed where time was spent across drain, shutdown, next start, and ready phases.
• Why it matters: Slow or degraded restarts were hard to diagnose from logs because close-path and startup timings were split across generic shutdown/startup output.
• What changed: Add OPENCLAW_GATEWAY_RESTART_TRACE support for parseable restart trace lines, covering restart signal, active-work drain, close-path phases, next start, ready spans, and memory metrics.
• What did NOT change (scope boundary): This does not add a standalone benchmark script, change restart budgets, change auth/config behavior, or change the gateway protocol.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes N/A
• Related N/A
• This PR fixes a bug or regression

Real behavior proof (required for external PRs)

Behavior addressed: With OPENCLAW_GATEWAY_RESTART_TRACE=1, a real dev gateway restart now emits parseable restart trace spans and still recovers /healthz and /readyz.

Real environment tested: Local macOS checkout on branch gw-restart-trace-bench, commit 8129cdd78a, Node v24.14.0, built dist/entry.js from this patch, dev gateway on loopback port 19984.

Exact steps or command run after this patch: Started node dist/entry.js --dev gateway run --auth none --bind loopback --port 19984 --force --allow-unconfigured --verbose with OPENCLAW_GATEWAY_RESTART_TRACE=1 OPENCLAW_NO_RESPAWN=1 FORCE_COLOR=0, probed /healthz and /readyz, sent SIGUSR1, waited for restart trace lines, then probed /healthz and /readyz again.

Evidence after fix:

{
  "initialHealthz": 200,
  "initialReadyz": 200,
  "postRestartHealthz": 200,
  "postRestartReadyz": 200,
  "startupMs": 2342,
  "signalLine": "restart trace: restart.signal.received 0.0ms total=0.0ms signal=SIGUSR1 reason=SIGUSR1 force=false waitMs=default",
  "closeLine": "restart trace: restart.close.total 58.0ms total=67.0ms reason=gateway_restarting restartExpectedMs=1500.0 rssMb=648.4 heapTotalMb=441.8 heapUsedMb=332.9 externalMb=7.1 arrayBuffersMb=0.5",
  "nextStartLine": "restart trace: restart.next-start 2.3ms total=69.3ms",
  "readyLine": "restart trace: restart.ready 362.1ms total=431.3ms rssMb=653.1 heapTotalMb=452.8 heapUsedMb=358.7 externalMb=7.1 arrayBuffersMb=0.5"
}

Observed result after fix: The gateway returned HTTP 200 before and after restart, emitted restart signal, close total, next-start, and ready trace lines, and shut down cleanly after the proof run.

What was not tested: Full-process respawn trace output, Linux/WSL2 runtime proof, and broad Testbox/Crabbox gates.

Root Cause (if applicable)

For bug fixes or regressions, explain why this happened, not just what changed. Otherwise write N/A. If the cause is unclear, write Unknown.

• Root cause: N/A. This is an observability feature, not a bug/regression fix.
• Missing detection / guardrail: Restart trace output did not previously have focused tests for parseable close and ready spans.
• Contributing context (if known): Existing startup trace and shutdown logs exposed related facts, but not as one restart-scoped timeline.

Regression Test Plan (if applicable)

For bug fixes or regressions, name the smallest reliable test coverage that should catch this. Otherwise write N/A.

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file: src/gateway/server-close.test.ts, with existing restart path coverage in src/cli/gateway-cli/run-loop.test.ts.
• Scenario the test should lock in: Enabling OPENCLAW_GATEWAY_RESTART_TRACE emits parseable close-path spans and ready child spans without shortening the parent ready span.
• Why this is the smallest reliable guardrail: The formatter and close/ready trace hooks can be verified without starting every channel/plugin runtime.
• Existing test that already covers this (if any): Existing run-loop tests cover restart drain, restart intent, SIGUSR1, and respawn behavior.
• If no new test is added, why not: N/A.

User-visible / Behavior Changes

When OPENCLAW_GATEWAY_RESTART_TRACE=1 is set, gateway restart logs include parseable restart trace: lines for drain, close, next-start, and ready timing. Default logging is unchanged when the env var is unset.

Diagram (if applicable)

Before:
SIGUSR1 restart -> drain/close/startup logs split across generic lifecycle output

After:
SIGUSR1 restart -> restart.signal -> restart.drain -> restart.close.* -> restart.next-start -> restart.ready -> health/ready recovered

Security Impact (required)

• New permissions/capabilities? (Yes/No) No
• Secrets/tokens handling changed? (Yes/No) No
• New/changed network calls? (Yes/No) No
• Command/tool execution surface changed? (Yes/No) No
• Data access scope changed? (Yes/No) No
• If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

• OS: macOS
• Runtime/container: Node v24.14.0, local dist/entry.js
• Model/provider: N/A
• Integration/channel (if any): None; dev gateway only
• Relevant config (redacted): --dev --auth none --bind loopback --port 19984 --allow-unconfigured, OPENCLAW_GATEWAY_RESTART_TRACE=1, OPENCLAW_NO_RESPAWN=1

Steps

1. Run node scripts/run-vitest.mjs src/gateway/server-close.test.ts src/cli/gateway-cli/run-loop.test.ts.
2. Start the dev gateway from dist/entry.js with restart tracing enabled.
3. Probe /healthz and /readyz.
4. Send SIGUSR1.
5. Confirm restart trace lines and probe /healthz and /readyz again.

Expected

• Targeted tests pass.
• Gateway stays recoverable after restart.
• Restart trace logs include signal, close total, next-start, and ready timing.

Actual

• node scripts/run-vitest.mjs src/gateway/server-close.test.ts src/cli/gateway-cli/run-loop.test.ts: passed, 2 files / 45 tests.
• Runtime proof returned HTTP 200 for initial and post-restart /healthz and /readyz.
• Runtime proof emitted parseable restart trace lines shown above.

Evidence

Attach at least one:

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

• Verified scenarios: Targeted restart tests passed; local dev gateway restarted in-process with trace enabled and recovered health/readiness.
• Edge cases checked: Close-path trace span format, ready child span format, parent ready span duration, memory metric formatting.
• What you did not verify: Full-process respawn, Linux/WSL2, broad Testbox/Crabbox, and production managed service restart.

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

• Backward compatible? (Yes/No) Yes
• Config/env changes? (Yes/No) Yes, optional OPENCLAW_GATEWAY_RESTART_TRACE=1 enables restart trace logging.
• Migration needed? (Yes/No) No
• If yes, exact upgrade steps: N/A

Risks and Mitigations

• Risk: Wrapping shutdown phases for measurement could accidentally perturb close ordering.
  • Mitigation: The implementation preserves the existing close sequence and targeted shutdown/restart tests passed.
• Risk: Trace output could become noisy when enabled.
  • Mitigation: It is behind OPENCLAW_GATEWAY_RESTART_TRACE; default behavior is unchanged.