Skip to content

fix(gateway): preserve restart continuations after reboot#63406

Merged
obviyus merged 8 commits into
openclaw:mainfrom
VACInc:codex/restart-sentinel-continuation
Apr 22, 2026
Merged

fix(gateway): preserve restart continuations after reboot#63406
obviyus merged 8 commits into
openclaw:mainfrom
VACInc:codex/restart-sentinel-continuation

Conversation

@VACInc

@VACInc VACInc commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: restart-sentinel wake preserved the restart notice, but requested post-restart continuation work could be dropped, delayed, or lose context after reboot.
  • Why it matters: gateway.restart is used specifically to resume work after reboot, so silent continuation loss breaks the main use case and is hard to diagnose.
  • What changed: the sentinel now carries an optional one-shot continuation, gateway.restart can request it, restart wake dispatches it back into the same session/thread, and the edge cases raised in review now warn or re-wake instead of silently failing.
  • What did NOT change (scope boundary): this does not add a durable continuation queue, change restart authorization, or broaden routing beyond the existing session-scoped restart sentinel flow.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause (if applicable)

  • Root cause: the original restart-sentinel path only needed to enqueue and wake a restart notice; once continuation support was added, the later continuation enqueue/dispatch path inherited assumptions that were only safe for the original single-event wake flow.
  • Missing detection / guardrail: there was no regression coverage for post-restart continuation delivery semantics, especially around schema shape, missing routing/session state, timestamp preservation, and systemEvent wake timing.
  • Contributing context (if known): gateway.restart allows continuation requests without an explicit sessionKey, and systemEvent continuation enqueue happens later than the original restart wake.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/gateway/server-restart-sentinel.test.ts, src/agents/tools/gateway-tool.test.ts, src/infra/restart-sentinel.test.ts
  • Scenario the test should lock in: restart continuation survives reboot in the same routed context, warns instead of silently disappearing when session/route state is missing, keeps stamped agent context, and re-wakes systemEvent continuations.
  • Why this is the smallest reliable guardrail: the failures happen in the orchestration seam between persisted sentinel payloads, restart wake scheduling, session routing, and tool schema generation, so helper-only tests would miss them.
  • Existing test that already covers this (if any): None before this PR.
  • If no new test is added, why not: N/A

User-visible / Behavior Changes

  • gateway.restart can resume a one-shot continuation after reboot in the same session/thread instead of only sending a restart notice.
  • agentTurn continuations now preserve the stamped agent-facing text used for time-sensitive follow-up work.
  • systemEvent continuations now trigger their own wake instead of depending on the earlier restart-notice wake timing.
  • If restart continuation cannot run because route or session state is missing, the failure is now surfaced as a warning instead of being silently dropped.

Diagram (if applicable)

Before:
[gateway.restart + continuation] -> [restart notice wake] -> [continuation may be dropped/delayed]

After:
[gateway.restart + continuation] -> [restart notice wake] -> [continuation dispatched or warning logged] -> [result visible]

Security Impact (required)

  • New permissions/capabilities? (Yes/No) Yes
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) Yes
  • Data access scope changed? (Yes/No) No
  • If any Yes, explain risk + mitigation: the owner-only gateway.restart tool now accepts optional continuation inputs and can resume one follow-up turn after reboot. Risk is constrained by the existing restart auth boundary, one-shot sentinel consumption, same-session routing, and explicit warning/fail-fast handling when continuation delivery is unavailable.

Repro + Verification

Environment

  • OS: Linux
  • Runtime/container: local Node/pnpm workspace
  • Model/provider: N/A
  • Integration/channel (if any): mocked channel routing in targeted tests
  • Relevant config (redacted): default local test config

Steps

  1. Invoke gateway.restart with continuationMessage in a routed session.
  2. Let restart sentinel wake run after startup.
  3. Exercise agentTurn, systemEvent, missing-route, and missing-sessionKey continuation cases.

Expected

  • Continuation resumes in the same routed context after reboot, or logs a visible warning when it cannot be delivered.
  • systemEvent continuation requests a wake after enqueue.
  • agentTurn continuation keeps stamped BodyForAgent context.

Actual

  • Before this fix set, continuation edge cases could be silently dropped, delayed, or lose the stamped agent context.
  • After this fix set, the tested restart continuation paths behave deterministically and fail visibly when delivery is unavailable.

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

  • Verified scenarios: ran targeted restart-sentinel/tool tests covering agent-turn continuation dispatch, systemEvent continuation wake, missing-route warning, missing-sessionKey warning, one-shot consumption, and flat enum tool schema generation.
  • Edge cases checked: timestamp preservation in BodyForAgent, no-route fail-fast path, no-sessionKey warning path, and delayed systemEvent wake after restart notice delivery.
  • What you did not verify: live restart behavior against a real channel/integration.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) No
  • Migration needed? (Yes/No) No
  • If yes, exact upgrade steps:

Risks and Mitigations

  • Risk: continuation behavior now depends on the persisted restart sentinel shape and the new gateway tool inputs staying aligned.
    • Mitigation: targeted tests cover sentinel payload, tool schema shape, restart wake dispatch, and the reviewed edge cases.
  • Risk: systemEvent continuation now schedules an extra heartbeat wake for the same session.
    • Mitigation: the wake is narrowly scoped to the continuation session and covered by regression test coverage in src/gateway/server-restart-sentinel.test.ts.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime agents Agent runtime and tooling size: L labels Apr 8, 2026
@VACInc
VACInc marked this pull request as ready for review April 8, 2026 22:44
@greptile-apps

greptile-apps Bot commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR extends the restart-sentinel flow with an optional one-shot continuation payload (systemEvent or agentTurn). After the gateway restarts and delivers the existing restart notice, it now dispatches the continuation into the same resumed session and routed thread. Two new schema parameters (continuationMessage and continuationKind) are added to the gateway tool's restart action. The prior Type.Union schema violation for continuationKind is correctly resolved using optionalStringEnum, and the sentinel remains one-shot because consumeRestartSentinel deletes the file before any dispatch.

Confidence Score: 5/5

Safe to merge — no P0/P1 findings; the prior schema violation concern has been correctly addressed.

All six changed files are clean. The Type.Union/anyOf schema issue flagged in the previous review round is resolved with optionalStringEnum. Continuation dispatch is one-shot (sentinel is consumed before any dispatch), error paths are caught and logged rather than bubbled, and test coverage spans the main agentTurn path, delivery failure, missing-route fallback, and the no-sessionKey warning branch.

No files require special attention.

Vulnerabilities

No security concerns identified. The continuationMessage and continuationKind inputs are sanitized via normalizeOptionalString and schema-level enum validation before being written to the sentinel file. The sentinel file is consumed (deleted) before dispatch, preventing replay. The CommandAuthorized: true flag is set on the injected inbound context, which is consistent with how other internal-sourced agent turns are handled in this codebase.

Reviews (2): Last reviewed commit: "gateway: handle restart continuation edg..." | Re-trigger Greptile

Comment thread src/agents/tools/gateway-tool.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 972783881a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/server-restart-sentinel.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8575755c0c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/server-restart-sentinel.ts
Comment thread src/gateway/server-restart-sentinel.ts Outdated
@VACInc

VACInc commented Apr 8, 2026

Copy link
Copy Markdown
Contributor Author

@greptile-apps review

@greptile-apps

greptile-apps Bot commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Tip:

Greploop — Automatically fix all review issues by running /greploops in Claude Code. It iterates: fix, push, re-review, repeat until 5/5 confidence.

Use the Greptile plugin for Claude Code to query reviews, search comments, and manage custom context directly from your terminal.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6603b42d53

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/server-restart-sentinel.ts
@VACInc VACInc changed the title [codex] Add restart sentinel continuations fix(gateway): preserve restart continuations after reboot Apr 8, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 559e33847d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/server-restart-sentinel.ts Outdated
@VACInc

VACInc commented Apr 8, 2026

Copy link
Copy Markdown
Contributor Author

@BunsDev tagging you as I know you were interested/helped merge in the last time I fixed the restart flow. I also know there's a broader effort to stabilize and get things more secure so I'm sure that's why it was lost in the cleanup.

That said, this is a bit more elegant than last time so hopefully it persists through future changes 🦞

@VACInc
VACInc force-pushed the codex/restart-sentinel-continuation branch 6 times, most recently from 8c4ab51 to 6691f05 Compare April 15, 2026 10:57

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6691f05248

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/server-restart-sentinel.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d69efec31

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/server-restart-sentinel.ts Outdated
@VACInc
VACInc force-pushed the codex/restart-sentinel-continuation branch from 8d69efe to b4b53c0 Compare April 21, 2026 02:32

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b4b53c0e33

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/server-restart-sentinel.ts Outdated
@obviyus
obviyus force-pushed the codex/restart-sentinel-continuation branch from b4b53c0 to a755686 Compare April 22, 2026 13:00
@obviyus obviyus self-assigned this Apr 22, 2026
@obviyus
obviyus force-pushed the codex/restart-sentinel-continuation branch from 03c5f62 to 3aa09d9 Compare April 22, 2026 14:33

@obviyus obviyus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified gateway restart continuations persist through reboot, resume the same session, and route replies back to the original channel/thread. Manual smoke covered PR-linked gateway restart and a valid restart-sentinel continuation; local tests cover stale routes, synthetic reply IDs, system-event wakeups, exhausted notice retries, and session-route fallback.

Maintainer follow-up: hardened route resolution, resolved current review threads, added restart-continuation edge coverage, documented the tool instruction, and added the Unreleased changelog entry.

Local gate: pnpm test src/gateway/server-restart-sentinel.test.ts src/agents/tools/gateway-tool.test.ts src/infra/restart-sentinel.test.ts; pnpm check:changed.

@obviyus
obviyus merged commit 962b25b into openclaw:main Apr 22, 2026
8 of 9 checks passed
@obviyus

obviyus commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Landed on main.

Thanks @VACInc.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3aa09d9f7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Timestamp: Date.now(),
Provider: route.channel,
Surface: route.channel,
ChatType: "direct",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve non-DM chat type in continuation context

Do not force restart agentTurn continuations to ChatType: "direct"; this misclassifies turns resumed from group/channel conversations. The reply pipeline keys behavior off ctx.ChatType (for example, group-context injection and related routing/prompt logic), so a continuation from a threaded/group route can be processed as a DM and produce the wrong follow-up behavior after reboot. Derive ChatType from the resolved session/route instead of hardcoding direct.

Useful? React with 👍 / 👎.

medikoo pushed a commit to medikoo/openclaw that referenced this pull request Apr 24, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
zhonghe0615 pushed a commit to zhonghe0615/openclaw that referenced this pull request May 7, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
globalcaos pushed a commit to globalcaos/tinkerclaw that referenced this pull request May 13, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
…anks @VACInc)

* gateway: add restart continuation sentinel

* gateway: address restart continuation review

* gateway: handle restart continuation edge cases

* gateway: keep restart continuations on threaded delivery path

* fix(gateway): harden restart continuation routing

* test(gateway): cover restart continuation edge cases

* docs(agent): clarify restart continuation usage

* fix: preserve restart continuations after reboot (openclaw#63406) (thanks @VACInc)

---------

Co-authored-by: VACInc <[email protected]>
Co-authored-by: Ayaan Zaidi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling gateway Gateway runtime size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support continuation prompt in restart sentinel

2 participants