Skip to content

fix(restart-sentinel): report persistence cleanup failures#106385

Merged
steipete merged 9 commits into
openclaw:mainfrom
zenglingbiao:fix/restart-sentinel-silent-catch
Jul 16, 2026
Merged

fix(restart-sentinel): report persistence cleanup failures#106385
steipete merged 9 commits into
openclaw:mainfrom
zenglingbiao:fix/restart-sentinel-silent-catch

Conversation

@zenglingbiao

@zenglingbiao zenglingbiao commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Restart sentinel persistence deliberately uses best-effort reads and cleanup during gateway recovery. SQLite failures and legacy-file removal failures were silently swallowed, leaving operators without any diagnostic signal when stale restart state could survive or pending recovery state could become unreadable.

Why This Change Was Made

  • Keep the shipped non-throwing contracts: clearRestartSentinel() completes, readRestartSentinel() returns null, and hasRestartSentinel() returns false on persistence failures.
  • Log each failure through the existing restart-sentinel subsystem logger and formatErrorMessage instead of raw console output.
  • Cover the complete persistence-cleanup boundary, including the legacy restart-sentinel.json removal path absorbed from fix(infra): log restart sentinel cleanup errors instead of silent catch #101687.
  • Keep internal sentinel implementation types private; no public API or config surface changes.

User Impact

Operators now get actionable restart-sentinel warnings for SQLite clear/read/check failures and failed legacy cleanup. Gateway shutdown and recovery remain best effort and retain their existing fallback values.

Evidence

  • Blacksmith Testbox: pnpm test src/infra/restart-sentinel.test.ts — 32 tests passed.
  • Real corrupt-SQLite probe on Linux: clear completed, read returned null, has returned false, and all three redacted subsystem warnings were emitted.
  • Real filesystem regression: a directory at the legacy sentinel path proves cleanup failure is logged without throwing.
  • git diff --check passed.
  • Fresh autoreview of both the fixup and its resolved current-main synthetic merge found no actionable issues (0.93 and 0.98 confidence).

@zenglingbiao
zenglingbiao force-pushed the fix/restart-sentinel-silent-catch branch from 5a82d45 to 9be7a92 Compare July 13, 2026 13:00
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jul 13, 2026
@clawsweeper

clawsweeper Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 13, 2026, 7:54 PM ET / 23:54 UTC.

Summary
The PR logs warnings from three restart-sentinel SQLite failure paths, preserves their best-effort fallback results, exports related sentinel types, and adds deterministic fault-injection tests.

PR surface: Source +10, Tests +87. Total +97 across 2 files.

Reproducibility: yes. Controlled database-boundary failures on current main reproduce the silent catches, while the same paths on the PR head emit warnings and retain their fallback results.

Review metrics: 1 noteworthy metric.

  • Failure contracts: 3 warnings added, 0 fallback results changed. The patch improves operator diagnostics without changing shutdown, read, or presence-check control flow.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Refresh the branch against current main and rerun the focused restart-sentinel tests plus required checks.

Risk before merge

  • [P1] The branch is behind current main, and restart-sentinel code has received recent changes; refresh and focused reruns are needed before merge, although GitHub currently reports the PR as cleanly mergeable and no concrete three-way regression is evident.

Maintainer options:

  1. Decide the mitigation before merge
    Land the warning-only error visibility and deterministic regression coverage after refreshing against current main, while retaining the existing best-effort return contracts and avoiding any broader restart or database policy change.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • No automated repair remains; refresh the clean patch against current main, complete required checks, and route it for ordinary maintainer merge review, likely starting with steipete or SebTardif.

Security
Cleared: The diff adds local diagnostic logging and test mocks without changing secrets, authorization, dependencies, workflows, package resolution, or executable supply-chain inputs.

Review details

Best possible solution:

Land the warning-only error visibility and deterministic regression coverage after refreshing against current main, while retaining the existing best-effort return contracts and avoiding any broader restart or database policy change.

Do we have a high-confidence way to reproduce the issue?

Yes. Controlled database-boundary failures on current main reproduce the silent catches, while the same paths on the PR head emit warnings and retain their fallback results.

Is this the best way to solve the issue?

Yes. Logging at the three ownership-boundary catches is the narrowest maintainable fix because callers intentionally depend on non-throwing best-effort behavior, and deterministic fault injection directly covers that invariant.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 345083a5ab95.

Label changes

Label justifications:

  • P3: This is a low-risk diagnostic improvement that preserves runtime behavior and primarily improves operator troubleshooting.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Before-and-after terminal output exercises all three database error paths and shows both the new warnings and unchanged safe fallback results; contributors should continue redacting private paths or other environment details from any additional logs.
  • proof: sufficient: Contributor real behavior proof is sufficient. Before-and-after terminal output exercises all three database error paths and shows both the new warnings and unchanged safe fallback results; contributors should continue redacting private paths or other environment details from any additional logs.
Evidence reviewed

PR surface:

Source +10, Tests +87. Total +97 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 17 7 +10
Tests 1 88 1 +87
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 105 8 +97

What I checked:

Likely related people:

  • steipete: Authored and merged recent restart-sentinel startup-path work that preserved its detection and wake behavior. (role: recent restart-sentinel contributor; confidence: high; commits: 3a05c7127af6; files: src/infra/restart-sentinel.ts, src/gateway/server-startup-post-attach.ts)
  • SebTardif: Introduced two merged warning-on-best-effort-failure changes cited by this PR, making them a useful reviewer for consistency of logging and fallback semantics. (role: adjacent diagnostics-pattern contributor; confidence: medium; commits: 907bc0371c3a, f75789f803f5; files: src/agents/subagent-registry-completion.ts, src/agents/subagent-registry.ts, src/infra/outbound/deliver.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (4 earlier review cycles)
  • reviewed 2026-07-13T13:22:59.627Z sha 9be7a92 :: needs changes before merge. :: [P2] Make the error-visibility tests fail before this fix
  • reviewed 2026-07-13T14:26:19.235Z sha c12170a :: needs changes before merge. :: [P2] Inject the database failure deterministically
  • reviewed 2026-07-13T16:26:45.795Z sha 436fc87 :: needs changes before merge. :: [P2] Remove the unused generic from the DB mock
  • reviewed 2026-07-13T23:45:04.332Z sha e088bb5 :: needs maintainer review before merge. :: none

@zenglingbiao
zenglingbiao force-pushed the fix/restart-sentinel-silent-catch branch 2 times, most recently from 9123044 to c12170a Compare July 13, 2026 13:53
@zenglingbiao

Copy link
Copy Markdown
Contributor Author

@clawsweeper review

@clawsweeper

clawsweeper Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@zenglingbiao
zenglingbiao force-pushed the fix/restart-sentinel-silent-catch branch from c12170a to 436fc87 Compare July 13, 2026 15:06
@zenglingbiao

Copy link
Copy Markdown
Contributor Author

@clawsweeper review

@clawsweeper

clawsweeper Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

Replace empty catch blocks in clearRestartSentinel, readRestartSentinel,
and hasRestartSentinel with sentinelLog.warn(...) calls so SQLite errors
are visible in diagnostics while preserving best-effort fallback semantics.

Tests use deterministic fault injection via vi.mock on openclaw-state-db.js
with hoisted throw flags, replacing the previous non-deterministic state-dir
removal approach. Each error path asserts both the warning message and the
correct fallback return value.
@zenglingbiao
zenglingbiao force-pushed the fix/restart-sentinel-silent-catch branch from 436fc87 to e088bb5 Compare July 13, 2026 23:41
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 13, 2026
@steipete steipete self-assigned this Jul 16, 2026
@steipete steipete changed the title fix(restart-sentinel): log DB errors instead of silently swallowing them fix(restart-sentinel): report persistence cleanup failures Jul 16, 2026
@steipete
steipete marked this pull request as draft July 16, 2026 05:48
@steipete
steipete marked this pull request as ready for review July 16, 2026 05:48
@steipete

steipete commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Land-ready maintainer pass complete at final head 814eafc1cbf73016415d2a581e92010a7f535900.

What changed:

  • Preserved the shipped best-effort contracts: clear remains non-throwing, read returns null, and presence checks return false on persistence failures.
  • Routed SQLite clear/read/has failures and legacy-file cleanup failures through the redacted restart-sentinel subsystem logger.
  • Kept the implementation types private, strengthened exact warning assertions, and absorbed fix(infra): log restart sentinel cleanup errors instead of silent catch #101687's missing legacy-cleanup coverage with contributor credit.

Proof:

  • Blacksmith Testbox lease tbx_01kxmvh0kztdddf7nze55j8ayd: pnpm test src/infra/restart-sentinel.test.ts passed 32 tests; focused queue-retarget and OpenAI provider-policy regressions passed 7 and 53 tests.
  • The exact two-worker fast-unit shard passed 1,052 files / 10,771 tests; pnpm check:changed then passed in 6m13s on the same lease.
  • Real corrupt-SQLite probe: clear completed, read returned null, presence returned false, and all three warnings were emitted. A real directory at the legacy sentinel path proved failed removal warns without throwing.
  • git diff --check passed. Implementation autoreview found no actionable issue at 0.98 confidence; fresh post-sync autoreview was also clean at 0.97.
  • Exact-head CI for the reviewed implementation, run 29479657111, passed 56 executed jobs with 11 intentional skips and no failures. The final head only integrates current main after a real changelog conflict and moves the same credit line to a stable anchor; the effective code diff is unchanged, the synthetic merge-tree is clean, and the repository's 24-hour same-PR green-cohort rule covers that non-code sync.

Earlier synthetic-merge failures were reproduced as deterministic regressions in the then-current main; both baselines were repaired before the green implementation run. No rebase or unrelated implementation churn was added to this contributor branch.

@steipete
steipete merged commit 47ffb49 into openclaw:main Jul 16, 2026
113 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 814eafc1cb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread CHANGELOG.md
- **macOS remote node readiness:** take the main-session key from the node hello snapshot instead of opening an operator connection during node admission, preventing remote tunnel recovery from leaving Computer Use and node exec stuck in lifecycle transition.
- **Claude CLI context budgets:** honor Anthropic model and per-agent `contextTokens` limits by passing the effective limit to Claude Code's native auto-compactor and persisting the same prepared budget in OpenClaw session state. Fixes #80933. (#93198) Thanks @mushuiyu886.
- **Transcript read failures:** propagate permission and I/O failures from streaming JSONL session reads instead of treating unreadable transcripts as empty. (#106412) Thanks @zenglingbiao.
- **Restart sentinel diagnostics:** report SQLite read/write and legacy-file cleanup failures while preserving best-effort restart recovery behavior. (#106385) Thanks @zenglingbiao and @wendy-chsy.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove the manual changelog entry

This commit changes runtime code and tests, so this is not the release-only changelog generation step; AGENTS.md:304 says CHANGELOG.md is release-only and should not be edited for normal fixes because release generation owns it. Keeping this manual entry can make the generated release notes duplicate or drift from the commit-derived changelog; put the release-note context in the commit/PR body instead and let the release process generate this line.

Useful? React with 👍 / 👎.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 17, 2026
…106385)

* fix(restart-sentinel): log DB errors instead of silently swallowing them

Replace empty catch blocks in clearRestartSentinel, readRestartSentinel,
and hasRestartSentinel with sentinelLog.warn(...) calls so SQLite errors
are visible in diagnostics while preserving best-effort fallback semantics.

Tests use deterministic fault injection via vi.mock on openclaw-state-db.js
with hoisted throw flags, replacing the previous non-deterministic state-dir
removal approach. Each error path asserts both the warning message and the
correct fallback return value.

* fix(restart-sentinel): report persistence failures

Co-authored-by: wendy-chsy <[email protected]>

* ci: retrigger pull request checks

* chore(ci): prune stale max-lines baseline

* chore(ci): retest against repaired protocol baseline

* chore: stabilize restart sentinel changelog entry

---------

Co-authored-by: Peter Steinberger <[email protected]>
Co-authored-by: wendy-chsy <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants