Skip to content

fix(daemon): hand off systemd self-restarts#66735

Closed
alexlomt wants to merge 2 commits into
openclaw:mainfrom
alexlomt:fix/systemd-self-restart-handoff
Closed

fix(daemon): hand off systemd self-restarts#66735
alexlomt wants to merge 2 commits into
openclaw:mainfrom
alexlomt:fix/systemd-self-restart-handoff

Conversation

@alexlomt

@alexlomt alexlomt commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Summary

On Linux/systemd, restarting the gateway from inside the running gateway service can kill the calling CLI process with SIGTERM even though the service restart succeeds.

This happens because the restart command runs inside the same openclaw-gateway.service cgroup, and the unit uses KillMode=control-group. A plain systemctl --user restart openclaw-gateway.service therefore kills the gateway and the caller together.

This change makes the Linux systemd adapter treat self-restarts differently from external restarts:

  • if the caller is already inside the target systemd service unit, schedule a detached systemd-run --user handoff and return scheduled
  • otherwise keep the existing synchronous restart path and return completed

Why this is safe

  • Linux only
  • systemd only
  • restart path only
  • external restart behaviour is preserved
  • no unit-file semantics changed
  • no change to launchd or macOS behaviour

Implementation

  • detect whether the current process is running inside the target systemd service cgroup
  • if yes, schedule a transient user unit outside that cgroup with systemd-run --user
  • wait for the current process PID to exit, then run systemctl --user restart <unit> from the detached handoff
  • surface handoff failures clearly
  • treat a matching expected SIGTERM during handoff as restart semantics instead of a normal stop

Validation performed locally

  • corepack pnpm exec oxfmt --check src/daemon/systemd.ts src/daemon/systemd.test.ts src/cli/gateway-cli/run-loop.ts src/cli/gateway-cli/run-loop.test.ts src/agents/subagent-registry.steer-restart.test.ts src/gateway/server.sessions.gateway-server-sessions-a.test.ts
  • corepack pnpm exec vitest run src/daemon/systemd.test.ts src/cli/gateway-cli/run-loop.test.ts --reporter=verbose
  • corepack pnpm exec vitest run --config test/vitest/vitest.agents.config.ts src/agents/subagent-registry.steer-restart.test.ts --reporter=verbose
  • corepack pnpm exec vitest run --config test/vitest/vitest.gateway-server.config.ts src/gateway/server.sessions.gateway-server-sessions-a.test.ts --reporter=verbose

Notes

This follows the same scheduled-restart contract already used elsewhere in the daemon lifecycle, rather than changing systemd unit kill behaviour.

Review Focus

  • src/daemon/systemd.ts
  • src/cli/gateway-cli/run-loop.ts
  • src/daemon/systemd.test.ts
  • src/cli/gateway-cli/run-loop.test.ts

Scope Boundary

  • Linux only
  • systemd only
  • restart path only
  • external restart behavior remains synchronous and unchanged

Security Impact

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command/tool execution surface changed? Yes
  • Data access scope changed? No
  • If any Yes, explain risk + mitigation: self-restarts inside the managed unit now schedule a detached systemd-run --user handoff only for the in-unit restart path. External restarts are unchanged, marker parsing is validated strictly, and tests cover malformed markers, custom unit names, and bounded restart draining.

Human Verification

  • Verified scenarios: self-restart handoff scheduling, marker-based SIGTERM restart handling, malformed marker rejection, custom unit normalization, and bounded supervised restart draining
  • Edge cases checked: detached restart failure cleanup, marker timestamp validation, current-main test harness after rebase
  • What I did not verify: live smoke on a real systemd user service manager

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: M labels Apr 14, 2026
@greptile-apps

greptile-apps Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a SIGTERM-kills-caller issue on Linux/systemd by detecting when restartSystemdService is invoked from within the target service cgroup and scheduling a detached systemd-run --user handoff instead of a synchronous systemctl restart. External restarts keep the existing synchronous path unchanged. Both paths have new regression test coverage and the cgroup-matching logic handles cgroups v1/v2 correctly.

Confidence Score: 5/5

  • Safe to merge; all findings are minor style/robustness suggestions with no correctness impact.
  • The core logic (cgroup detection, argument passing, handoff scheduling, error surfacing) is correct and well-tested. Both remaining comments are P2: an unbounded PID-wait loop that is operationally low-risk given normal SIGTERM handling, and an unnecessarily broad parameter type.
  • No files require special attention.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/daemon/systemd.ts
Line: 125-135

Comment:
**Unbounded PID-wait loop in handoff script**

The `while kill -0 "$wait_pid"` loop has no timeout. If the gateway process ignores `SIGTERM` and never exits, the transient handoff unit will spin indefinitely, and the target service will never restart. Consider adding a deadline (e.g. 30 s) after which the loop falls through or bails with a non-zero exit so the failure is surfaced.

```suggestion
function buildSystemdRestartHandoffScript(): string {
  return `wait_pid="$1"
 target_unit="$2"
 deadline=$(( $(date +%s) + 30 ))
 if [ -n "$wait_pid" ] && [ "$wait_pid" -gt 1 ] 2>/dev/null; then
   while kill -0 "$wait_pid" >/dev/null 2>&1; do
     if [ "$(date +%s)" -ge "$deadline" ]; then
       echo "openclaw-systemd-restart-handoff: timed out waiting for pid $wait_pid to exit" >&2
       exit 1
     fi
     sleep 0.1
   done
 fi
 exec systemctl --user restart "$target_unit"
 `;
}
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/daemon/systemd.ts
Line: 80-90

Comment:
**Unnecessary `undefined` in parameter type**

`cgroupPath` is typed as `string | undefined`, but every call site passes a `string` (the elements of `string[]` from `readCurrentProcessCgroupPaths`). The `| undefined` branch is dead code. Narrowing the type keeps the signature honest.

```suggestion
function matchesSystemdCgroupUnit(cgroupPath: string, unitName: string): boolean {
  const normalizedPath = normalizeOptionalString(cgroupPath);
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(daemon): hand off systemd self-resta..." | Re-trigger Greptile

Comment thread src/daemon/systemd.ts
Comment thread src/daemon/systemd.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d78d9eb6ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/daemon/systemd.ts Outdated
@alexlomt

Copy link
Copy Markdown
Contributor Author

Pushed f852688b9f with the follow-up fixes.

What changed:

  • self-restart detection now uses current-process evidence instead of merged target env
  • cgroup detection stays primary, with fallback limited to real systemd runtime hints plus matching gateway markers
  • the detached handoff script now has a 30s deadline and warn-and-continue behavior
  • narrowed matchesSystemdCgroupUnit(cgroupPath) to string
  • added regression coverage for merged gateway env staying synchronous and for the runtime-hint fallback

Validation:

  • node scripts/test-projects.mjs src/daemon/systemd.test.ts src/daemon/service.test.ts
  • live external smoke against a disposable user unit: returned {"outcome":"completed"} and restarted immediately
  • live self-restart smoke from inside a disposable user unit: returned {"outcome":"scheduled"} and restarted via detached handoff

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f852688b9f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/daemon/systemd.ts
@alexlomt

Copy link
Copy Markdown
Contributor Author

Unrelated security-fast workflow fix is split out into #66884 so this daemon PR can stay focused. That PR hardens openclaw-cross-os-release-checks-reusable.yml against the current zizmor template-injection findings and adds explicit inputs.mode validation.

@openclaw-barnacle openclaw-barnacle Bot added cli CLI command changes size: L and removed size: M labels Apr 19, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 232b40df87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli/gateway-cli/run-loop.ts Outdated
@alexlomt

Copy link
Copy Markdown
Contributor Author

Follow-up added in 232b40df87:

  • write a short-lived systemd restart expectation marker before the detached handoff restarts the unit
  • treat the resulting gateway SIGTERM as restart semantics instead of a normal stop when that marker is present
  • add targeted coverage in src/cli/gateway-cli/run-loop.test.ts alongside the daemon handoff tests

Local verification for the follow-up:

  • corepack pnpm exec oxfmt --check src/daemon/systemd.ts src/daemon/systemd.test.ts src/cli/gateway-cli/run-loop.ts src/cli/gateway-cli/run-loop.test.ts
  • corepack pnpm exec vitest run --config test/vitest/vitest.daemon.config.ts src/daemon/systemd.test.ts
  • corepack pnpm exec vitest run --config test/vitest/vitest.cli.config.ts src/cli/gateway-cli/run-loop.test.ts

@openclaw-barnacle openclaw-barnacle Bot added the agents Agent runtime and tooling label Apr 20, 2026
@alexlomt
alexlomt force-pushed the fix/systemd-self-restart-handoff branch from dd5ac56 to cf56098 Compare April 20, 2026 00:12

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cf560980da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli/gateway-cli/run-loop.ts Outdated
Comment thread src/daemon/systemd.ts Outdated
@alexlomt
alexlomt force-pushed the fix/systemd-self-restart-handoff branch from cf56098 to f6be80b Compare April 20, 2026 00:56

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f6be80b9b6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli/gateway-cli/run-loop.ts Outdated
@alexlomt
alexlomt force-pushed the fix/systemd-self-restart-handoff branch from f6be80b to d43dff0 Compare April 24, 2026 21:04
@openclaw-barnacle openclaw-barnacle Bot removed the agents Agent runtime and tooling label Apr 24, 2026
@alexlomt

Copy link
Copy Markdown
Contributor Author

High-value ops fix for Linux/systemd installs: restarting from inside the managed unit can SIGTERM the caller because of KillMode=control-group. This PR only changes the self-restart path; external restarts remain unchanged. Review focus: src/daemon/systemd.ts, src/cli/gateway-cli/run-loop.ts, and the matching daemon/run-loop tests. Current branch is green, including custom-unit normalization, malformed marker rejection, and supervisor-bounded restart draining.

@alexlomt
alexlomt force-pushed the fix/systemd-self-restart-handoff branch from d43dff0 to 7a191b5 Compare April 26, 2026 12:15
@vincentkoc

Copy link
Copy Markdown
Member

ProjectClownfish pushed a narrow repair to this branch so the original contributor path can stay canonical.

Source PR: #66735
Validation: pnpm check:changed
Contributor credit is preserved in the branch history and PR context.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels May 19, 2026
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 19, 2026
@clawsweeper

clawsweeper Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 6, 2026
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 6, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jun 7, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 14, 2026
@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@clawsweeper clawsweeper Bot closed this Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clawsweeper Tracked by ClawSweeper automation cli CLI command changes gateway Gateway runtime merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: L status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants