Summary
A gateway restart that times out while active agent work is still running can leave affected sessions in a state that the restart recovery path does not pick up.
In the observed case, config reload deferred the restart for about 300 seconds while active operations/replies/embedded runs were still present, then forced restart. The affected sessions remained persisted as status: "running" with abortedLastRun: false, so main-session recovery skipped them and no automatic continuation was scheduled.
This is adjacent to restart/session-recovery issues such as #57425, #64585, #73160, #74424, and #51620, but the specific bug here is the mismatch between:
- restart timeout/drain behavior, which can cross a process boundary with active work still present
- recovery selection, which only resumes entries already marked
abortedLastRun: true
Environment
- OpenClaw:
2026.5.16-beta.1 (dc8790d)
- Runtime: Linux/systemd gateway
- Channel observed: Telegram
- Multi-agent gateway with long-running embedded/ACP work
- Evidence below uses line numbers from the installed npm
dist/ bundle for this exact version
Observed behavior
During a config-triggered restart, the gateway correctly noticed active work and deferred:
restart still deferred after 30253ms with 2 operation(s), 2 reply(ies), 1 embedded run(s) active
restart still deferred after 277478ms with 2 operation(s), 1 reply(ies), 1 embedded run(s) active
restart timeout after 300003ms with 2 operation(s), 1 reply(ies), 1 embedded run(s) still active; forcing restart
Immediately after the forced restart signal:
SIGUSR1 restart ignored (not authorized; commands.restart=false or use gateway tool).
A later explicit gateway-tool restart request then did not repair the situation because restart state still appeared in flight:
gateway tool: restart requested ...
request coalesced (already in-flight) ...
Afterward, a sanitized scan of the session store showed affected main sessions persisted as:
{
"status": "running",
"abortedLastRun": false,
"hasActiveRun": null
}
That state is important because the recovery code below ignores it.
Relevant implementation path
Config reload forces restart after timeout while active work remains
In server-reload-handlers-CxCiE34c.js, reload deferral waits until active work finishes, but on timeout it proceeds with a forced restart:
const active = getActiveCounts();
if (active.totalActive > 0) {
...
deferGatewayRestartUntilIdle({
getPendingCount: () => getActiveCounts().totalActive,
maxWaitMs: resolveGatewayRestartDeferralTimeoutMs(...),
hooks: {
onTimeout: (_pending, elapsedMs) => {
const remaining = formatActiveDetails(getActiveCounts());
restartPending = false;
params.logReload.warn(
`restart timeout after ${elapsedMs}ms with ${remaining.join(", ")} still active; forcing restart`
);
}
}
});
}
There is no durable marking here that says "these active sessions were interrupted by restart and should be resumed."
Restart drain aborts active runs, but does not mark main sessions resumable
In run-BOLhmm0S.js, the SIGUSR1 restart path drains active tasks and embedded runs. If the drain times out, it proceeds and aborts embedded runs:
const [tasksDrain, runsDrain] = await Promise.all([
activeTasks > 0 ? waitForActiveTasks(restartDrainTimeoutMs) : Promise.resolve({ drained: true }),
activeRuns > 0 ? waitForActiveEmbeddedRuns(restartDrainTimeoutMs) : Promise.resolve({ drained: true })
]);
if (tasksDrain.drained && runsDrain.drained) gatewayLog.info("all active work drained");
else {
gatewayLog.warn("drain timeout reached; proceeding with restart");
abortEmbeddedPiRun(void 0, { mode: "all" });
}
...
await server?.close({ reason: "gateway restarting", restartExpectedMs: 1500 });
...
if (isRestart) await handleRestartAfterServerClose(restartReason);
Again, this path aborts active work and crosses the process boundary, but does not appear to update affected main-session store entries to abortedLastRun: true or otherwise make them eligible for continuation.
Main-session recovery only resumes running + abortedLastRun: true
In main-session-restart-recovery-cbFmhQXh.js:
for (const [sessionKey, entry] of Object.entries(store).toSorted(([a], [b]) => a.localeCompare(b))) {
if (!entry || entry.status !== "running" || entry.abortedLastRun !== true) continue;
...
if (await resumeMainSession(...)) {
params.resumedSessionKeys.add(sessionKey);
result.recovered++;
}
}
So a session left as status: "running" and abortedLastRun: false is permanently skipped by immediate restart recovery, even if it was active during the forced restart.
Startup stale-lock recovery is too delayed/indirect for this case
Startup can mark main sessions from cleaned stale transcript locks, but only through stale lock cleanup:
const result = await cleanStaleLockFiles({
sessionsDir,
staleMs: SESSION_LOCK_STALE_MS,
removeStale: true,
...
});
if (result.cleaned.length > 0) {
await markRestartAbortedMainSessionsFromLocks({ sessionsDir, cleanedLocks: result.cleaned });
}
...
scheduleRestartAbortedMainSessionRecovery();
This means immediate recovery depends on either abortedLastRun: true already being set or stale-lock cleanup later finding something. That is not enough for normal restart timeout/drain-timeout interruption.
Secondary issue: restart authorization/in-flight mismatch
The observed logs also show an authorization/in-flight mismatch:
- forced restart emits SIGUSR1
- handler logs
SIGUSR1 restart ignored (not authorized...)
- later gateway-tool restart is coalesced as
already in-flight
The relevant code in restart-BH0g42VV.js sets emittedRestartToken before emitting SIGUSR1:
if (hasUnconsumedRestartSignal()) return false;
...
emittedRestartToken = ++restartCycleToken;
emittedRestartReason = reasonOverride ?? pendingRestartReason;
authorizeGatewaySigusr1Restart();
try {
if (process.listenerCount("SIGUSR1") > 0) process.emit("SIGUSR1");
...
} catch {
rollBackGatewayRestartEmission();
return false;
}
Then scheduleGatewaySigusr1Restart() coalesces when hasUnconsumedRestartSignal() remains true:
if (hasUnconsumedRestartSignal()) {
restartLog.warn(`restart request coalesced (already in-flight) reason=${reason ?? "unspecified"} ...`);
return { ok: true, signal: "SIGUSR1", coalesced: true, ... };
}
If the SIGUSR1 listener refuses/ignores the restart, the in-flight token should be cleared or rolled back; otherwise later restart requests cannot repair the failed transition.
Expected behavior
When restart deferral or restart drain times out while work is still active:
- The gateway should durably mark affected active main sessions/tasks as interrupted/resumable before crossing the process boundary.
- Main-session restart recovery should be able to continue or at least notify on those sessions after startup.
- If SIGUSR1 restart is ignored/refused by the listener, the emitted restart token should be cleared so a later gateway-tool restart can proceed.
- Recovery should not depend only on stale lock cleanup for sessions that the restart path itself knows are being interrupted.
Suggested fix direction
A robust fix probably needs both sides:
- Before forced restart or drain-timeout restart proceeds, write a restart interruption marker for active main sessions, e.g. set
abortedLastRun: true plus a restart reason/timestamp/continuation hint.
- Make restart recovery consume that marker and clear it only after successful continuation or explicit failure.
- Treat SIGUSR1 listener refusal as a failed emission and roll back
emittedRestartToken / in-flight restart state.
- Add tests for:
- restart deferral timeout with active main session
- restart drain timeout with active embedded run
- recovery skips
running + abortedLastRun: false today but should recover marked sessions
- ignored SIGUSR1 does not leave later restart requests coalesced forever
Impact
This makes gateway updates/config restarts dangerous for long-running agents. The user-visible symptom is that agents stop replying after a gateway restart and do not resume their interrupted work, even though their session store still says they are running.
Operators then have to manually inspect/reset sessions or ask agents to continue, and some in-flight work may be orphaned.
Summary
A gateway restart that times out while active agent work is still running can leave affected sessions in a state that the restart recovery path does not pick up.
In the observed case, config reload deferred the restart for about 300 seconds while active operations/replies/embedded runs were still present, then forced restart. The affected sessions remained persisted as
status: "running"withabortedLastRun: false, so main-session recovery skipped them and no automatic continuation was scheduled.This is adjacent to restart/session-recovery issues such as #57425, #64585, #73160, #74424, and #51620, but the specific bug here is the mismatch between:
abortedLastRun: trueEnvironment
2026.5.16-beta.1 (dc8790d)dist/bundle for this exact versionObserved behavior
During a config-triggered restart, the gateway correctly noticed active work and deferred:
Immediately after the forced restart signal:
A later explicit gateway-tool restart request then did not repair the situation because restart state still appeared in flight:
Afterward, a sanitized scan of the session store showed affected main sessions persisted as:
{ "status": "running", "abortedLastRun": false, "hasActiveRun": null }That state is important because the recovery code below ignores it.
Relevant implementation path
Config reload forces restart after timeout while active work remains
In
server-reload-handlers-CxCiE34c.js, reload deferral waits until active work finishes, but on timeout it proceeds with a forced restart:There is no durable marking here that says "these active sessions were interrupted by restart and should be resumed."
Restart drain aborts active runs, but does not mark main sessions resumable
In
run-BOLhmm0S.js, the SIGUSR1 restart path drains active tasks and embedded runs. If the drain times out, it proceeds and aborts embedded runs:Again, this path aborts active work and crosses the process boundary, but does not appear to update affected main-session store entries to
abortedLastRun: trueor otherwise make them eligible for continuation.Main-session recovery only resumes
running + abortedLastRun: trueIn
main-session-restart-recovery-cbFmhQXh.js:So a session left as
status: "running"andabortedLastRun: falseis permanently skipped by immediate restart recovery, even if it was active during the forced restart.Startup stale-lock recovery is too delayed/indirect for this case
Startup can mark main sessions from cleaned stale transcript locks, but only through stale lock cleanup:
This means immediate recovery depends on either
abortedLastRun: truealready being set or stale-lock cleanup later finding something. That is not enough for normal restart timeout/drain-timeout interruption.Secondary issue: restart authorization/in-flight mismatch
The observed logs also show an authorization/in-flight mismatch:
SIGUSR1 restart ignored (not authorized...)already in-flightThe relevant code in
restart-BH0g42VV.jssetsemittedRestartTokenbefore emitting SIGUSR1:Then
scheduleGatewaySigusr1Restart()coalesces whenhasUnconsumedRestartSignal()remains true:If the SIGUSR1 listener refuses/ignores the restart, the in-flight token should be cleared or rolled back; otherwise later restart requests cannot repair the failed transition.
Expected behavior
When restart deferral or restart drain times out while work is still active:
Suggested fix direction
A robust fix probably needs both sides:
abortedLastRun: trueplus a restart reason/timestamp/continuation hint.emittedRestartToken/ in-flight restart state.running + abortedLastRun: falsetoday but should recover marked sessionsImpact
This makes gateway updates/config restarts dangerous for long-running agents. The user-visible symptom is that agents stop replying after a gateway restart and do not resume their interrupted work, even though their session store still says they are
running.Operators then have to manually inspect/reset sessions or ask agents to continue, and some in-flight work may be orphaned.