Skip to content

fix(discord): escalate repeated health-monitor restarts#72309

Merged
vincentkoc merged 1 commit into
mainfrom
clownfish/ghcrawl-165993-agentic-merge
Apr 26, 2026
Merged

fix(discord): escalate repeated health-monitor restarts#72309
vincentkoc merged 1 commit into
mainfrom
clownfish/ghcrawl-165993-agentic-merge

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

Summary

Repairs and lands the narrow Discord health-monitor restart escalation path from #40413 for #38596.

Credit

Based on contributor work in #40413 by @jellyAI-dev, with existing follow-up commits credited in that PR. If reconnect-grace behavior from #45712 is incorporated, credit @cass-clearly as well.

Plan

Fixes #38596. Related: #45712, #71546.

ProjectClownfish replacement details:

@aisle-research-bot

aisle-research-bot Bot commented Apr 26, 2026

Copy link
Copy Markdown

🔒 Aisle Security Analysis

We found 1 potential security issue(s) in this PR:

# Severity Title
1 🟡 Medium Restart throttling and manual-stop bypass via eviction of per-account lifecycle state on whole-channel reload
1. 🟡 Restart throttling and manual-stop bypass via eviction of per-account lifecycle state on whole-channel reload
Property Value
Severity Medium
CWE CWE-841
Location src/gateway/server-channels.ts:285-303

Description

During a whole-channel reload (startChannelInternal(channelId) with no accountId), the new evictStaleChannelAccountState() function deletes stored lifecycle state for any accountId not returned by plugin.config.listAccountIds(cfg).

This eviction clears two safety controls:

  • restartAttempts (used to enforce backoff / max restart attempts)
  • manuallyStopped (used to prevent auto-restart after a manual stop)

If listAccountIds(cfg) is influenced by config changes, transient plugin failures, or returns an incomplete/empty list temporarily, an attacker/admin (or buggy plugin) can cause state for real accounts to be evicted and then reintroduced, effectively resetting restart throttling and clearing manual-stop protections. This can enable:

  • restart loops and resource exhaustion (DoS)
  • unexpected restarts of accounts that were intentionally manually stopped

Vulnerable code:

store.runtimes.delete(id);
restartAttempts.delete(restartKey(channelId, id));
manuallyStopped.delete(restartKey(channelId, id));

Recommendation

Avoid clearing throttling/manual-stop state purely based on a transient listAccountIds() result.

Options:

  • Keep restartAttempts and manuallyStopped entries for a grace period (TTL) after an account disappears from listAccountIds.
  • Only evict store.runtimes (UI snapshot state) but do not clear restartAttempts / manuallyStopped unless there is an explicit administrative action confirming removal.
  • If eviction is required, persist safety state keyed by accountId in durable storage, or track an evictedAt timestamp and require time-based expiration.

Example (preserve safety state):

// only evict runtime snapshot; preserve safety controls
store.runtimes.delete(id);// DO NOT delete restartAttempts/manuallyStopped here

Or with TTL:

const rKey = restartKey(channelId, id);
evictedAt.set(rKey, Date.now());// clear manuallyStopped/restartAttempts only if evictedAt is older than TTL

Analyzed PR: #72309 at commit 24b797e

Last updated on: 2026-04-26T18:01:05Z

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: S maintainer Maintainer-authored PR labels Apr 26, 2026
@greptile-apps

greptile-apps Bot commented Apr 26, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes Discord gateway recovery looping (#38596) via two targeted changes: moving health-monitor restart accounting before the restart attempt so that failed restarts count toward cooldown and hourly caps, and evicting stale account lifecycle state during full channel reloads so old status cannot re-trigger the recovery loop. Both changes are covered by new dedicated tests.

Confidence Score: 4/5

Safe to merge; logic is correct and well-tested with one minor P2 cleanup gap in eviction coverage.

No P0 or P1 issues. A single P2 note about orphan restartAttempts/manuallyStopped keys not being covered by the eviction loop when a runtime entry no longer exists; in practice this is unlikely to cause misbehavior but represents incomplete state cleanup.

src/gateway/server-channels.ts — evictStaleChannelAccountState eviction loop

Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/gateway/server-channels.ts
Line: 291-303

Comment:
**Stale `restartAttempts`/`manuallyStopped` keys not cleaned up when no runtime entry exists**

`evictStaleChannelAccountState` iterates only `store.runtimes.keys()`, so any account that has an entry in `restartAttempts` or `manuallyStopped` but whose runtime was already cleaned up by a previous stop would retain stale state even after a channel reload. In practice this can prevent the "reset to clean state" goal of the eviction: on the next full reload the hourly health-monitor caps from the old account will silently survive. Consider also iterating the keyset of `restartAttempts`/`manuallyStopped` scoped to this `channelId` (or clearing them unconditionally for removed accounts).

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(discord): escalate repeated health-m..." | Re-trigger Greptile

Comment on lines +291 to +303
for (const id of store.runtimes.keys()) {
if (
activeAccountIds.has(id) ||
store.aborts.has(id) ||
store.starting.has(id) ||
store.tasks.has(id)
) {
continue;
}
store.runtimes.delete(id);
restartAttempts.delete(restartKey(channelId, id));
manuallyStopped.delete(restartKey(channelId, id));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Stale restartAttempts/manuallyStopped keys not cleaned up when no runtime entry exists

evictStaleChannelAccountState iterates only store.runtimes.keys(), so any account that has an entry in restartAttempts or manuallyStopped but whose runtime was already cleaned up by a previous stop would retain stale state even after a channel reload. In practice this can prevent the "reset to clean state" goal of the eviction: on the next full reload the hourly health-monitor caps from the old account will silently survive. Consider also iterating the keyset of restartAttempts/manuallyStopped scoped to this channelId (or clearing them unconditionally for removed accounts).

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/server-channels.ts
Line: 291-303

Comment:
**Stale `restartAttempts`/`manuallyStopped` keys not cleaned up when no runtime entry exists**

`evictStaleChannelAccountState` iterates only `store.runtimes.keys()`, so any account that has an entry in `restartAttempts` or `manuallyStopped` but whose runtime was already cleaned up by a previous stop would retain stale state even after a channel reload. In practice this can prevent the "reset to clean state" goal of the eviction: on the next full reload the hourly health-monitor caps from the old account will silently survive. Consider also iterating the keyset of `restartAttempts`/`manuallyStopped` scoped to this `channelId` (or clearing them unconditionally for removed accounts).

How can I resolve this? If you propose a fix, please make it concise.

@vincentkoc vincentkoc self-assigned this Apr 26, 2026
@vincentkoc
vincentkoc merged commit b4cdd55 into main Apr 26, 2026
69 of 71 checks passed
@vincentkoc
vincentkoc deleted the clownfish/ghcrawl-165993-agentic-merge branch April 26, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime maintainer Maintainer-authored PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discord: health monitor restart loop — post-connection zombie sessions evade circuit breakers

1 participant