fix(gateway): guard getRuntimeSnapshot() and preserve probe intent in health refresh#52770
fix(gateway): guard getRuntimeSnapshot() and preserve probe intent in health refresh#52770ajayr wants to merge 2 commits into
Conversation
Greptile SummaryThis PR addresses two issues from PR #42543: wrapping One remaining issue:
Confidence Score: 3/5Not safe to merge — one unresolved P1 in health-state.ts where probe:false callers during in-flight refreshes trigger unnecessary follow-ups that can self-sustain under load. Prior P1s (infinite loop, duplicated guard) are confirmed fixed. The new P1 — spurious follow-up chain caused by probe:false setting pendingProbe — is a concrete regression with a known call site (server-methods/health.ts:18) and a clear fix. Until that line is changed, the PR should not merge. src/gateway/server/health-state.ts — specifically lines 113–115 (pendingProbe assignment) and line 146 (hadPendingProbe condition in finally block).
|
| Filename | Overview |
|---|---|
| src/gateway/server/health-state.ts | Core logic for the two fixes: added pendingProbe tracking and safeGetRuntimeSnapshot lazy getter. The infinite-loop P1 from a prior iteration is resolved, but probe:false callers during an in-flight refresh set pendingProbe=false, causing hadPendingProbe to fire in the finally block and schedule an unnecessary follow-up — see inline comment. |
| src/gateway/server.impl.ts | Extracts safeGetRuntimeSnapshot helper (resolving the duplicated try/catch noted in a prior review), registers it via setRuntimeSnapshotGetter, and threads it into startGatewayMaintenanceTimers. Clean and correct. |
| src/gateway/server-maintenance.ts | Adds optional getRuntimeSnapshot param and calls setRuntimeSnapshotGetter lazily after setBroadcastHealthUpdate. Straightforward wiring; no issues. |
| src/commands/health.ts | Threads optional runtimeSnapshot parameter through getHealthSnapshot and applies it per-account via a spread before the explicit fields. The spread ordering ensures accountId, enabled, and configured always win over runtime values. Safe. |
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/gateway/server/health-state.ts
Line: 113-152
Comment:
**`probe: false` callers unintentionally trigger a follow-up refresh**
When a caller passes `probe: false` while a refresh is in-flight (e.g. `server-methods/health.ts:18` fires `void refreshHealthSnapshot({ probe: false })` as a fire-and-forget background warm-up whenever the cache is fresh), the code at lines 113–115 unconditionally sets `pendingProbe = false`. The `else if` branch (line 156) doesn't catch it because `opts?.probe` is falsy, so the caller falls through to `return healthRefresh` and gets back the in-flight result.
The problem surfaces in the `finally` block at line 146:
```ts
const hadPendingProbe = pendingProbe !== undefined; // true when pendingProbe === false
```
`hadPendingProbe` becomes `true` even though the "new" caller only wanted `probe: false`. Lines 152–153 then schedule a follow-up with `probe: false`, which:
1. Overwrites the probe:true cache entry with a lower-quality non-probed snapshot.
2. Creates a self-sustaining chain under load: the probe:false follow-up completes → cache is fresh → the next health request fires another background `probe: false` → another follow-up is scheduled → repeat.
The fix is to only count a `probe: true` arrival as something that requires a follow-up:
```suggestion
if (opts?.probe === true) {
pendingProbe = opts.probe;
}
```
And correspondingly tighten the `finally` check:
```ts
const hadPendingProbe = pendingProbe === true;
```
(The `followUpProbe` line can stay as `pendingProbe ?? false`, which is still correct once only `true` values land in `pendingProbe`.)
This preserves the original intent — a `probe: true` caller that arrives during a `probe: false` in-flight refresh gets a follow-up that actually probes — without treating a background `probe: false` call as a signal that a follow-up is needed.
How can I resolve this? If you propose a fix, please make it concise.Reviews (18): Last reviewed commit: "fix(health): lazy runtimeSnapshot captur..." | Re-trigger Greptile
| refreshGatewayHealthSnapshot: (opts) => { | ||
| let runtimeSnapshot: Awaited<ReturnType<typeof getRuntimeSnapshot>> | undefined; | ||
| try { | ||
| runtimeSnapshot = getRuntimeSnapshot(); | ||
| } catch (err) { | ||
| log.warn("getRuntimeSnapshot() failed during health refresh", { error: err }); | ||
| } | ||
| return refreshGatewayHealthSnapshot({ ...opts, runtimeSnapshot }); | ||
| }, |
There was a problem hiding this comment.
Duplicated
getRuntimeSnapshot guard logic
The identical try/catch wrapper for getRuntimeSnapshot() is copy-pasted verbatim at lines 811–819 and again at lines 1082–1090. If the error-handling or logging details need to change in the future they'd need to be updated in both places.
Consider extracting it into a small local helper near the call sites, e.g.:
const safeGetRuntimeSnapshot = () => {
try {
return getRuntimeSnapshot();
} catch (err) {
log.warn("getRuntimeSnapshot() failed during health refresh", { error: err });
return undefined;
}
};Then each call site becomes a one-liner: return refreshGatewayHealthSnapshot({ ...opts, runtimeSnapshot: safeGetRuntimeSnapshot() });
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/server.impl.ts
Line: 811-819
Comment:
**Duplicated `getRuntimeSnapshot` guard logic**
The identical try/catch wrapper for `getRuntimeSnapshot()` is copy-pasted verbatim at lines 811–819 and again at lines 1082–1090. If the error-handling or logging details need to change in the future they'd need to be updated in both places.
Consider extracting it into a small local helper near the call sites, e.g.:
```ts
const safeGetRuntimeSnapshot = () => {
try {
return getRuntimeSnapshot();
} catch (err) {
log.warn("getRuntimeSnapshot() failed during health refresh", { error: err });
return undefined;
}
};
```
Then each call site becomes a one-liner: `return refreshGatewayHealthSnapshot({ ...opts, runtimeSnapshot: safeGetRuntimeSnapshot() });`
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Fixed in f94c29d — extracted safeGetRuntimeSnapshot() helper so the try/catch guard lives in one place.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5abb60f82e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (pendingRuntimeSnapshot !== undefined) { | ||
| void refreshGatewayHealthSnapshot({ probe: followUpProbe }); |
There was a problem hiding this comment.
Avoid scheduling refreshes on ignored runtime snapshots
This follow-up refresh is triggered whenever pendingRuntimeSnapshot is set, but getHealthSnapshot currently ignores runtimeSnapshot entirely (its params in src/commands/health.ts only read timeoutMs and probe). In practice, concurrent health calls can now cause back-to-back full refreshes with identical payloads, which increases probe/load cost under polling without improving freshness. Either plumb runtimeSnapshot into the health computation or stop using it as the trigger for a second refresh.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Already addressed in f94c29d — getHealthSnapshot() now accepts a runtimeSnapshot param and merges runtime channel state (connected, lastConnectedAt, lastEventAt, etc.) into the channel snapshot. The follow-up refresh correctly carries the snapshot forward.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f94c29da14
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // If a newer runtimeSnapshot arrived while the refresh was in-flight, kick | ||
| // off a follow-up refresh so the latest runtime state is reflected. | ||
| if (pendingRuntimeSnapshot !== undefined) { | ||
| void refreshGatewayHealthSnapshot({ probe: followUpProbe }); |
There was a problem hiding this comment.
Catch errors from queued health follow-up refresh
refreshGatewayHealthSnapshot schedules a second refresh with void refreshGatewayHealthSnapshot({ probe: followUpProbe }) and drops the returned promise. If that follow-up call fails (for example, any exception bubbling out of getHealthSnapshot), it becomes an unhandled rejection. This is production-impacting because the global unhandled-rejection handler exits the process, so concurrent health requests that trigger this path can crash the gateway instead of just logging a refresh failure.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5291275 — added .catch(() => {}) to the void follow-up call so any exceptions are silently swallowed.
5291275 to
13c2390
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57a6ed0560
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // If a newer runtimeSnapshot arrived while the refresh was in-flight, kick | ||
| // off a follow-up refresh so the latest runtime state is reflected. | ||
| if (pendingRuntimeSnapshot !== undefined) { | ||
| void refreshGatewayHealthSnapshot({ probe: followUpProbe }).catch(() => {}); |
There was a problem hiding this comment.
Schedule probe follow-ups when runtimeSnapshot capture fails
If safeGetRuntimeSnapshot() catches the getRuntimeSnapshot() error and returns undefined (src/gateway/server.impl.ts:757-764), a concurrent probe:true call only updates pendingProbe. Because this finally block only queues a second refresh when pendingRuntimeSnapshot exists, that verbose health request still resolves to the in-flight probe:false snapshot instead of re-probing. In other words, the config-error path this patch is targeting can still silently drop the caller's probe intent.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 596102e — the finally block now also triggers a follow-up refresh when hadPendingProbe is set, so a probe request that arrives after safeGetRuntimeSnapshot() has already returned undefined is not silently dropped.
57a6ed0 to
59fe362
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59fe362a5e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (pendingRuntimeSnapshot !== undefined) { | ||
| void refreshGatewayHealthSnapshot({ probe: followUpProbe }).catch(() => {}); |
There was a problem hiding this comment.
Queue follow-up when probe intent changes without snapshot
The finally path only triggers a second refresh when pendingRuntimeSnapshot is present, so if a concurrent caller requests probe: true while safeGetRuntimeSnapshot() returns undefined (its caught-error path), the probe intent is still dropped. In that sequence this function returns the in-flight probe:false result, clears pendingProbe, and never schedules a re-probe, so the caller gets stale non-probed health despite explicitly asking for probing.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 596102e — same root cause as the previous item. The finally block now captures hadPendingProbe before clearing pendingProbe, and triggers a follow-up refresh when either pendingRuntimeSnapshot or hadPendingProbe is set. This ensures a probe:true call that arrives after safeGetRuntimeSnapshot() has already returned undefined still triggers a refresh with probe:true.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26466e90a5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
a78a73f to
f3cf7fc
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f3cf7fc820
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
All unaddressed review comments from bot reviewers are now addressed in commit f3cf7fc8:
CI is running on f3cf7fc8 (rebased onto current main c2c136a). |
f3cf7fc to
32a5151
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32a5151de3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| refreshHealthSnapshot: (opts) => | ||
| refreshGatewayHealthSnapshot({ ...opts, runtimeSnapshot: safeGetRuntimeSnapshot() }), |
There was a problem hiding this comment.
Capture runtime snapshots only after dedupe decides to refresh
This wrapper calls safeGetRuntimeSnapshot() before refreshGatewayHealthSnapshot decides whether a refresh is already in flight, so every overlapping health request still pays the full snapshot cost even when dedupe would otherwise reuse the existing promise. In busy polling setups, this repeatedly walks all channel accounts (getRuntimeSnapshot iterates plugins/accounts and resolves each account), which increases CPU/load and can spam warning logs in the same config-error scenario this patch is trying to soften.
Useful? React with 👍 / 👎.
32a5151 to
4724280
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4724280cab
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (opts?.probe) { | ||
| pendingProbe = opts.probe; |
There was a problem hiding this comment.
Preserve queued probe=false overrides during refresh
refreshGatewayHealthSnapshot only records pending probe intent when opts.probe is true, so a later probe:false call cannot clear an earlier queued probe. In the sequence “in-flight probe:false refresh → concurrent probe:true request → later concurrent probe:false request”, the finally block still schedules a follow-up probe and the later caller can end up waiting for that extra probe cycle, which adds avoidable probe latency/load and violates the function’s own “latest caller preference” behavior.
Useful? React with 👍 / 👎.
… runtimeSnapshot - Wrap getRuntimeSnapshot() in safeGetRuntimeSnapshot() to prevent health RPC crashes from config errors (e.g., unresolved SecretRef) - Add pendingProbe tracking so follow-up refreshes use the most recent caller's probe intent instead of the original caller's - Clear pendingProbe when refresh starts to prevent infinite follow-up loops - Only track probe:true (false is default, storing it triggers needless follow-ups) - Wait for follow-up when probe intent arrives during in-flight refresh - Plumb runtimeSnapshot into getHealthSnapshot() and merge runtime channel state into health snapshots - Reset pendingProbe in __resetHealthStateForTest()
4724280 to
2d75865
Compare
Address review comments from chatgpt-codex-connector: - #3003609871: Move getRuntimeSnapshot() call lazily inside the health refresh cycle (after dedupe decides to proceed), not eagerly at every timer tick. Add setRuntimeSnapshotGetter() init pattern — the getter is registered once at startup and called inside the refresh function. - #3003634321: Track explicit probe:false alongside probe:true. Previously pendingProbe was only set when probe:true (false was the default, so storing it would trigger needless follow-ups). Now we track whenever opts.probe is explicitly provided (true or false), and the follow-up only fires when the new intent differs from what the in-flight refresh is already using.
Fixes two review issues flagged on the original PR #42543:
P1 — can throw if any channel's
config.resolveAccountfails (e.g. unresolved SecretRef). Wrapped in try/catch in both call sites so health RPCs don't crash hard on config errors.P2 — When a
probe:truerequest arrives during an in-flightprobe:falserefresh, the queued follow-up now uses the latest caller's probe intent instead of the original caller's value. AddedpendingProbetracking.Co-Authored-By: Claude Opus 4 [email protected]