Skip to content

Fix Discord queue backpressure and related read-path bottlenecks#79562

Closed
neuroradgist wants to merge 9 commits into
openclaw:mainfrom
neuroradgist:fix/discord-queue-backpressure
Closed

Fix Discord queue backpressure and related read-path bottlenecks#79562
neuroradgist wants to merge 9 commits into
openclaw:mainfrom
neuroradgist:fix/discord-queue-backpressure

Conversation

@neuroradgist

@neuroradgist neuroradgist commented May 8, 2026

Copy link
Copy Markdown

Adds regression coverage and targeted performance/backpressure fixes for Discord queues, media downloads, preview writes, transcript idempotency, Codex RPC writes, and session-store reads.

Real behavior proof

Behavior or issue addressed:
This PR addresses real backpressure/read-path risk in the OpenClaw runtime: Discord queues and media handling can build up work, Codex app-server stdin writes can ignore backpressure, transcript idempotency checks can repeatedly scan transcript files, and session-history reads can synchronously load more session-store data than needed.

Real environment tested:
Real local OpenClaw checkout on macOS at /Users/openclaw/dev/openclaw, on branch fix/discord-queue-backpressure, pushed to neuroradgist/openclaw and opened as #79562.

Exact steps or command run after this patch:
Ran these commands from the real checkout after the patch:

  • git status
  • git log --oneline -8
  • gh pr checks 79562 --repo openclaw/openclaw --watch
  • pnpm lint --threads=8
  • pnpm run lint:extensions:bundled
  • pnpm tsgo:core
  • pnpm tsgo:test
  • pnpm test src/config/sessions/transcript.test.ts extensions/discord/src/monitor/message-utils.test.ts extensions/discord/src/monitor/message-run-queue.test.ts -- --reporter=verbose

Evidence after fix:
Copied terminal output from the real checkout:

git status
On branch fix/discord-queue-backpressure
Your branch is up to date with 'neuroradgist/fix/discord-queue-backpressure'.

gh pr checks 79562 --repo openclaw/openclaw --watch
89 successful, 19 skipped, and 0 pending checks.
CI/check passed.
CI/check-additional passed.
CI/check-lint passed.
CI/check-prod-types passed.
CI/check-test-types passed.
CI/check-additional-extension-bundled passed.

pnpm lint --threads=8
Found 0 warnings and 0 errors.

pnpm run lint:extensions:bundled
Found 0 warnings and 0 errors.

pnpm tsgo:core
passed.

pnpm tsgo:test
passed.

pnpm test src/config/sessions/transcript.test.ts extensions/discord/src/monitor/message-utils.test.ts extensions/discord/src/monitor/message-run-queue.test.ts -- --reporter=verbose
[test] passed 2 Vitest shards in 4.91s.
Runtime-config transcript tests: 16 passed.
Extension-discord message/media/run-queue tests: 49 passed.

Observed result after fix:
The PR branch is pushed and the affected code checks are green on GitHub Actions. The earlier type and lint failures were fixed. The real checkout shows the branch clean/up to date after pushing, and the PR checks show no pending code checks.

What was not tested:
I did not run a live Discord bot against production Discord traffic, did not run a live Codex app-server process, and did not perform a long-duration production load test.

@openclaw-barnacle openclaw-barnacle Bot added channel: discord Channel integration: discord gateway Gateway runtime extensions: codex size: XL triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 8, 2026
@clawsweeper

clawsweeper Bot commented May 8, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 3, 2026, 11:47 PM ET / 03:47 UTC.

Summary
The PR changes Discord queue/backpressure handling, media and preview write behavior, Codex app-server RPC writes/timeouts, targeted session-store reads, transcript idempotency indexing, and related tests/config schemas.

PR surface: Source +577, Tests +772. Total +1349 across 22 files.

Reproducibility: yes. for the review findings by source inspection: current main and the PR diff differ on Discord replay/cleanup, Codex timeout semantics, transcript sidecar state, and generated config metadata; no live reproduction was established for the original load claim.

Review metrics: 3 noteworthy metrics.

  • Discord Queue Config Surfaces: 2 added. The max-pending and queued-age keys expose operator-controlled drop behavior that needs compatibility, metadata, and docs review before merge.
  • Transcript Sidecar State Formats: 1 added. A new runtime sidecar changes session-state ownership, cleanup, rebuild, and migration expectations.
  • Codex Omitted-Timeout Default: 1 changed. Callers that omit timeoutMs would get a 60s deadline instead of current no-client-deadline behavior.

Stored data model
Persistent data-model change detected: persistent cache schema: src/config/sessions.cache.test.ts, serialized state: extensions/codex/src/app-server/client.test.ts, serialized state: src/config/sessions.cache.test.ts, serialized state: src/config/sessions/store-load.ts, serialized state: src/config/sessions/transcript.test.ts, serialized state: src/config/sessions/transcript.ts, and 5 more. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Add redacted live Discord/Gateway/Codex proof or linked artifacts showing the changed behavior after the fix.
  • [P1] Rebase or split the dirty branch against current main so each owner-bounded fix can be reviewed independently.
  • Preserve existing retry, cleanup, timeout, and transcript-storage semantics unless maintainers explicitly approve a new policy.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body supplies terminal output for CI, lint, typecheck, and unit tests only, and explicitly omits live Discord, live Codex app-server, and long-duration load proof. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Mantis proof suggestion
A redacted live Discord/Gateway artifact would materially help prove same-session messages still deliver correctly under queue pressure and drop settings. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis visual task: capture redacted Discord/Gateway proof that same-session messages still deliver in order when queue caps and stale-age settings are exercised.

Risk before merge

  • [P1] The new Discord queue cap and stale-age paths can mark unprocessed inbound deliveries as consumed, suppressing retry under pressure.
  • [P1] Deactivation can clear accepted but unstarted Discord queue jobs without current replay-release and typing cleanup behavior.
  • [P1] Omitted Codex app-server request timeouts change from caller-opt-in to a blanket 60s client deadline without an upstream protocol contract for that default.
  • [P1] The transcript idempotency sidecar adds persistent runtime state outside current transcript/session storage ownership and cleanup paths.
  • [P1] The new Discord queue config keys expose operator-controlled drop behavior and need generated metadata, docs, and upgrade proof if kept.
  • [P1] The PR head is conflicting/dirty against main, so the reviewed branch is not a reliable final merge result.
  • [P1] The supplied proof is lint/typecheck/CI/unit-test output only and explicitly omits live Discord, live Codex app-server, and long-duration load proof.

Maintainer options:

  1. Split And Repair Before Merge (recommended)
    Preserve skipped Discord cleanup and replay semantics, remove or make explicit the blanket Codex timeout, redesign or remove the transcript sidecar, refresh generated config metadata if queue keys stay, and re-review after rebasing.
  2. Accept New Runtime Policies Explicitly
    Maintainers could intentionally accept queue drops that consume replay keys, transcript idempotency sidecar state, and omitted Codex deadlines only with documented semantics, upgrade proof, and live behavior evidence.
  3. Close In Favor Of Narrower PRs
    If the broad conflicting branch cannot be repaired cleanly, close it and land independent media, preview, session-read, or Codex write-backpressure fixes as smaller PRs.

Next step before merge

  • [P1] Human review is needed because the blockers are proof, merge conflicts, delivery/storage/Codex semantics, and product/upgrade policy rather than one safe mechanical repair.

Security
Cleared: No concrete security or supply-chain issue was found; the diff does not change dependencies, lockfiles, workflows, publishing scripts, permissions, or secrets handling.

Review findings

  • [P1] Do not consume unprocessed Discord queue drops — extensions/discord/src/monitor/message-run-queue.ts:107-127
  • [P1] Preserve skipped queued-message cleanup — extensions/discord/src/monitor/message-run-queue.ts:138-140
  • [P1] Keep Codex RPC timeouts caller-opt-in — extensions/codex/src/app-server/client.ts:541-543
Review details

Best possible solution:

Split or rework this into narrow owner-bounded PRs that preserve retry and cleanup semantics, avoid blanket Codex deadlines, keep transcript idempotency in owned storage, align generated config metadata if queue controls stay, and add redacted live behavior proof.

Do we have a high-confidence way to reproduce the issue?

Yes for the review findings by source inspection: current main and the PR diff differ on Discord replay/cleanup, Codex timeout semantics, transcript sidecar state, and generated config metadata; no live reproduction was established for the original load claim.

Is this the best way to solve the issue?

No; this is a broad mitigation rather than the best fix, and the safer path is smaller owner-bounded changes that preserve existing delivery, timeout, and storage contracts unless maintainers explicitly approve new policy.

Full review comments:

  • [P1] Do not consume unprocessed Discord queue drops — extensions/discord/src/monitor/message-run-queue.ts:107-127
    This commits replay keys when the queue cap is exceeded and again for stale jobs before they reach processDiscordQueuedMessage. Current skipped-job cleanup releases replay claims for accepted but not-started work, so committing here can suppress retry for a Discord delivery that was never processed.
    Confidence: 0.92
  • [P1] Preserve skipped queued-message cleanup — extensions/discord/src/monitor/message-run-queue.ts:138-140
    deactivate() now clears pending jobs and deactivates the run queue without running the current cleanup that releases replay claims and cleans reply typing feedback for accepted but not-started jobs. On shutdown or abort, those jobs can leave stale replay or typing state that affects later delivery behavior.
    Confidence: 0.9
  • [P1] Keep Codex RPC timeouts caller-opt-in — extensions/codex/src/app-server/client.ts:541-543
    When timeoutMs is omitted, this helper now gives every app-server RPC a 60s deadline. Current main only times out calls that explicitly pass a timeout, and the checked sibling Codex app-server source does not define a universal 60s JSON-RPC deadline, so legitimate long-running operations can fail without caller opt-in.
    Confidence: 0.9
  • [P2] Keep transcript idempotency in owned storage — src/config/sessions/transcript.ts:449-452
    This writes a runtime ${transcriptPath}.idempotency.json sidecar next to the transcript. OpenClaw-owned runtime state/cache needs clear storage ownership, cleanup, and rebuild semantics before adding another file format beside transcript files.
    Confidence: 0.86
  • [P2] Regenerate bundled Discord config metadata — src/config/zod-schema.providers-core.ts:714-715
    The PR accepts maxPendingPerSession and maxQueuedAgeMs in the runtime schema, but bundled generated channel config metadata still only exposes inboundWorker.runTimeoutMs. Packaged config/help surfaces can drift from runtime unless the generated metadata is updated or the new config surface is removed.
    Confidence: 0.82

Overall correctness: patch is incorrect
Overall confidence: 0.91

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9d68f877ac3e.

Label changes

Label justifications:

  • P1: The PR changes active Discord delivery, session transcript state, and Codex app-server behavior with blockers that can affect real workflows.
  • merge-risk: 🚨 compatibility: The PR adds Discord config surfaces and changes omitted Codex app-server timeout behavior for existing callers.
  • merge-risk: 🚨 message-delivery: The new Discord queue cap and stale-age paths can mark unprocessed deliveries as consumed rather than retryable.
  • merge-risk: 🚨 session-state: The PR changes skipped queued-job cleanup and adds transcript idempotency sidecar state outside the current storage path.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body supplies terminal output for CI, lint, typecheck, and unit tests only, and explicitly omits live Discord, live Codex app-server, and long-duration load proof. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +577, Tests +772. Total +1349 across 22 files.

View PR surface stats
Area Files Added Removed Net
Source 13 670 93 +577
Tests 9 772 0 +772
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 22 1442 93 +1349

Acceptance criteria:

  • [P1] Re-run targeted Discord queue tests after preserving retry/cleanup semantics.
  • [P2] Run focused Codex app-server client tests after removing or making explicit any timeout change.
  • [P1] Validate generated config metadata if Discord inboundWorker keys remain.
  • [P1] Collect redacted live Discord/Gateway and Codex app-server proof for the changed runtime paths.

What I checked:

Likely related people:

  • steipete: Authored recent shared Discord queue and channel run-timeout removal commits in the lifecycle boundary this PR changes. (role: recent Discord queue and channel lifecycle contributor; confidence: high; commits: 364c67bcb5b7, 07631fb931fc; files: extensions/discord/src/monitor/message-run-queue.ts, extensions/discord/src/monitor/timeouts.ts, src/plugin-sdk/channel-lifecycle.core.ts)
  • zhuisDEV: Authored the recent queued reply typing feedback cleanup work directly adjacent to the skipped-job cleanup this PR removes. (role: recent Discord queue contributor; confidence: high; commits: 6f20f29688d7; files: extensions/discord/src/monitor/message-run-queue.ts)
  • kevinslin: Authored recent remote app-server plugin support in the same Codex client path whose request timeout and write behavior this PR changes. (role: recent Codex app-server contributor; confidence: medium; commits: bc5081c58730; files: extensions/codex/src/app-server/client.ts)
  • jalehman: Authored recent transcript writer seam refactoring adjacent to the idempotency storage path changed here. (role: recent transcript writer seam contributor; confidence: medium; commits: 00a75db4280b; files: src/config/sessions/transcript.ts)
  • vincentkoc: Authored recent delivery mirror dedupe work adjacent to the transcript idempotency and channel-final behavior touched by the sidecar change. (role: adjacent delivery mirror contributor; confidence: medium; commits: f5eddc2b6d1f; files: src/config/sessions/transcript.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 9, 2026
@neuroradgist
neuroradgist force-pushed the fix/discord-queue-backpressure branch 2 times, most recently from 7085180 to 9575627 Compare May 9, 2026 05:55
@neuroradgist
neuroradgist force-pushed the fix/discord-queue-backpressure branch from 9575627 to bd224fb Compare May 9, 2026 06:00
@neuroradgist

Copy link
Copy Markdown
Author

Checks are passing and the branch is up to date with main. Ready for maintainer review/merge when convenient.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 1, 2026
@clawsweeper clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label Jun 1, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jun 2, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 14, 2026
@clawsweeper clawsweeper Bot added merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 14, 2026
@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution. I’m closing this broad branch because it changes Discord delivery, session state, configuration, and Codex timeout behavior without the live proof needed for those compatibility-sensitive surfaces. Please split any still-current fixes into focused PRs against current main with surface-specific proof.

@steipete steipete closed this Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: discord Channel integration: discord extensions: codex gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants