Skip to content

Avoid post-run auth success lane delay#85829

Merged
vincentkoc merged 3 commits into
openclaw:mainfrom
TurboTheTurtle:fix/post-run-auth-success-nonblocking-85822
Jun 28, 2026
Merged

Avoid post-run auth success lane delay#85829
vincentkoc merged 3 commits into
openclaw:mainfrom
TurboTheTurtle:fix/post-run-auth-success-nonblocking-85822

Conversation

@TurboTheTurtle

@TurboTheTurtle TurboTheTurtle commented May 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • move post-run auth-profile success bookkeeping off the reply-blocking embedded-run path
  • log slow or failed post-run auth-success bookkeeping so future lock contention is visible
  • redact auth profile IDs in the new slow/failure diagnostics
  • add regression coverage proving a completed embedded run returns while auth-success bookkeeping is still pending

Fixes #85822

Real behavior proof

  • Behavior or issue addressed: A completed embedded runner turn should return its reply without waiting for post-run auth-profile success bookkeeping. The follow-up fix also keeps auth-profile IDs redacted in the new slow/failure logs.
  • Real environment tested: Local OpenClaw checkout on macOS, branch fix/post-run-auth-success-nonblocking-85822, head 25abd9b133e837d929f8fae8dc6977618325344a.
  • Exact steps or command run after this patch:
    • /Users/andy/openclaw/openclaw-85521/node_modules/.bin/oxfmt --write --threads=1 src/agents/pi-embedded-runner/run.ts
    • node scripts/run-vitest.mjs src/agents/pi-embedded-runner/run.overflow-compaction.loop.test.ts src/agents/auth-profiles/profiles.test.ts src/process/command-queue.test.ts
    • node scripts/run-vitest.mjs src/agents/pi-embedded-runner/run/attempt-stage-timing.test.ts src/logging/diagnostic-stuck-session-recovery.runtime.test.ts
    • node scripts/check-changed.mjs --base upstream/main
    • node scripts/pr85829-live-proof.mjs from a temporary PR worktree with local-only proof instrumentation and a 3000ms auth-success bookkeeping delay
  • Evidence after fix: Terminal output from the after-fix checkout:
RUN  v4.1.7 /Users/andy/openclaw/openclaw-85822

Test Files  5 passed (5)
Tests  94 passed (94)

RUN  v4.1.7 /Users/andy/openclaw/openclaw-85822

Test Files  2 passed (2)
Tests  20 passed (20)

[check:changed] lanes=core, coreTests
[check:changed] src/agents/pi-embedded-runner/run.overflow-compaction.loop.test.ts: core test
[check:changed] src/agents/pi-embedded-runner/run.ts: core production
[check:changed] runtime-sidecar-loaders: local runtime sidecar loaders look OK.
Import cycle check: 0 runtime value cycle(s).
  • Live gateway-backed ordering proof: I also ran a local OpenClaw gateway on an ephemeral loopback port with qa-channel durable transport, an isolated OPENCLAW_HOME, and a mock OpenAI Responses endpoint. The only extra local instrumentation added proof markers and injected a 3000ms delay inside post-run auth-profile success bookkeeping after embedded run completion.
2026-05-24T02:23:23.758Z [pr85829-proof] gateway ready: http://127.0.0.1:<gateway-port>/readyz
2026-05-24T02:23:23.758Z [pr85829-proof] command start: node scripts/run-node.mjs agent --agent qa --message Return exactly PR85829_LIVE_PROOF_OK. --model openai/gpt-5.5 --deliver --reply-channel qa-channel --reply-to channel:<proof-room> --timeout 60
2026-05-24T02:23:29.309Z [gateway stdout] 2026-05-23T19:23:29.309-07:00 [ws] ⇄ res ✓ agent 206ms runId=<redacted> conn=<redacted> id=<redacted>
2026-05-24T02:23:38.253Z [gateway stderr] 2026-05-23T19:23:38.253-07:00 [agent/embedded] [pr85829-proof] embedded run done: runId=<redacted> sessionId=<redacted> durationMs=8421 aborted=false
2026-05-24T02:23:38.255Z [gateway stderr] 2026-05-23T19:23:38.254-07:00 [agent/embedded] [pr85829-proof] delaying post-run auth-profile success bookkeeping: runId=<redacted> sessionId=<redacted> delayMs=3000 provider=openai profileId=<redacted>
2026-05-24T02:23:38.466Z [gateway stdout] 2026-05-23T19:23:38.465-07:00 [pr85829-proof] runEmbeddedPiAgent result available to delivery path: runId=<redacted> session=<redacted> durationMs=8425
2026-05-24T02:23:38.468Z [gateway stdout] 2026-05-23T19:23:38.467-07:00 [pr85829-proof] durable delivery start: runId=<redacted> channel=qa-channel payloads=1
2026-05-24T02:23:38.474Z [qa-bus stdout] [pr85829-proof] qa-channel outbound accepted: messageId=qa-msg-1 to=channel:<proof-room>
2026-05-24T02:23:38.477Z [gateway stdout] 2026-05-23T19:23:38.476-07:00 [pr85829-proof] durable delivery completed: runId=<redacted> status=sent elapsedMs=9
2026-05-24T02:23:38.530Z [pr85829-proof] command exit: code=0
2026-05-24T02:23:41.313Z [gateway stderr] 2026-05-23T19:23:41.312-07:00 [agent/embedded] post-run auth-profile success bookkeeping completed after 3055ms: runId=<redacted> sessionId=<redacted> provider=openai profileId=<redacted>
2026-05-24T02:23:43.271Z [pr85829-proof] delivered messages: 1
  • Observed result after fix: The unit regression exercises the real runEmbeddedPiAgent path with auth-success bookkeeping intentionally left unresolved, and the run still resolves successfully instead of waiting for that bookkeeping promise. The live gateway-backed proof shows durable qa-channel delivery completed at 02:23:38.477Z, while delayed post-run auth-success bookkeeping completed at 02:23:41.313Z, about 2.8s later.
  • What was not tested: This was gateway-backed qa-channel transport rather than a live Discord API send, so Discord-specific API behavior was not exercised. The gateway reply/durable-delivery ordering ClawSweeper requested is covered.

Validation

  • /Users/andy/openclaw/openclaw-85521/node_modules/.bin/oxfmt --write --threads=1 src/agents/pi-embedded-runner/run.ts
  • node scripts/run-vitest.mjs src/agents/pi-embedded-runner/run.overflow-compaction.loop.test.ts src/agents/auth-profiles/profiles.test.ts src/process/command-queue.test.ts
    • Test Files: 5 passed (5)
    • Tests: 94 passed (94)
  • node scripts/run-vitest.mjs src/agents/pi-embedded-runner/run/attempt-stage-timing.test.ts src/logging/diagnostic-stuck-session-recovery.runtime.test.ts
    • Test Files: 2 passed (2)
    • Tests: 20 passed (20)
  • node scripts/check-changed.mjs --base upstream/main
  • node scripts/pr85829-live-proof.mjs
    • gateway-backed qa-channel delivery observed
    • delivered messages: 1
    • delayed auth-success bookkeeping completed after delivery

Attribution

If this PR is squashed or reworked, please preserve author attribution or include:

Co-authored-by: Andy Ye [email protected]

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 23, 2026
@clawsweeper

clawsweeper Bot commented May 23, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 28, 2026, 6:59 PM ET / 22:59 UTC.

Summary
The PR moves post-run auth-profile success bookkeeping in runEmbeddedAgent from an awaited step to background work, adds slow/failure logging with redacted profile IDs, and adds a regression test for nonblocking return ordering.

PR surface: Source +34, Tests +17. Total +51 across 2 files.

Reproducibility: yes. at source level: current main and v2026.6.10 await markAuthProfileSuccess immediately after embedded run done, and the PR proof shows delivery before delayed success bookkeeping finishes. I did not run a fresh live Discord reproduction in this read-only review.

Review metrics: 1 noteworthy metric.

  • Auth-success timing change: 1 awaited persisted auth-state write moved to background. This is the central merge decision because auth metadata no longer has to persist before the embedded run result can return.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #85822
Summary: This PR is the open candidate fix for the canonical post-run lane-retention issue; stale-lock recovery and nonfatal-save work are related but do not supersede this branch.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Get an explicit maintainer decision on fire-and-forget versus drainable background auth-success bookkeeping.

Risk before merge

  • [P1] Reply delivery can now complete before auth-success lastGood and usageStats persistence; a crash or immediate shutdown before the background promise settles can leave that metadata stale.
  • [P1] The after-fix proof covers the shared gateway-backed qa-channel ordering path, not a fresh live Discord API send on this exact head.

Maintainer options:

  1. Make the background auth write drainable
    Track the auth-success promise in a bounded shutdown or post-terminal lifecycle so replies return promptly while restart/shutdown can still wait for metadata persistence.
  2. Accept stale metadata on crash
    Maintainers can intentionally land the current fire-and-forget behavior if faster reply delivery is worth the chance that lastGood and usage stats lag after a crash or immediate shutdown.
  3. Pause for auth-state ownership
    If neither tradeoff is acceptable, pause this PR until the owner boundary for nonblocking auth bookkeeping is decided.

Next step before merge

  • [P2] Human maintainer review should accept or reshape the auth/session-state durability tradeoff; there is no narrow automated repair required right now.

Security
Cleared: No concrete security or supply-chain issue was found; the diff touches auth bookkeeping but adds no dependency, workflow, permission, token, or secret-handling surface and redacts profile IDs in new logs.

Review details

Best possible solution:

Land the latency fix after maintainers either accept the bounded stale-auth-metadata risk or require the background auth-success write to be tracked by a drainable lifecycle without re-blocking reply delivery.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main and v2026.6.10 await markAuthProfileSuccess immediately after embedded run done, and the PR proof shows delivery before delayed success bookkeeping finishes. I did not run a fresh live Discord reproduction in this read-only review.

Is this the best way to solve the issue?

Acceptable mitigation, with one maintainer decision remaining. Targeting shared embedded-runner auth bookkeeping is better than a Discord-only workaround, but maintainers must decide whether fire-and-forget persistence is acceptable or should be made drainable.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 87aae9f11d27.

Label changes

Label justifications:

  • P1: The PR targets a real agent/channel workflow delay where replies can sit for roughly 50 seconds after the model run has completed.
  • merge-risk: 🚨 auth-provider: The diff changes when successful auth-profile lastGood and usage metadata are persisted after provider use.
  • merge-risk: 🚨 session-state: The diff changes post-run bookkeeping ordering relative to reply delivery and gateway restart or shutdown durability.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The PR body and follow-up include after-fix terminal output plus gateway-backed qa-channel logs showing delivery completed before injected delayed auth-success bookkeeping finished.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body and follow-up include after-fix terminal output plus gateway-backed qa-channel logs showing delivery completed before injected delayed auth-success bookkeeping finished.
Evidence reviewed

PR surface:

Source +34, Tests +17. Total +51 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 46 12 +34
Tests 1 17 0 +17
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 63 12 +51

What I checked:

Likely related people:

  • mcaxtr: Authored and merged the PR that added markAuthProfileSuccess and moved successful embedded runs to the locked success-write path now under review. (role: introduced auth-success consolidation; confidence: high; commits: 78b9b1d5502d; files: src/agents/auth-profiles/profiles.ts, src/plugin-sdk/agent-runtime.ts)
  • TurboTheTurtle: Authored this PR and previously contributed merged adjacent embedded-runner/session recovery work in the same runtime area. (role: linked repair author and adjacent area contributor; confidence: medium; commits: 65b697981620, 2b0b940e31dd, fe90336c68c3; files: src/agents/embedded-agent-runner/run.ts, src/agents/embedded-agent-runner/run.overflow-compaction.loop.test.ts, src/agents/command/session-store.ts)
  • Alix-007: Authored the merged stale-lock expiry fix tied to the same production diagnosis; that work reduces one cause of the tail but does not replace this nonblocking reply-path change. (role: adjacent lock recovery contributor; confidence: medium; commits: daa7b1d06ba0; files: src/infra/stale-lock-file.ts, src/plugin-sdk/file-lock.ts, src/infra/stale-lock-file.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels May 23, 2026
@clawsweeper

clawsweeper Bot commented May 23, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

✨ Hatched: 🌱 uncommon Tiny Shellbean

Hatch command

Comment @clawsweeper hatch when this PR is hatchable.

Hatchability rules:

  • Merged PRs are hatchable.
  • Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
  • Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

Rarity: 🌱 uncommon.
Trait: hums during re-review.
Image traits: location CI tidepool; accessory commit compass; palette sunrise gold and clean white; mood celebratory; pose balancing on a branch marker; shell polished stone shell; lighting golden review-room light; background delicate sparkle particles.
Share on X: post this hatch
Copy: My PR egg hatched a 🌱 uncommon Tiny Shellbean in ClawSweeper.

What is this egg doing here?
  • Eggs appear after the PR passes real-behavior proof. It is here for vibes, not verdicts: it does not change labels, ratings, merge decisions, or automation.
  • The shell reacts to review momentum: open follow-up work warms it up, re-review makes it wobble, and a clean final review lets it hatch.
  • Hatchability usually comes from sufficient real-behavior proof, no blocking P0/P1/P2 findings, no security attention needed, and clean correctness. A merged PR is already final, so merge makes the egg hatchable independently.
  • The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
  • Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 23, 2026
@clawsweeper clawsweeper Bot added merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. and removed merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels May 23, 2026

Copy link
Copy Markdown
Contributor Author

Added the live gateway-backed proof ClawSweeper asked for to the PR body.

Tested commit: 25abd9b133e837d929f8fae8dc6977618325344a
Environment: local OpenClaw gateway on loopback with qa-channel durable transport, isolated OPENCLAW_HOME, and mock OpenAI Responses provider
Delay injected: 3000ms in post-run auth-profile success bookkeeping via local-only proof instrumentation

The redacted ordering proof shows durable delivery completed at 02:23:38.477Z, while delayed auth-success bookkeeping completed at 02:23:41.313Z, about 2.8s later.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels May 24, 2026
@0xCNAI

0xCNAI commented May 26, 2026

Copy link
Copy Markdown

Production confirmation from 2026.5.22: making markAuthProfileSuccess() non-reply-blocking is the right fix for the Discord latency shape.

We applied the equivalent local dist hotfix plus a separate stale-lock fix and restarted Gateway. Discord delivery dropped from the previous ~50-60s post-run tail to:

  • 16:02:17 inbound -> 16:02:27 delivered, ~9.4s
  • 16:02:46 inbound -> 16:02:52 delivered, ~5.7s

The underlying waiter in our case was a stale auth-profiles.json.lock whose PID had been reused by macOS, so markAuthProfileSuccess() could block on file-lock retry/recovery after embedded run done. This PR fixes the user-visible reply-path impact even if that bookkeeping gets stuck again.

Separate stale-lock bug filed as #86814 because current src/infra/stale-lock-file.ts still checks PID before createdAt, so an expired lock can be retained when the OS reuses the recorded PID.

@BingqingLyu

This comment was marked as spam.

@TurboTheTurtle
TurboTheTurtle force-pushed the fix/post-run-auth-success-nonblocking-85822 branch from 25abd9b to 6bf4978 Compare May 29, 2026 17:24
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 29, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 29, 2026
@TurboTheTurtle
TurboTheTurtle force-pushed the fix/post-run-auth-success-nonblocking-85822 branch from 6bf4978 to 4be7576 Compare May 31, 2026 19:49
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 31, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 31, 2026
@TurboTheTurtle
TurboTheTurtle force-pushed the fix/post-run-auth-success-nonblocking-85822 branch from 0a5e39b to 37d534c Compare June 14, 2026 20:02
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 14, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 14, 2026
@TurboTheTurtle
TurboTheTurtle force-pushed the fix/post-run-auth-success-nonblocking-85822 branch from 37d534c to b152946 Compare June 18, 2026 06:48
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 18, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 18, 2026
@TurboTheTurtle
TurboTheTurtle force-pushed the fix/post-run-auth-success-nonblocking-85822 branch from b152946 to fe90336 Compare June 28, 2026 22:48
@vincentkoc
vincentkoc merged commit 68b533d into openclaw:main Jun 28, 2026
93 checks passed
@TurboTheTurtle
TurboTheTurtle deleted the fix/post-run-auth-success-nonblocking-85822 branch June 29, 2026 02:27
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 29, 2026
* fix: avoid post-run auth success lane delay

* fix: redact post-run auth profile logs

* Fix embedded runner auth-success test rename
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
* fix: avoid post-run auth success lane delay

* fix: redact post-run auth profile logs

* Fix embedded runner auth-success test rename
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
* fix: avoid post-run auth success lane delay

* fix: redact post-run auth profile logs

* Fix embedded runner auth-success test rename
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
* fix: avoid post-run auth success lane delay

* fix: redact post-run auth profile logs

* Fix embedded runner auth-success test rename
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Silent ~48s post-embedded run done lane retention on Discord turns (2026.5.20) — no trace-level emit in the gap

4 participants