Skip to content

fix(auto-reply): per-model runtime policy in memory-flush gates#86361

Closed
potterdigital wants to merge 1 commit into
openclaw:mainfrom
potterdigital:fix/memory-flush-honor-cli-runtime-policy
Closed

fix(auto-reply): per-model runtime policy in memory-flush gates#86361
potterdigital wants to merge 1 commit into
openclaw:mainfrom
potterdigital:fix/memory-flush-honor-cli-runtime-policy

Conversation

@potterdigital

@potterdigital potterdigital commented May 25, 2026

Copy link
Copy Markdown
Contributor

Summary

runMemoryFlushIfNeeded and runPreflightCompactionIfNeeded in src/auto-reply/reply/agent-runner-memory.ts gate the embedded memory-flush dispatch on isCliProvider(run.provider, cfg). That helper only inspects agents.defaults.cliBackends keys — it never resolves the per-model agents.defaults.models[<key>].agentRuntime.id policy that resolveCliRuntimeExecutionProvider and maybeCompactAgentHarnessSession already honor.

A session with run.provider = "anthropic" and agents.defaults.models["anthropic/<model>"].agentRuntime.id = "claude-cli" reports "not CLI" at the gate and dispatches runEmbeddedPiAgent directly, even though every other dispatch path for the same session correctly routes through the claude CLI subprocess. The embedded harness then submits whatever Anthropic-prefixed auth profile it picks to /v1/messages. When the profile is a Claude Code OAuth subscription token (sk-ant-oat01-…) — common in user configs because openclaw auth login --provider anthropic stores subscription tokens under provider: anthropic, type: token — Anthropic rejects with 404 model_not_found by design (subscription tokens are only valid through the claude CLI subprocess). The 404 cascades through every anthropic auth profile into a 40-minute cooldown, and the agent stops replying on the affected lane until the cooldown ages out.

This change resolves resolveAgentHarnessPolicy().runtime and combines the result with isCliRuntimeAliasForProvider in both gates, matching the contract that resolveCliRuntimeExecutionProvider and maybeCompactAgentHarnessSession already use.

Scope note on the preflight change. The runPreflightCompactionIfNeeded gate has the same defective isCliProvider-only check, but its downstream call into maybeCompactAgentHarnessSession (at src/agents/harness/selection.ts:471) already short-circuits via isCliRuntimeAliasForProvider for CLI-routed sessions today. That means the preflight bypass is currently protected by the downstream gate, not the local one. This PR fixes the local gate too as defense-in-depth: the defect class is identical, the fix is mechanically identical, and a future refactor of the downstream protection would otherwise re-expose the bypass silently. Reviewers preferring strict-scope can request the preflight change be split into its own PR — happy to do it that way.

Real behavior proof

Behavior addressed: Slack DM lane wedged with 404 status code (no body) on anthropic/claude-opus-4-7[1m] after a long claude-cli turn, with all anthropic auth profiles cooled down for 40 minutes despite the model being pinned to the claude-cli runtime via agents.defaults.models["anthropic/<model>"].agentRuntime.id.

Real environment tested: production gateway on Proxmox LXC (openclaw v2026.5.20, node v24.13.0) running live Slack DMs for a single user across 8 agents. Config has agents.defaults.cliBackends.claude-cli.command = /usr/local/bin/claude and agents.defaults.models["anthropic/claude-opus-4-7[1m]"].agentRuntime.id = "claude-cli".

Exact steps or command run after this patch:

  • Patched the production dist/model-runtime-aliases-*.js with a temporary [DBG-CLI-RESOLVE] log line on every call to resolveCliRuntimeExecutionProvider so the dispatch resolver could be observed live.
  • Bundle-patched the live dist/agent-runner.runtime-*.js (the compiled agent-runner-memory.ts equivalent of this PR) on the production gateway and restarted via systemctl --user restart openclaw-gateway.
  • Drove a fresh Slack DM thread for ~50 minutes through Dobby (the main agent on opus-4-7[1m]) until the chat UI reported 100% context used 1.3M / 1M — the exact precondition for the previous wedge.
  • Continued sending user messages on the same thread after the 100% reading.
  • node scripts/run-vitest.mjs src/auto-reply/reply/agent-runner-memory.test.ts — 28/28 passing (26 existing + 2 new regression tests + a non-CLI-runtime negative test + a non-anthropic CLI-alias test added in response to adversarial review).
  • Regression-capability proof: git stash the source change, re-run the 4 new tests → all fail loudly with runEmbeddedPiAgent mock unexpectedly called. git stash pop → all pass.
  • pnpm tsgo:core — clean.
  • pnpm check:changed — clean on all lanes except a pre-existing src/plugins/document-extractors.runtime.test.ts:53 TS2554 on origin/main (unrelated; reproducible without this branch).

Evidence after fix:

  • Pre-fix production failure signature (run a96335a9-…): [agent/embedded] embedded run agent end isError=true model=claude-opus-4-7[1m] provider=anthropic error=404 status code (no body) followed by auth profile failure state updated profile=sha256:… reason=model_not_found windowType=cooldown against three separate auth profile hashes within 5 seconds, then FailoverError: No available auth profile for anthropic across the fallback chain. Zero [DBG-CLI-RESOLVE] entries in the 53 seconds preceding the embedded run — confirming the failing run bypassed resolveCliRuntimeExecutionProvider entirely. SessionId rotated from the slack thread's stored 03f422f8-… to a brand-new 52a6e154-…, which matches the post-flush incrementCompactionCount rotation produced inside runMemoryFlushIfNeeded.
  • Post-fix production verification, same gateway: 100% context used 1.3M / 1M reported by the chat UI, user message sent. Gateway log shows three [DBG-CLI-RESOLVE] result="claude-cli" lines (preflight gate, memory-flush gate, model-fallback assertion) followed immediately by [agent/cli-backend] cli exec: provider=claude-cli model=claude-opus-4-7[1m] trigger=user useResume=true session=present resumeSession=4a30bca18050 reuse=reusable. No [agent/embedded] events. No auth profile failure events. Claude-cli session resumed and a 277,173ms / 1912-rawLine assistant turn completed cleanly, followed by additional user turns continuing to dispatch through the CLI subprocess.
  • Multi-agent fleet trace during the same window: main (opus-4-7[1m]), forge, architect, researcher, gabe, gabe-voice, scribe all observed dispatching through [agent/cli-backend] for their cron heartbeats — every dispatch preceded by a [DBG-CLI-RESOLVE] … result="claude-cli" line — confirming the same gate fix protects every agent, not just the one used for the verification.
  • Regression-capability proof in the test suite: the two new regression tests fail with AssertionError: expected runEmbeddedPiAgentMock not to be called when the source change is reverted; both pass when the source change is applied.

Observed result after fix: legitimate cron and slack-DM turns continue to resolve to claude-cli via the existing dispatch path; the memory-flush and preflight-compaction gates now early-return for sessions whose per-model runtime policy pins them to a CLI runtime, so the embedded harness can no longer be invoked with OAuth subscription tokens through this code path. No 404 cascade has occurred on the production gateway since the fix was applied, including across the 1.3M-context boundary that previously wedged the lane.

7

What was not tested:

  • The codex / codex-cli paths in resolveAgentHarnessPolicy. The fix's added check is isCliRuntimeAliasForProvider, which evaluates against the static CLI_RUNTIME_BY_PROVIDER map. Codex entries are cli: false in that map, so they never trip the new gate condition — but no codex-routed lane was available for live verification on the test gateway. The new tests cover anthropic + google CLI-alias positive cases and the pi runtime negative case to bound the behavior.
  • The runEmbeddedPiAgent paths in src/commitments/runtime.ts, src/hooks/llm-slug-generator.ts, src/crestodian/assistant.ts, and src/talk/agent-consult-runtime.ts. These use synthetic sessionKeys (temp:…, commitments:…) and do not currently route through the slack-DM autoreply path that triggered the production failure. They are out of scope for this PR but share the broader pattern of calling runEmbeddedPiAgent without first consulting resolveCliRuntimeExecutionProvider; flagging as a separate audit item.

Companion misconfiguration worth flagging (not in scope for this PR, addressed separately on the affected gateway): when auth.profiles stores OAuth subscription tokens (sk-ant-oat01-…) under { provider: "anthropic", type: "token" } rather than { provider: "claude-cli", type: "oauth" }, the embedded harness — when reached — will pick those profiles and submit subscription tokens to /v1/messages, which 404s deterministically. The right user-facing fix is reclassification of those profiles (via openclaw config patch against the central openclaw.json auth.profiles and the per-agent auth-profiles.json credential records, since both layers exist). This PR's gate fix prevents the embedded harness from reaching those profiles for CLI-routed models regardless of how the profiles are typed, eliminating the cascade as a side effect of the configuration issue.

🤖 Generated with Claude Code

The memory-flush and preflight-compaction gates in agent-runner-memory.ts
used `isCliProvider(provider, cfg)`, which only inspects `cliBackends`
keys. A session with `provider="anthropic"` and a model whose
`agents.defaults.models[<key>].agentRuntime.id` was pinned to
"claude-cli" therefore reported "not CLI" and dispatched
`runEmbeddedPiAgent` directly. The embedded harness then sent the
claude-cli OAuth subscription token (sk-ant-oat01-...) to /v1/messages,
which Anthropic rejects with 404 model_not_found because OAuth
subscription tokens are only valid through the claude CLI subprocess.
The 404 cascaded through every anthropic auth profile into 40-minute
cooldowns, leaving the agent unable to reply until the cooldowns aged
out.

Resolve `resolveAgentHarnessPolicy().runtime` and combine with
`isCliRuntimeAliasForProvider` so the gate also honors the per-model
runtime policy, matching the contract used elsewhere in the dispatch
path (`resolveCliRuntimeExecutionProvider`,
`maybeCompactAgentHarnessSession`).
@openclaw-barnacle openclaw-barnacle Bot added size: S proof: supplied External PR includes structured after-fix real behavior proof. labels May 25, 2026
@clawsweeper

clawsweeper Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Keep open: the reported P1 memory-flush/runtime bug is valid, but this patch should not merge as-is because it adds a second partial CLI-runtime resolver instead of using the existing dispatch resolver. That leaves supported auth-profile and providerless matched-provider CLI routes on the embedded maintenance path, and an overlapping resolver-shaped PR is already open at #86224.

Canonical path: Close this PR as superseded by #86224.

So I’m closing this here and keeping the remaining discussion on #86224.

Review details

Best possible solution:

Close this PR as superseded by #86224.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main's gates only call isCliProvider on the canonical provider while the dispatch path resolves CLI runtime policy first. I did not rerun the live Slack scenario, but the PR body includes production gateway logs and regression-test reverse proof for the reported per-model path.

Is this the best way to solve the issue?

No, not as written: the change fixes the reported per-model slice but duplicates only part of the existing dispatch resolver. The narrower maintainable fix is to reuse resolveCliRuntimeExecutionProvider so auth-profile and providerless matched-provider routes stay aligned too.

Security review:

Security review cleared: The diff changes runtime gate logic and tests only; I found no new dependency, CI, script, secret-handling, or supply-chain surface.

AGENTS.md: found and applied where relevant.

What I checked:

  • linked superseding PR: [AI-assisted] fix(agents): resolve CLI runtime in preflight + memory-flush gates #86224 ([AI-assisted] fix(agents): resolve CLI runtime in preflight + memory-flush gates) is still open as the canonical replacement.
  • cluster evidence: the durable review links that PR in the work cluster or recommended risk path.
  • no human follow-up: live comments and timeline hydrated by apply contain no non-automation activity after the ClawSweeper review.

Likely related people:

  • TurboTheTurtle: GitHub metadata maps the blamed current-main commit for the memory gate and runtime-alias helper to Clean up browser MCP subprocess tree #85832, authored by TurboTheTurtle. (role: recent area contributor; confidence: medium; commits: 8dc6b4d33083; files: src/auto-reply/reply/agent-runner-memory.ts, src/agents/model-runtime-aliases.ts)
  • pashpashpash: Recently changed the same preflight compaction boundary for Codex sessions in Fail Codex compaction at the Codex boundary #85958. (role: recent adjacent contributor; confidence: medium; commits: dd47e479aedb; files: src/auto-reply/reply/agent-runner-memory.ts)
  • steipete: Merged the current blame commit for the touched memory/runtime area, so they are a useful routing candidate for cleanup decisions around that recent refactor. (role: merger of adjacent refactor; confidence: low; commits: 8dc6b4d33083; files: src/auto-reply/reply/agent-runner-memory.ts, src/agents/model-runtime-aliases.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 73189e3ecb0c.

@potterdigital

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels May 25, 2026
@clawsweeper

clawsweeper Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: S status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant