Summary

When memoryFlush triggers but the assistant model required to produce the handoff summary crashes (e.g. Codex app-server connection closed during startup), the flush silently fails. The session's on-disk transcript stays over threshold. Every subsequent user message hits the same wedge — memoryFlush check re-fires, the same flush attempt fails (because the same broken provider is still broken or the session is now even larger), and the user-facing channel sees no reply. No lifecycle escalation, no fallback path.

A gateway restart clears the in-memory stalled_agent_run but does NOT clear the on-disk bloat. Without a separate, manual transcript truncation, the next inbound message wedges again within seconds.

Symptom (one observed incident)

agent=forge, session c0b5jmpcq4s, Slack channel inbound, 2026-05-23 UTC:

04:38:41  memoryFlush check: tokenCount=163332 threshold=130000   ← flush needed
04:38:59  codex app-server connection closed during startup       ← assistant died mid-flush
04:39:25  retries exhausted                                       ← flush silently failed
04:41:50+ stalled session: state=processing                       ← wedge for next user msg
04:55:00  manual gateway restart (cleared in-memory wedge only)
04:57:00  user re-sent message
04:58:21  memoryFlush check: tokenCount=163327                    ← STILL bloated on disk
05:02:42  memoryFlush check: tokenCount=163327                    ← re-wedge, identical state
05:04:17  manual transcript truncation (~750KB → 128 bytes)       ← finally resolved

Two consecutive wedges on the same session in <30 minutes, both invisible to monitoring beyond the existing stalled_agent_run event (which doesn't surface the on-disk-bloat distinction).

Root cause

memoryFlush is implemented as an assistant turn — the gateway prompts the session's own assistant to write a handoff summary, then truncates older history once that summary is written. This is fine when the assistant is healthy. It fails closed when the assistant itself is the broken dependency:

1. Provider crash mid-flush → no summary produced → truncation never runs → on-disk transcript stays bloated.
2. The flush retry path uses the same provider, so retries don't help when the provider is what's broken.
3. There's no per-session tracker for "flush attempted at T, tokenCount=X, still over threshold at T+N".
4. No escalation to a model-independent reset (sessions cleanup / forced truncation) when flush keeps failing.

Result: the session is stuck in a permanent flush-attempt loop with no observable degradation signal beyond the underlying provider error (which is reported as a provider failure, not a session-state failure).

Reproduction

Hard to reproduce on demand, but the precondition is:

1. A session crosses memoryFlush threshold (>130k tokens in our config — softThresholdTokens semantics per openclaw-memoryflush-semantics-reversed).
2. The provider configured on the session is unhealthy at the exact moment flush fires (Codex app-server failed to start, OAuth refresh chain poisoned, rate-limited beyond retry budget, etc.).
3. Flush completes with retries-exhausted, the gateway logs the provider error, but the session's flush state advances to "already tried" or stays stuck without escalation.

Reliable repro recipe (untested upstream, matches our gateway behavior):

# 1. Force a session past threshold (synthetic conversation or replay)
# 2. Kill the session's provider transport mid-flush
#    e.g. for Codex: SIGKILL the codex app-server child during the flush turn
# 3. Send a follow-up user message
#    → memoryFlush re-fires, fails the same way, channel sees no reply

Suggested fix

Add a per-session flush-failure escalation tracker in the gateway:

state per session:
  memoryFlushLastAttempt: timestamp
  memoryFlushLastTokenCount: number
  memoryFlushFailureCount: number (since last successful flush)

on memoryFlush check:
  if tokenCount > threshold:
    if last attempt failed AND (now - lastAttempt) >= ESCALATION_WINDOW (e.g. 5min):
      → escalate: force-truncate transcript independent of assistant
                  (write a minimal summary marker, keep last N turns,
                   drop the rest — same effect as `sessions cleanup --fix`)
      → emit lifecycle event: memory.flush.escalated
                              (sessionId, tokenCountBefore, tokenCountAfter, reason)
    else:
      run normal flush attempt
      on failure: increment failureCount, record lastAttempt
      emit lifecycle event: memory.flush.failed
                            (sessionId, tokenCount, providerError, attempt#)

The escalation path needs to be provider-independent — it should be able to truncate the JSONL transcript and emit a placeholder "[session reset: prior context dropped due to repeated flush failures]" entry without calling out to any LLM. This is the safety valve: a degraded session that loses context is far better than a silent permanent wedge.

Configurability:

{
  \"agents\": {
    \"defaults\": {
      \"memoryFlush\": {
        \"escalation\": {
          \"enabled\": true,
          \"thresholdMinutes\": 5,
          \"maxFailures\": 2,
          \"action\": \"force-truncate\"  // or \"alert-only\" for observability-first
        }
      }
    }
  }
}

Related upstream issue #30452 (flush-then-reset mode) covers the configurable-mode angle; this issue is the failure-mode angle — even with the default mode, you need an escape hatch when the assistant required by flush is the failing dependency.

Observability

Independent of the fix, two lifecycle events would have caught this in monitoring:

• memory.flush.failed (currently the only signal is the underlying provider error, not session-attributed)
• memory.flush.escalated (would distinguish "transcript force-truncated" from "transcript still bloated")

Workaround

1. openclaw gateway restart — clears in-memory stalled_agent_run. Does NOT clear on-disk bloat.
2. Manually truncate the session JSONL at ~/.openclaw/workspace-<agent>/sessions/<sessionId>.jsonl (or use a script that does the equivalent).
3. Verify the next memoryFlush check tokenCount reads below threshold before declaring recovery.

Skipping step 2 → next inbound message re-wedges within seconds.

Environment

• OpenClaw 2026.5.12 (fork: rhclaw/openclaw-server)
• Linux 6.8.0-111-generic, Node v22.x
• Provider on the affected session: openai-codex/gpt-5.5 (Codex OAuth via ChatGPT subscription)
• memoryFlush.softThresholdTokens configured such that flush fires at ~130k tokenCount; affected session at 163k on-disk
• 8GB context window, default compaction mode (not safeguard)

Related

• [Feature]: Compaction: configurable model override + flush-then-reset mode #30452 — Compaction: configurable model override + flush-then-reset mode (adjacent, config-level)
• memoryFlush never triggers when compactionCount and memoryFlushCompactionCount are both 0 #47143 — memoryFlush never triggers when counters both 0 (adjacent, different failure mode)
• Bug: in-turn auto-compaction can bypass memoryFlush because memoryFlush only runs pre-turn #62420 — in-turn auto-compaction bypasses memoryFlush (adjacent, different bypass)
• Memory compaction blocks main processing lane, causing unresponsive bot for 10+ minutes #53008 — Compaction blocks main processing lane (adjacent, perf not correctness)
• Group session lane wedges indefinitely: stuck-recovery skips with active_reply_work false positive #85639 — Group session lane wedges with active_reply_work false positive (adjacent, different wedge class)

Downstream tracking: rhclaw/openclaw-server#291