Skip to content

Surface frontier-vs-runtime projection mismatch for native Codex hosts #658

Description

@100yenadmin

Summary

LCM currently preserves and assembles the active frontier correctly, but when the host runtime is native Codex, OpenClaw can project that frontier into a much smaller model-visible prompt. LCM does not surface this as a distinct diagnostic, so users see /status drop sharply and reasonably suspect LCM lost its session key, overcompacted, or failed to bootstrap.

This issue is for the LCM-side diagnostic/provenance gap, not the upstream OpenClaw projection cap itself.

Confirmed Evidence

Local incident after an OpenClaw beta update and route switch to native codex / openai-codex:

  • Active LCM conversation remained intact:
    • conversation_id=1872
    • session_key=agent:main:main
    • active conversation still present
    • active session file fully processed
    • DB quick check passed
  • LCM frontier was healthy:
    • 27811 messages
    • 14241378 raw message tokens
    • 695 summaries
    • 247 context items in the frontier
    • about 186764 frontier tokens
  • LCM assemble selected a large full-fit context:
    • contextItems=215
    • selectionMode=full-fit
    • estimatedTokens=179134
    • rawMessageCount=93
    • summaryCount=122
  • Native Codex runtime/status then showed only about 76k/258k, about 30%.

There was no new LCM compaction checkpoint explaining the drop, and the fresh turn had:

  • shouldCompact=false
  • reason: below-context-threshold-floor

The actual model-visible drop is explained by OpenClaw's native Codex context-engine projection cap, but LCM's user-facing diagnostics did not make that boundary visible.

Why This Matters

LCM has at least three separate quantities that users/operators need to distinguish:

  1. Durable stored history in SQLite.
  2. LCM-selected frontier returned by assemble().
  3. Host/runtime-projected prompt that actually reaches the model.

In this incident, (1) and (2) were healthy. Only (3) was much smaller. Without explicit LCM diagnostics, this looks exactly like:

  • DB loss
  • session key mismatch
  • overcompaction
  • bootstrap failure
  • bad summary pruning

That sends operators down the wrong recovery path.

Related Host Boundary

OpenClaw native Codex currently projects context-engine output through:

  • extensions/codex/src/app-server/context-engine-projection.ts

That projection has a fixed rendered-context cap:

  • MAX_RENDERED_CONTEXT_CHARS = 24_000
  • MAX_TEXT_PART_CHARS = 6_000

LCM cannot fully fix the model-visible context from inside the plugin, but it can detect and report the mismatch.

Expected LCM Behavior

LCM diagnostics should clearly report:

  • active conversation id/session key/session file
  • last bootstrap offset vs session file size
  • selected frontier counts/tokens from the latest assemble()
  • latest provider/runtime prompt usage from afterTurn
  • whether selected frontier tokens are much larger than provider-visible prompt tokens
  • a warning when the likely cause is host projection/truncation rather than LCM compaction

Example diagnostic:

LCM frontier: 186k tokens selected, full-fit.
Runtime prompt usage: 76k tokens.
Warning: selected frontier exceeds provider-visible prompt by 110k tokens.
Likely host projection/truncation boundary; LCM storage and frontier are intact.

Actual Behavior

LCM logs contain enough raw evidence to prove the frontier is healthy, but the user-facing status/debug surface does not present the frontier-vs-runtime-projection distinction. The operator has to manually inspect SQLite, assemble logs, trajectory, and OpenClaw source to prove no LCM loss occurred.

Fix Direction

Recommended LCM-side fix:

  1. Persist latest assembly telemetry per conversation/session key:
    • contextItems
    • selectedRawMessageCount
    • selectedSummaryCount
    • estimatedTokens
    • selectionMode
    • messagesHash
    • assembledAt
  2. Compare latest assembly telemetry with latest provider/runtime prompt usage.
  3. Add a /lossless doctor or /lossless status warning when:
    • assembly was full-fit
    • selected/estimated tokens are far above provider-visible prompt tokens
    • no compaction event occurred
  4. Include session-key lineage/relink status in the same report when available.
  5. Document that /status is host runtime prompt usage, while /lossless should expose LCM frontier health.

This would have made the incident immediately diagnosable as a Codex host projection issue rather than apparent LCM memory loss.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpriority:P2Important bug, compatibility, performance, or reliability issue

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions