Skip to content

Codex context-engine projection lacks exact pre-turn token accounting #80765

Description

@100yenadmin

Summary (edit to make not stale)

Follow-up to #80760 and #80761.

#80761 makes the native Codex context-engine projection budget-aware and maps the rendered-context cap through the existing openclaw.json compaction reserve fields:

  • agents.defaults.compaction.reserveTokens
  • agents.defaults.compaction.reserveTokensFloor

That fixes the main send-side bug where LCM/context-engine could assemble a large frontier, but native Codex still received only a fixed 24k-character projected context block.

There is still a smaller accounting gap: before turn/start, the Codex app-server projection currently sizes the rendered context using contextTokenBudget, reserve config, and OpenClaw's normal approximate 4 chars/token text heuristic. Exact provider/runtime token usage is only available after the turn completes.

Why this matters

The new projection cap is correctly proportional, but it is still estimate-based. That can leave a slight mismatch between:

  • what OpenClaw expects to send before the turn,
  • what Codex actually tokenizes for the rendered turn/start input,
  • what LCM/status can later report from observed post-turn usage.

This is usually small for ordinary Latin text, but it can drift for high-token-density content, CJK/multilingual content, unusual formatting, large quoted context blocks, or future Codex tokenizer/model changes.

Current behavior

The active Codex context-engine path now effectively does:

  1. context engine assembles messages for the current runtime budget
  2. OpenClaw projects those messages into a quoted Codex prompt string
  3. projection uses contextTokenBudget - reserveTokens and a 4 chars/token estimate to choose a char cap
  4. Codex receives turn/start
  5. exact-ish usage/observed prompt tokens are only known after the turn

So OpenClaw can protect against gross overflow, but cannot pre-turn prove the exact rendered prompt token count.

Expected direction

Add a native Codex pre-turn accounting path or telemetry seam so projection/status can report exact or authoritative token accounting where available.

Possible implementation shapes:

  • expose a Codex app-server/tokenizer API for the exact turn/start input token count before send
  • add a provider/model tokenizer utility for the rendered Codex prompt string and developer instructions
  • return projection stats from the Codex app-server route, including projectedPromptChars, estimated prompt tokens, reserve tokens, cap chars, and exact input tokens when available
  • pass those stats into context-engine afterTurn/status telemetry so LCM can distinguish selected frontier, projected prompt, and provider-observed usage

Acceptance criteria

  • Native Codex context-engine projection can report pre-turn exact token count when the runtime/tokenizer surface supports it, or explicitly mark it as estimated when not available.
  • /status or equivalent diagnostics can distinguish:
    • LCM/frontier tokens selected by the context engine
    • rendered Codex projection chars/tokens before send
    • provider-observed prompt/input tokens after the turn
  • Tests cover the estimate-vs-exact marker and ensure configured reserve fields remain honored.

Notes

This is intentionally separate from #80761. The PR fixes the observed fixed-cap truncation bug. This issue tracks the remaining observability/accounting gap so future debugging does not confuse LCM frontier size with Codex rendered prompt usage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions