Skip to content

feat(trajectory): allow OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES override for per-event byte cap#80681

Open
wAngByg wants to merge 3 commits into
openclaw:mainfrom
wAngByg:fix/trajectory-event-max-bytes-env
Open

feat(trajectory): allow OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES override for per-event byte cap#80681
wAngByg wants to merge 3 commits into
openclaw:mainfrom
wAngByg:fix/trajectory-event-max-bytes-env

Conversation

@wAngByg

@wAngByg wAngByg commented May 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The trajectory per-event byte cap is currently fixed at 256 KiB, so large but valid events are replaced by a truncation sentinel.
  • This PR adds a startup-scoped OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES override for both trajectory writers while preserving the existing 256 KiB default.
  • The override accepts byte counts and human-friendly suffixes such as 512kb, 1mb, and 2gb.
  • Invalid, empty, zero, or negative values fall back to the default.
  • The file-level trajectory cap still applies, so this does not make trajectory storage unbounded.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: Oversized trajectory runtime events are currently truncated at the hard-coded 256 KiB per-event cap. The after-fix behavior should keep the default cap unchanged, but allow an operator to raise the cap at recorder creation time through OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES.
  • Real environment tested: Windows local checkout, Node.js v24.15.0, exact PR head 8f7d4e1a6b95c0561a83b0b71e0b6005d449ee87, using the real createTrajectoryRuntimeRecorder path and a synthetic oversized payload shaped to stay under the inner sanitizer limits while exceeding the outer event-byte cap.
  • Exact steps or command run after this patch:
    1. Created a real trajectory recorder with OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES unset and recorded a payload with 60 fields of 5000 chars each.
    2. Created a second real trajectory recorder with the identical payload and OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES=1mb.
    3. Compared the emitted trajectory JSONL line size and truncation metadata.
    4. Ran focused trajectory tests, the extension trajectory test file, the previously failing gateway CI files, touched-file lint, core and extension typechecks, and diff whitespace checks.
  • Evidence after fix:
head=8f7d4e1a6b95c0561a83b0b71e0b6005d449ee87
environment=win32 x64, Node v24.15.0
PHASE default-cap
env (unset)
trajectory file emitted for default-cap run
bytes 558
truncated true
originalBytes 301299
limitBytes 262144
reason trajectory-event-size-limit
PHASE override-1mb
env 1mb
trajectory file emitted for override-1mb run
bytes 301302
truncated false
field_count 60
node scripts/run-vitest.mjs src/trajectory/paths.test.ts src/trajectory/runtime.test.ts

Test Files  2 passed
Tests       20 passed, 2 skipped

Supplemental extension trajectory validation:

Test Files  1 passed (1)
Tests       10 passed, 1 skipped

Supplemental validation for the previously failing gateway CI files:

node scripts/run-vitest.mjs run --config test/vitest/vitest.gateway-server.config.ts src/gateway/server-startup-config.secrets.test.ts src/gateway/server-import-boundary.test.ts --reporter=dot --pool=forks --maxWorkers=1

Test Files  2 passed (2)
Tests       19 passed (19)

Additional local checks at the same PR head:

node scripts/run-oxlint.mjs <touched trajectory/docs files> -> passed
node scripts/run-tsgo.mjs -p tsconfig.core.json -> passed
node scripts/run-tsgo.mjs -p tsconfig.extensions.json -> passed
git diff --check origin/main...HEAD -> passed
  • Observed result after fix: With the default cap, the event is truncated to a 558-byte sentinel with originalBytes=301299, limitBytes=262144, and reason=trajectory-event-size-limit. With OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES=1mb, the same payload is preserved as a 301302-byte line with truncated=false and all 60 fields present.
  • What was not tested: Filesystem-level I/O performance at very large per-event limits. The file-level cap still applies and prevents unbounded trajectory growth.

Security Impact (required)

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command execution surface changed? No
  • Data access scope changed? No
  • New operator-controlled config/env surface? Yes - OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES controls only the per-event trajectory recording cap at recorder creation time.
  • Risk + mitigation: A very high cap can increase per-event trajectory disk usage. The existing file-level trajectory cap remains in force, invalid values fall back to the 256 KiB default, and the override is resolved once from the recorder environment rather than reread from ambient process state per event.

Repro + Verification

Environment

  • OS: Windows
  • Runtime: Node.js v24.15.0
  • PR head: 8f7d4e1a6b95c0561a83b0b71e0b6005d449ee87

Steps

  1. Record the same oversized trajectory event with the override unset.
  2. Record it again with OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES=1mb.
  3. Inspect the resulting trajectory JSONL lines.

Expected

  • Default cap: event is replaced by a truncation sentinel with limitBytes=262144.
  • 1 MiB override: event is preserved in full and is not marked truncated.

Actual

  • Default cap emitted a 558-byte truncation sentinel.
  • 1 MiB override emitted the full 301302-byte event with all 60 fields present.

Review scope clarification

The current head resolves the cap from the recorder's explicit environment at recorder creation time:

const eventMaxBytes = resolveTrajectoryRuntimeEventMaxBytes(env);

The truncate helper receives that resolved eventMaxBytes value instead of rereading process-wide environment state. This keeps the override startup-scoped and aligned with the recorder's already resolved environment.

@openclaw-barnacle openclaw-barnacle Bot added size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 11, 2026
@clawsweeper

clawsweeper Bot commented May 11, 2026

Copy link
Copy Markdown
Contributor

Codex review: found issues before merge. Reviewed July 3, 2026, 11:45 PM ET / 03:45 UTC.

Summary
The PR adds a documented startup env var, OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES, to override per-event trajectory byte caps in core and bundled Codex recorders, with parser tests and trajectory docs.

PR surface: Source +127, Tests +187, Docs +5. Total +319 across 7 files.

Reproducibility: not applicable. this is a feature/config PR rather than a bug report. Current main still uses a fixed 256 KiB per-event cap, while the PR body supplies copied live recorder output for the proposed override.

Review metrics: 1 noteworthy metric.

  • Config/env surface change: 1 added. The PR documents and reads a new startup env var that changes diagnostic retention, which repository policy treats as compatibility-sensitive before merge.

Stored data model
Persistent data-model change detected: serialized state: src/trajectory/runtime.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Get maintainer acceptance for the new trajectory env contract.
  • Rebase over current main and preserve usage, promptCache, and droppedFields in both recorders.

Risk before merge

  • [P1] The PR adds a new documented operator env var that can increase trajectory retention and disk usage, so maintainers need to accept the config surface before merge.
  • [P1] The current branch is conflicting and predates merged preserved-field truncation semantics; an unsafe rebase could drop usage, promptCache, and droppedFields from oversized trajectory events.

Maintainer options:

  1. Sponsor and rebase safely
    If maintainers want this env contract, rebase onto current main and preserve usage, promptCache, and droppedFields in both trajectory recorders before merge.
  2. Keep the fixed cap
    If maintainers do not want another trajectory env var, close the PR and keep the current fixed cap plus usage-preserving truncation behavior.

Next step before merge

  • [P2] The remaining blockers are maintainer sponsorship of the new env surface and a conflict-aware rebase, not a safe autonomous repair lane.

Security
Cleared: No dependency, workflow, permission, network, command-execution, credential, or supply-chain change was found; the remaining concern is product/storage impact from larger diagnostic retention.

Review findings

  • [P2] Preserve preserved-field truncation when threading the cap — src/trajectory/runtime.ts:143-150
  • [P2] Keep Codex truncation metadata on rebase — extensions/codex/src/app-server/trajectory.ts:138-145
Review details

Best possible solution:

If maintainers sponsor the env contract, rebase onto current main and parameterize the existing preserved-field truncation helpers; otherwise keep the fixed 256 KiB cap.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a feature/config PR rather than a bug report. Current main still uses a fixed 256 KiB per-event cap, while the PR body supplies copied live recorder output for the proposed override.

Is this the best way to solve the issue?

No, not as-is. The override shape is plausible, but the best solution requires maintainer acceptance of the env contract and a rebase that preserves current usage-aware truncation semantics.

Full review comments:

  • [P2] Preserve preserved-field truncation when threading the cap — src/trajectory/runtime.ts:143-150
    Current main preserves usage, promptCache, and droppedFields when an oversized trajectory event is replaced by a truncation stub. This branch parameterizes the older bare-stub helper, so the rebase must thread limitBytes through the current preserved-field builder instead of dropping accounting metadata.
    Confidence: 0.9
  • [P2] Keep Codex truncation metadata on rebase — extensions/codex/src/app-server/trajectory.ts:138-145
    The bundled Codex recorder now has the same preserved-field truncation behavior for oversized model.completed rows. This helper currently builds only a bare truncation stub with the new cap, so a conflict resolution that keeps this shape would regress Codex trajectory accounting.
    Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9d68f877ac3e.

Label changes

Label justifications:

  • P2: This is a bounded trajectory diagnostics feature with compatibility and rebase blockers, not an urgent production regression.
  • merge-risk: 🚨 compatibility: The PR adds a documented operator env var and must be rebased without regressing current trajectory truncation metadata semantics.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body provides copied live Windows recorder output showing default truncation versus a 1 MiB override, plus focused trajectory validation at the stated PR head.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides copied live Windows recorder output showing default truncation versus a 1 MiB override, plus focused trajectory validation at the stated PR head.
Evidence reviewed

PR surface:

Source +127, Tests +187, Docs +5. Total +319 across 7 files.

View PR surface stats
Area Files Added Removed Net
Source 3 144 17 +127
Tests 3 212 25 +187
Docs 1 6 1 +5
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 7 362 43 +319

What I checked:

  • Repository policy applied: Root AGENTS.md was read fully, plus scoped docs and extensions guides; the review applied the repo's high bar for new env/config surfaces and bundled plugin boundary guidance. (AGENTS.md:1, 9d68f877ac3e)
  • Current main still has a fixed cap: Current main exports a fixed 256 KiB runtime event cap and has no OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES resolver. (src/trajectory/paths.ts:11, 9d68f877ac3e)
  • Current main preserves core truncation accounting: The current core truncation helper preserves selected usage and promptCache fields and emits droppedFields when replacing oversized events. (src/trajectory/runtime.ts:124, 9d68f877ac3e)
  • Current core tests cover the invariant: Current main has regression tests asserting oversized runtime events preserve usage and promptCache, drop large snapshots, and stay under the byte limit. (src/trajectory/runtime.test.ts:136, 9d68f877ac3e)
  • Current Codex recorder shares the invariant: The bundled Codex recorder has the same preserved-field and dropped-field behavior for oversized trajectory rows. (extensions/codex/src/app-server/trajectory.ts:80, 9d68f877ac3e)
  • Current Codex tests cover usage preservation: Current Codex trajectory tests assert recordCodexTrajectoryCompletion keeps usage in truncated model completion events. (extensions/codex/src/app-server/trajectory.test.ts:260, 9d68f877ac3e)

Likely related people:

  • scoootscooob: Authored the commit that added trajectory bundle export, default-on runtime capture, and the core/Codex trajectory files touched by this PR. (role: introduced current trajectory subsystem; confidence: high; commits: a3d9c53db299; files: src/trajectory/runtime.ts, extensions/codex/src/app-server/trajectory.ts, docs/tools/trajectory.md)
  • lin-hongkuan: Authored the merged usage-preservation trajectory changes that current main now requires this PR to preserve. (role: recent adjacent contributor; confidence: high; commits: bf2a8ecfdb89, 67118d5ab912, 898ca9741cba; files: src/trajectory/runtime.ts, src/trajectory/runtime.test.ts, extensions/codex/src/app-server/trajectory.ts)
  • joshavant: Merged the usage-preservation PR and is the committer on the commits that define current main's truncation-accounting behavior. (role: merger and recent area signal; confidence: medium; commits: bf2a8ecfdb89, 67118d5ab912, 898ca9741cba; files: src/trajectory/runtime.ts, extensions/codex/src/app-server/trajectory.ts)
  • obviyus: Authored multiple May 2026 fixes to trajectory runtime ordering, flush, and window semantics around the same core runtime path. (role: recent area contributor; confidence: medium; commits: 82c0a6077724, 6361c46fe294, d879340ed999; files: src/trajectory/runtime.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation extensions: codex size: M proof: supplied External PR includes structured after-fix real behavior proof. and removed size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 12, 2026
@wAngByg
wAngByg force-pushed the fix/trajectory-event-max-bytes-env branch 2 times, most recently from 51ea14e to 66e75d5 Compare May 14, 2026 11:38
@wAngByg
wAngByg force-pushed the fix/trajectory-event-max-bytes-env branch from 66e75d5 to 9d13c9f Compare May 14, 2026 12:41
@clawsweeper

clawsweeper Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 17, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. P2 Normal backlog priority with limited blast radius. labels May 17, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 17, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 17, 2026
@RomneyDa

Copy link
Copy Markdown
Member

Heads up: this PR needs to be updated against current main before the new required Dependency Guard check can pass.

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 29, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels May 29, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 29, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 31, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 31, 2026
@clawsweeper

clawsweeper Bot commented May 31, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 31, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 31, 2026
@wAngByg
wAngByg force-pushed the fix/trajectory-event-max-bytes-env branch from f6ae71e to 8f7d4e1 Compare June 1, 2026 05:53
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 1, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 1, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@wAngByg
wAngByg force-pushed the fix/trajectory-event-max-bytes-env branch from b314f3c to 01c46cc Compare June 10, 2026 15:12
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@wAngByg
wAngByg force-pushed the fix/trajectory-event-max-bytes-env branch from 01c46cc to 635b348 Compare June 10, 2026 15:26
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 10, 2026
@wAngByg
wAngByg force-pushed the fix/trajectory-event-max-bytes-env branch from 635b348 to de304e5 Compare June 11, 2026 01:42
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 11, 2026
@wAngByg

wAngByg commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Rebased to current main at 86f237ec3b60e3ec5b3facb8329dec2d9d5fc9f9; no code changes beyond replaying the existing trajectory-cap commits. Real behavior proof is green on the current head. The remaining red checks-node-agentic-* shards are in unrelated src/agents/* tests and are also reproducing on #79910 after rebase.

@clawsweeper

clawsweeper Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@clawsweeper

clawsweeper Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: feat(trajectory): allow OPENCLAW_TRAJECTORY_RUNTIME_EVENT_MAX_BYTES override for per-event byte cap This is item 1/1 in the current shard. Shard 10/22.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation extensions: codex merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: M stale Marked as stale due to inactivity status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants