Repository navigation
Commit 89d6a03
Codex: dedupe fork-replayed history by content, not position (#37)
* Codex: dedupe fork-replayed history by content, not position
Fork/spawn child rollouts replay the parent session's history with
every copied line's timestamp rewritten to the fork instant, so the
positional dedup key (session, timestamp, line index) counted replayed
token_count events again — inflating token totals, LIMITS samples, and
MODES effort mix (issue #36).
- Usage events: keyed on (session, last_token_usage vector,
total_token_usage vector). The cumulative is non-decreasing between
compactions, so the pair identifies one turn's consumption across
every file it appears in; re-emissions with an unchanged cumulative
collapse too. Logs without the cumulative fall back to the
positional key.
- Rate-limit samples: same usage fingerprint plus the snapshot's own
fields (used_percent, window_minutes, resets_at) so a verbatim
replay collapses while a genuine re-notification survives.
- Effort events: keyed by turn_context.payload.turn_id, which replays
copy verbatim; positional fallback for logs predating turn_id.
- merge_into: keyed duplicates keep the earliest observed timestamp —
scan order (archived_sessions sorts before sessions wholesale) can't
let a replay shift an event's day attribution. The five dedup loops
fold into a shared dedupe_into helper.
- CACHE_VERSION 9 -> 10: cached FileEvents carry the old positional
keys, so parsing semantics changed even though the layout did not.
Known remainder (follow-up): session touches and task durations from
replayed lines are still counted.
Co-Authored-By: Claude Fable 5 <[email protected]>
* Codex: harden semantic keys against malformed payloads
Address second-model review of the GH-36 fix:
- Gate the semantic-key path on a fingerprintable usage object
(total_tokens parses as an integer) — total_token_usage null/{}
would fingerprint as all-zero and falsely collide distinct turns.
Same gate on the rate-limit key's last_token_usage.
- Clamp used_percent before keying so an out-of-range raw value can't
mint a second key for the same stored sample value.
- Soften the CHANGELOG attribution claim: earliest-timestamp-wins holds
when both original and replay are scanned; an out-of-window original
leaves the replay's fork-instant timestamp (known remainder).
- Pin the review-raised invariants in tests: malformed-cumulative
fallback, rate-limit re-notification survival, cross-turn re-emission
collapse (why turn_id is not in the usage key), same-timestamp
distinct turn_ids, cache_write discrimination, and Claude fragment
merge (larger volume + earliest timestamp + merged metadata).
Co-Authored-By: Claude Fable 5 <[email protected]>
* Codex: skip the fork replay burst structurally
Layer a parse-time skip under the content-based keys (GH-36) — the
approach ccusage took for the same bug (ccusage/ccusage#950) — with a
broader trigger: a second session_meta marks a fork-shaped rollout
(catches Desktop thread branches that carry no thread_spawn marker),
and event lines stamped in the same second as that meta are the replay
burst.
- Replayed history now contributes nothing even when the parent rollout
is outside the scan (deleted or past the mtime window), where
cross-file dedup has no original to collapse into and previously the
replay was counted once at the fork instant.
- Durations (unkeyed) and session touches from replayed lines stop
counting, closing two known remainders.
- Content-based keys stay as the safety net for burst lines that leak
past the fork second, whole-file copies, and re-emissions.
Real-data check: same-instant A/B with and without the skip yields
byte-identical Codex totals on an all-parents-in-scan corpus — the skip
only changes outcomes where originals are missing from the scan.
Co-Authored-By: Claude Fable 5 <[email protected]>
---------
Co-authored-by: Claude Fable 5 <[email protected]>1 parent 86769ac commit 89d6a03
3 files changed
Lines changed: 695 additions & 104 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
8 | 27 | | |
9 | 28 | | |
10 | 29 | | |
| |||
262 | 281 | | |
263 | 282 | | |
264 | 283 | | |
| 284 | + | |
| 285 | + | |
265 | 286 | | |
266 | 287 | | |
267 | 288 | | |
| |||
0 commit comments