fix(codex): ignore duplicate token_count totals - #824

Summary

Codex session logs can emit duplicate token_count events where the cumulative total_token_usage does not advance. The loader previously trusted last_token_usage unconditionally, which causes duplicated deltas and inflates all token totals (often close to x2). This change skips last_token_usage when totals are unchanged, and only derives deltas from totals when last is missing.

Why this happens

Codex logs contain both:

• info.last_token_usage: per-event delta
• info.total_token_usage: cumulative totals

In practice, some sessions emit the same token_count payload twice without advancing the cumulative totals. When we sum last_token_usage for every event, those duplicates are counted again even though total_token_usage shows no progress.

Example from real logs (simplified)

Event A (first emission):

• last_token_usage: input=1000, cached=250, output=200, total=1200
• total_token_usage: input=1000, cached=250, output=200, total=1200

Event B (duplicate emission, totals unchanged):

• last_token_usage: input=1000, cached=250, output=200, total=1200
• total_token_usage: input=1000, cached=250, output=200, total=1200

Old behavior:

• Adds Event A delta + Event B delta = 2400 total tokens (double counted)

New behavior:

• Detects that totals did not change for Event B
• Skips Event B and keeps 1200 total tokens (correct)

How the fix works

• Track the previous total_token_usage per session
• If the current total_token_usage equals the previous total, treat the
  event as a duplicate and ignore last_token_usage
• If last_token_usage is missing but totals exist, derive a delta from the
  totals (existing behavior)

What changed

• Added a small helper to compare totals for equality
• Updated the event processing logic to skip duplicated totals
• Added an in-source test to reproduce the duplicate pattern

Summary by CodeRabbit

• Bug Fixes
  • Improved detection and handling of duplicate token usage events to prevent redundant processing and ensure accurate token count tracking.

_{✏️ Tip: You can customize this high-level summary in your review settings.}