fix(codex): read Zstandard-compressed rollouts - #1890

The generated Codex fixture is mostly zstd-compressed rollouts, some in
`archived_sessions/`. Releases before #1890 skip `.jsonl.zst` files
(and before #1230 skip archived sessions), so a history run across
that change compared an old ref reading a fraction of the input with a
new ref reading all of it, and reported the difference as a CPU and
RSS regression.

The generator gains `--codex-layout legacy`, which writes the same
rollouts uncompressed under `sessions/`. bench-history keeps the
current layout when every ref is a commit containing #1890, and
otherwise generates the legacy layout and says so at the top of the
report; binary paths count as unknown. With a custom Codex fixture
that holds `.zst` files and a ref that cannot read them, and for
`--real` runs across the change, the report leads with a warning
instead. The PR comparison always builds refs that read the current
layout, so CI keeps compressed coverage.

Reported by CodeRabbit on #2061.

The compatibility check only looked for `.jsonl.zst` files, but
releases before #1230 also skip `archived_sessions/`, so a custom
fixture with archived plain rollouts got no warning and a history run
across #1230 compared different input.

Replace the single zstd commit with a table of Codex input classes taken
from the Codex loader's history (`paths.rs`, `reader.rs`): archived
sessions (a9b12c4, #1230, v20.0.7) and zstd-compressed rollouts
(44a3aa8, #1890, v20.0.33). Each ref is checked per class by git
ancestry; the generated fixture switches to the legacy layout when any
ref skips any class, and a custom fixture or `--real` run warns per
class with the refs that skip it. Claude discovery has scanned
`projects/` recursively throughout v20 and the date-window skips (#1665,
#1794) do not apply to unbounded reports, so they are not classes.
Probing each ref's source for the reader code was rejected: the same
strings appear in tests and comments.

Reported by Cubic on #2061.

#2061)

* build(perf): shape the large fixture like current Claude Code and Codex logs

The CI performance comparison missed the double parse that #1719 added
to every Claude line containing `null`, because the generated fixture
never contained `null`: each line was a minimal assistant usage record
padded with a single text block of `x` characters. Real Claude Code
lines now carry `stop_sequence`, `container`, `stop_details` and other
null-valued fields, nested `usage.iterations` and cache-creation TTL
splits, and most lines are not usage records at all. On the old fixture
v20.0.20 -> v20.0.21 moved CPU time by +8%; on real-shaped lines it is
+55%.

The generator now writes sessions whose line types, per-type byte
share, and field shapes follow aggregate statistics taken from real
logs (structure only; every value is synthesized from a seeded PRNG,
so the output is deterministic):

- Claude: user/tool-result, attachment, api-request(-blob) and metadata
  lines around assistant responses written one line per content block
  with repeated usage, null-valued fields, `usage.iterations` including
  occasional advisor iterations, 5m/1h cache-creation split, escaped
  and non-ASCII text, 48 projects, sidechain `subagents/agent-*.jsonl`
  files that replay parent message IDs, and legacy files with
  `progress` lines wrapping AgentProgress assistant messages.
- Codex: dated `sessions/YYYY/MM/DD/rollout-*.jsonl` files with
  session_meta, turn_context, reasoning, tool call/output and
  item_completed lines, plus token_count (with null-heavy rate limits)
  and token_usage_record per response. As in real Codex homes, rollouts
  older than two weeks are zstd-compressed (`.jsonl.zst`) and some sit
  in `archived_sessions/`; on real data about 90% of Codex bytes are
  now read through zstd, a path the old fixture never touched.

The file-size distribution (of uncompressed bytes) and the CLI are
unchanged.

* build(perf): build base and PR from source and judge them by CPU time

The PR performance comment could not show a regression or a real
improvement:

- Unequal artifacts. Base ran the published `ccusage` npm package (Node
  launcher spawning the native binary) while the Rust leg ran the bare
  native binary, so every PR showed a ~2.6x "win" on the committed
  fixtures that was only launcher startup.
- One sample. The large fixture ran with 0 warmups and 1 run, and the
  base and PR runs were not interleaved, so runner noise dominated.
- The committed fixtures are a few KiB and measure process startup.
- No verdict. Nothing said whether a number was a regression.

The job now builds both binaries with `nix build .#ccusage-static`, the
derivation the Linux x64 native package ships, on the same runner: the
PR head from the checkout and the base from
`github.event.pull_request.base.sha`, the same base commit as before.
The walk back over pricing-only commits only existed to find a commit
with a pkg.pr.new package, so it goes. The job reuses the
build-native-packages sticky disk for dependency artifacts and no
longer waits for pkg.pr.new, so the JS launcher leg, the
committed-fixture section, and the package diagnostics are gone too.

The harness runs `claude daily`, `claude session`, `codex daily` and
the unified `daily` over the realistic generated fixture in interleaved
rounds (each round one hyperfine invocation with both binaries, order
rotated), recording wall time, CPU time, peak RSS and, where the
platform exposes it, instructions. It reports medians with min-max
spread and per-round PR/base ratios. A command is flagged when the
median CPU-time or peak-RSS ratio moves past 10% and at least four in
five rounds agree; the verdict line and a GitHub warning annotation say
so. Wall time is shown but not judged: it moves with runner load, while
CPU time paired within a round does not.

The check warns rather than fails: a failing check needs a calibrated
false-positive rate across many runs on these runners and an escape
hatch for intentional trade-offs, and neither exists yet.

* build(perf): add `just bench-history` to compare refs

Finding which release introduced a slowdown meant building each tag by
hand and timing it ad hoc. `just bench-history <ref|binary>...` builds
each ref in release mode into `rust/target/bench-history` (one shared
cargo target dir, binaries cached by commit), generates the realistic
fixture once per generator revision and size, and runs every report
command over all refs in the same interleaved rounds as the PR
comparison. Each ref is judged against the previous one with the same
CPU-time/peak-RSS rule, so a regression shows up at the step that
introduced it. `--real` adds a section over the logs the current
environment points at.

It reuses the PR harness modules instead of adding a parallel tool, and
the profile skill now points at it.

* fix(perf): print the regression annotation on stdout

GitHub Actions reads workflow commands only from stdout, but the
`::warning` line was printed while `*out*` was bound to `*err*`, so a
flagged regression never produced an annotation. Print it on stdout and
pass the GitHub Actions check in, so a test can assert the stream.

Reported by Pullfrog on #2061.

* fix(perf): reject bench-history runs below the minimum

`parse-history-cli` validated a copy with `:runs` raised to 3 but
returned the original options, so `--runs 2` passed validation and then
measured two rounds, too few for the four-in-five agreement rule.
Validate the options that are returned, so fewer than three rounds is
rejected as in the PR comparison, and say so in both `--runs` helps.

Reported by Pullfrog on #2061.

* fix(perf): run fixture benchmarks in an empty environment

The fixture runs pointed Claude and Codex at the fixture and moved HOME
and XDG_* to an empty directory, but every other agent has its own
data-dir variable (AMP_DATA_DIR, OPENCODE_DATA_DIR, PI_AGENT_DIR,
GEMINI_DATA_DIR and a dozen more), and a value set in the caller's shell
still pointed the unified `daily` report at real logs. Listing every
variable would rot as agents are added, so the fixture commands now run
under `env -i` with only the variables the run needs; with XDG_* unset
the agents fall back to the empty HOME. The local-logs mode keeps the
caller's environment on purpose.

Reported by Cubic on #2061.

* fix(perf): make bench-history safe to point at binaries and run twice

Four ways `just bench-history` could measure the wrong thing or fail:

- Binary-path variants were labelled by basename, so two builds at
  `a/ccusage` and `b/ccusage` collided in hyperfine's results. Label
  them by the argument as given, and reject a ref or path passed twice.
- `--fixture-dir` alone silently reused the Claude logs as the Codex
  fixture. Require both directories or neither.
- The cached binary was copied without attributes; copy them so the
  executable bit is kept on every platform.
- Every run checked refs out into the shared `<cache>/src` worktree and
  removed it on exit, so a second run sharing the cache deleted the
  first run's checkout mid-build. Each run now uses its own temporary
  worktree; the shared cargo target is safe because cargo locks it.

The first three were reported by Cubic on #2061; the fourth showed up
while bisecting with two runs at once.

* fix(perf): report memory-only improvements and history instruction ratios

The overall verdict counted only CPU improvements, so a PR that cut
peak RSS read "no significant change" above a table saying "less
memory". Count CPU and RSS improvements alike and name what improved
per command. The history tables also dropped the instruction-count
ratio the PR comparison shows; add it as a column whenever hardware
counters are available.

Reported by Cubic on #2061.

* test(perf): restore duplicate-name and time-unit hyperfine cases

The hyperfine reader rewrite kept the missing-name and malformed-output
cases but lost the duplicate-name and non-second-unit ones, so a change
that accepted ambiguous or mis-scaled samples would pass. Restore both
against the new per-run reader.

Reported by Cubic on #2061.

* fix(perf): print the generated fixture's size on disk

With zstd-compressed Codex rollouts the generator's `Size` line showed
the uncompressed bytes, about four times what it wrote. Print the size
on disk from `Bun.write` and the uncompressed size separately.

Reported by Cubic on #2061.

* fix(ci): measure the PR merged onto its base commit

The job built the PR side from `pull_request.head.sha` and the base
from `pull_request.base.sha`. When a branch is behind its base, the two
binaries differ by the base's newer commits as well as the PR's own
change, and those commits show up as PR regressions or improvements.

Build the checked-out merge commit (`github.sha`) instead. It is the PR
applied on top of the base commit, so the diff measured is exactly the
PR. A step fails the job if the base is not an ancestor of the built
commit, so the report's "merged onto Base:" line always names what was
actually built. BASE_SHA/HEAD_SHA fall back to `merge_group.base_sha`
and `github.sha`, and the PR-comment step is limited to pull_request
events, so the job also works if a merge_group trigger is added.

Reported by CodeRabbit on #2061.

* fix(perf): cache built bench-history binaries atomically

`build-commit!` copied the release binary straight to the cache path
that later runs trust when it exists, so an interrupted copy left a
truncated binary that every later run reused. Copy to a `.partial`
sibling and rename it into place with an atomic move.

Reported by CodeRabbit on #2061.

* fix(perf): start fixture generation from an empty directory

A cached fixture without its `.complete` marker was regenerated into
the same directory, so files from an interrupted run could remain next
to the new output. Delete the directory first, so the marker always
describes exactly what the generator wrote.

Reported by CodeRabbit on #2061.

* fix(perf): order Codex usage records like real rollouts and add compactions

The generator wrote each response's `token_count` before its
`token_usage_record`. Real rollouts write the record first, and the
parser settles a pending record only when a later `token_count`
advances the cumulative total, so every synthetic record stayed in the
compaction pending map until the end of its file. That inflated peak
RSS in a way real logs never do: on the corrected fixture `codex daily`
peaks at 56 MiB on main instead of 69 MiB.

Write the record before the `token_count`, as Codex does, and add the
remote compactions real rollouts contain: about one turn in 30 ends
with a usage record whose usage is not in the cumulative total,
followed by a `compacted` marker naming it and carrying a large
replacement history. In real logs every marker names an earlier record.

Reported by Cubic on #2061.

* fix(ci): gate the perf job on preflight and keep its token read-only

The old perf job depended on the pkg.pr.new publish job and so, through
it, on `preflight` (secret scan and workflow lint); building from
source dropped that dependency, letting a PR that fails the scan run
the benchmark anyway. Depend on `preflight` directly.

The job also ran PR-built binaries, the fixture generator and the
harness with `pull-requests: write` and `issues: write`. Split it:
`ccusage-perf` builds and measures with a read-only token and uploads
the report, and `ccusage-perf-comment` downloads it and only runs the
comment upsert with the write token.

Reported by Cubic on #2061.

* fix(perf): lock bench-history builds and fixture generation

Runs sharing the cache could interleave writes: one build could replace
`target/release/ccusage` in the shared cargo target before another run
copied it into its commit's cache slot, and two runs could delete and
regenerate the same fixture under each other. Build-and-copy now runs
under a cache-wide file lock, and each fixture key under its own, both
re-checking the cache once they hold the lock. A per-commit target dir
would avoid the lock but rebuild every dependency for every ref.

The lock is a FileChannel lock released by closing the channel, and a
test runs three processes against it and checks their critical
sections never overlap.

Reported by Cubic on #2061.

* fix(ci): drop the perf job's unreachable merge_group fallbacks

The perf job only runs on `pull_request`, and #2062, which adds the
`merge_group` trigger, keeps the perf comment PR-only, so the
`merge_group.base_sha` and `github.sha` fallbacks and the comment
promising merge_group support could never apply. Remove them.

Reported by Cubic on #2061.

* fix(perf): give bench-history a Codex fixture every compared ref reads

The generated Codex fixture is mostly zstd-compressed rollouts, some in
`archived_sessions/`. Releases before #1890 skip `.jsonl.zst` files
(and before #1230 skip archived sessions), so a history run across
that change compared an old ref reading a fraction of the input with a
new ref reading all of it, and reported the difference as a CPU and
RSS regression.

The generator gains `--codex-layout legacy`, which writes the same
rollouts uncompressed under `sessions/`. bench-history keeps the
current layout when every ref is a commit containing #1890, and
otherwise generates the legacy layout and says so at the top of the
report; binary paths count as unknown. With a custom Codex fixture
that holds `.zst` files and a ref that cannot read them, and for
`--real` runs across the change, the report leads with a warning
instead. The PR comparison always builds refs that read the current
layout, so CI keeps compressed coverage.

Reported by CodeRabbit on #2061.

* fix(perf): check every Codex input class a compared ref may skip

The compatibility check only looked for `.jsonl.zst` files, but
releases before #1230 also skip `archived_sessions/`, so a custom
fixture with archived plain rollouts got no warning and a history run
across #1230 compared different input.

Replace the single zstd commit with a table of Codex input classes taken
from the Codex loader's history (`paths.rs`, `reader.rs`): archived
sessions (a9b12c4, #1230, v20.0.7) and zstd-compressed rollouts
(44a3aa8, #1890, v20.0.33). Each ref is checked per class by git
ancestry; the generated fixture switches to the legacy layout when any
ref skips any class, and a custom fixture or `--real` run warns per
class with the refs that skip it. Claude discovery has scanned
`projects/` recursively throughout v20 and the date-window skips (#1665,
#1794) do not apply to unbounded reports, so they are not classes.
Probing each ref's source for the reader code was rejected: the same
strings appear in tests and comments.

Reported by Cubic on #2061.

* fix(ci): run the perf comment upsert from the base commit

The comment job holds `pull-requests: write` and `issues: write`, but it
checked out the PR merge commit and ran that tree's setup action and
`upsert-pr-comment.nu`, so a PR that edited either ran them with the
write token. Check out `pull_request.base.sha` instead, still without
persisted credentials. The PR's report reaches the job only as an
artifact the script reads as text.

This PR does not change `upsert-pr-comment.nu`, and the base version
reads the same inputs (COMMENT_FILE as raw markdown, COMMENT_MARKER,
PR_NUMBER, GITHUB_REPOSITORY, GITHUB_TOKEN), so the first runs after
merge need no fallback.

The workflow file itself still comes from the PR on `pull_request`
events, so this removes the PR's scripts from the write job rather than
making the job a hard trust boundary.

Reported by CodeRabbit on #2061.

* fix(perf): match Codex input classes to the rollouts the loader reads

The input-class matchers looked at absolute paths: any `.zst` file
counted as a compressed rollout, and any path containing
`/archived_sessions/`, including a parent of the fixture, counted as
an archived session. Match paths relative to the Codex home against
what the loader reads: `*.jsonl` and `*.jsonl.zst` (reader.rs) under
`sessions/` and `archived_sessions/` (paths.rs). Tests cover
non-rollout `.zst` files, rollouts outside those directories,
non-rollout files in `archived_sessions/`, and a fixture under a
directory named `archived_sessions`.

Reported by Cubic on #2061.