Skip to content

fix(track-memory-allocations): reserve transform scoping capacity#22621

Merged
graphite-app[bot] merged 1 commit into
mainfrom
codex/fix-track-memory-allocs
May 20, 2026
Merged

fix(track-memory-allocations): reserve transform scoping capacity#22621
graphite-app[bot] merged 1 commit into
mainfrom
codex/fix-track-memory-allocs

Conversation

@Dunqing

@Dunqing Dunqing commented May 20, 2026

Copy link
Copy Markdown
Member

Issue

The Allocations CI job can fail on PRs that do not touch the transformer because cargo allocs can rewrite allocs_transformer.snap differently across platforms. In the observed failure, Linux x64 reported one extra transformer system allocation for App.tsx, while macOS/arm64 still reported the old count.

Why

The allocation tracker was passing transformer a Scoping built differently from the production compiler path. Production builds semantic data with SemanticBuilder::with_excess_capacity(2.0) before running transforms, because transforms may add scopes, symbols, and references. The allocation tracker did not reserve that extra semantic capacity.

That made the transformer snapshot accidentally include growth of semantic scoping storage. In this case, JSX automatic runtime import generation creates a new root-scope binding. The traced Linux-only allocation happened under JsxImpl::get_create_element -> AutomaticModuleBindings::import_jsx -> TraverseScoping::add_binding -> Scoping::create_symbol, when pushing the generated symbol forced the semantic Scoping arena to allocate another chunk.

Linux got a different allocation count because this is a capacity-boundary effect, not different transformer behavior. The Scoping arena layout depends on target layout and alignment. On Linux x64, the generated binding crossed the arena chunk boundary and allocated a new 16-byte-aligned chunk, so the measured transformer count became 27. On macOS/arm64, the same push still fit in the existing semantic arena capacity, so the measured count stayed 26.

So the root cause is that transformer allocation tracking was measuring platform-sensitive semantic storage growth. Any unrelated PR could expose the same stale snapshot whenever CI regenerated allocations on a platform that lands on the other side of that capacity boundary.

Fix

Keep allocs_semantic.snap measuring regular semantic analysis, but build a separate production-like semantic Scoping for transformer measurement using with_excess_capacity(2.0). This matches the compiler and transformer benchmarks, and removes the semantic arena capacity cliff from the transformer allocation snapshot.

The transformer allocation snapshot is regenerated after the measurement fix.

Validation

Verified cargo allocs leaves the branch clean on macOS/arm64 and in Linux amd64 Docker.

AI usage: I used AI assistance to investigate, implement, and validate this change.

@Dunqing
Dunqing requested review from camc314 and leaysgur May 20, 2026 09:55
@Dunqing
Dunqing marked this pull request as ready for review May 20, 2026 09:59
@Dunqing Dunqing added the 0-merge Merge with Graphite Merge Queue label May 20, 2026

Dunqing commented May 20, 2026

Copy link
Copy Markdown
Member Author

Merge activity

…2621)

## Issue

The `Allocations` CI job can fail on PRs that do not touch the transformer because `cargo allocs` can rewrite `allocs_transformer.snap` differently across platforms. In the observed failure, Linux x64 reported one extra transformer system allocation for `App.tsx`, while macOS/arm64 still reported the old count.

## Why

The allocation tracker was passing transformer a `Scoping` built differently from the production compiler path. Production builds semantic data with `SemanticBuilder::with_excess_capacity(2.0)` before running transforms, because transforms may add scopes, symbols, and references. The allocation tracker did not reserve that extra semantic capacity.

That made the transformer snapshot accidentally include growth of semantic scoping storage. In this case, JSX automatic runtime import generation creates a new root-scope binding. The traced Linux-only allocation happened under `JsxImpl::get_create_element -> AutomaticModuleBindings::import_jsx -> TraverseScoping::add_binding -> Scoping::create_symbol`, when pushing the generated symbol forced the semantic `Scoping` arena to allocate another chunk.

Linux got a different allocation count because this is a capacity-boundary effect, not different transformer behavior. The `Scoping` arena layout depends on target layout and alignment. On Linux x64, the generated binding crossed the arena chunk boundary and allocated a new 16-byte-aligned chunk, so the measured transformer count became 27. On macOS/arm64, the same push still fit in the existing semantic arena capacity, so the measured count stayed 26.

So the root cause is that transformer allocation tracking was measuring platform-sensitive semantic storage growth. Any unrelated PR could expose the same stale snapshot whenever CI regenerated allocations on a platform that lands on the other side of that capacity boundary.

## Fix

Keep `allocs_semantic.snap` measuring regular semantic analysis, but build a separate production-like semantic `Scoping` for transformer measurement using `with_excess_capacity(2.0)`. This matches the compiler and transformer benchmarks, and removes the semantic arena capacity cliff from the transformer allocation snapshot.

The transformer allocation snapshot is regenerated after the measurement fix.

## Validation

Verified `cargo allocs` leaves the branch clean on macOS/arm64 and in Linux amd64 Docker.

AI usage: I used AI assistance to investigate, implement, and validate this change.
@graphite-app
graphite-app Bot force-pushed the codex/fix-track-memory-allocs branch from 59df140 to b9171ea Compare May 20, 2026 10:05
@graphite-app
graphite-app Bot merged commit b9171ea into main May 20, 2026
27 checks passed
@graphite-app graphite-app Bot removed the 0-merge Merge with Graphite Merge Queue label May 20, 2026
@graphite-app
graphite-app Bot deleted the codex/fix-track-memory-allocs branch May 20, 2026 10:09
graphite-app Bot pushed a commit that referenced this pull request Jul 8, 2026
…24292)

## Issue

The `Allocations` CI job can fail on PRs with a 1-allocation `Sys allocs` difference that local `cargo allocs` cannot reproduce. #22621 fixed one instance of this in the transformer stage. #24112 and #24017 hit the same failure again in the minifier stage: `kitchen-sink.tsx` measures 2309 Sys allocs on macOS/arm64 but 2310 on Linux/x64, so whichever platform regenerates the snapshot last breaks CI for the other.

## Why

The tracker counts every call to the system allocator. That includes the chunks bump arenas request. Whether an arena needs one more chunk depends on the total number of bytes in the arena, and byte totals are not the same on every target. For example, hashbrown tables use 16-byte control groups on x86_64 but 8-byte groups on aarch64. When an arena's content size sits close to a chunk boundary, one target crosses it and the other does not.

In the observed failure, the minifier's mangler-phase semantic build creates a fresh `Scoping`. Its arena holds symbol names, resolved-reference vectors, and the per-scope `bindings` hash tables, which are slightly larger on x86_64. The traced Linux-only allocation happened under `Scoping::add_resolved_reference -> ArenaVec::push -> Arena::new_chunk`, when the push overflowed the current chunk and requested a new 720880-byte chunk. On aarch64 the same content fit in the existing chunk, so the count stayed one lower.

Every other allocation class in the measurement grows on element counts, which are identical across platforms. Logging every allocation size during the minifier stage on macOS/arm64, Linux/arm64, and Linux/x64 showed the three sequences match one-to-one except for arena chunk requests.

Any PR that shifts arena content near a chunk boundary can trip this again in any stage, so fixing one stage at a time does not close the class.

## Fix

Count chunk allocations in `oxc_allocator` under the tracker-only `track_allocations` feature, and subtract them from the reported Sys allocs in the tracker. The counter is global rather than per-arena because short-lived arenas (like the one backing `Scoping`) are created and dropped inside the measured operation.

Arena usage is still measured by the `Arena allocs` / `Arena reallocs` columns; only the platform-sensitive chunk-request count is excluded from `Sys allocs`.

Parser and transformer rows are unchanged (the main arena is pre-warmed, and #22621 already removed transformer scoping growth). Semantic, minifier, and formatter rows drop by their per-stage chunk counts.

## Validation

`cargo allocs` produces byte-identical snapshots on macOS/arm64, Linux/arm64 (Docker), and Linux/x64 (Docker), and is idempotent across reruns. `cargo test -p oxc_allocator` passes; clippy is clean with `track_allocations` both on and off, and the counter compiles out entirely without the feature.

AI usage: AI assistance was used to investigate, implement, and validate this change.
graphite-app Bot pushed a commit that referenced this pull request Jul 15, 2026
The `track_allocations` tracking already counts how many chunks arenas
request from the system allocator (so `tasks/track_memory_allocations` can
exclude them from its system-allocator counts, which chunk requests would
otherwise make platform-dependent — see #22621). Record the total byte size
of those chunks too, and extend the accessor to
`Allocator::global_chunk_allocation_stats() -> (count, bytes)`, matching the
`get_allocation_stats()` shape.

No snapshot changes. This enables the next commit to exclude chunk bytes
from a new total-heap-allocated-bytes metric, the same way chunk counts are
excluded from `sys allocs` today.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
graphite-app Bot pushed a commit that referenced this pull request Jul 15, 2026
The `track_allocations` tracking already counts how many chunks arenas
request from the system allocator (so `tasks/track_memory_allocations` can
exclude them from its system-allocator counts, which chunk requests would
otherwise make platform-dependent — see #22621). Record the total byte size
of those chunks too, and extend the accessor to
`Allocator::global_chunk_allocation_stats() -> (count, bytes)`, matching the
`get_allocation_stats()` shape.

No snapshot changes. This enables the next commit to exclude chunk bytes
from a new total-heap-allocated-bytes metric, the same way chunk counts are
excluded from `sys allocs` today.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
graphite-app Bot pushed a commit that referenced this pull request Jul 15, 2026
…ependent (#24565)

The `Allocations` CI job fails whenever the snapshots are regenerated on a different CPU architecture than CI (e.g. https://github.com/oxc-project/oxc/actions/runs/29420345285/job/87368903834): every `arena size` value differs by 32–128 bytes between aarch64 and x86_64. The cause is target-dependent type layout — hashbrown's table control groups are 16 bytes wide on x86_64 (SSE2) but 8 bytes on aarch64 (NEON) — so each live `HashMap` in the arena shifts `used_bytes()` slightly. This is the same root cause that #22621 handled for chunk counts; the `arena size` gauge added in #24553 re-exposed it. Allocation counts are unaffected and stay exact.

Exact byte equality across architectures isn't achievable (the arena genuinely occupies a different number of bytes), so instead the snapshot writer tolerates layout noise:

- Snapshots are now YAML (`allocs_*.yaml`) with the same layout as before: one value per line, file names at column 0 for git hunk headers, exact numbers as the diffed values, and human-readable sizes moved to trailing `#` comments.
- When regenerating, `cargo allocs` parses the committed snapshot (with `saphyr`, already a workspace dependency used by `oxc_coverage`) and keeps the committed `arena size` when the measured value is within `ARENA_SIZE_TOLERANCE` (1024 bytes, ~8× the observed worst-case cross-arch drift). Differences beyond the tolerance — real regressions such as growing an AST node type change sizes by orders of magnitude more — rewrite the value and fail the `git diff --exit-code` check as before.
- The trailing comments are rendered from the recorded value, so a snapshot regenerated on any platform stays byte-identical while within tolerance. No CI workflow changes are needed, and drift can't accumulate silently: each run compares against the committed value, so creeping sub-tolerance changes eventually cross the threshold and surface.

The committed values here are aarch64-measured; x86_64 CI measures 32–128 bytes higher and keeps them. Open PRs that touch the old `.snap` files will need a rebase and a `cargo allocs` rerun.

🤖 Implemented with [Claude Code](https://claude.com/claude-code).
graphite-app Bot pushed a commit that referenced this pull request Jul 16, 2026
…r heap and arena (#24587)

Follow-up to #24565. Restructures the task so that adding a new heap or arena metric is a small, local change instead of edits scattered across measurement, diffing, rendering, and committed-value lookup:

- `HeapTracker` groups the global-allocator atomics into one static `HEAP` with `read()`/`reset()`; its doc comment describes how to add a heap counter.
- `Counters { heap: HeapCounters, arena: ArenaCounters }` replaces the flat `AllocatorStats`, with `record()` / `diff_since()` (the arena-chunk exclusion logic from #22621 is unchanged, just relocated).
- `StageStats::metrics()` is now the single declaration of what a snapshot contains: an ordered list of `Metric`s, each a label plus a `MetricKind`.
- `MetricKind` encodes the per-metric snapshot policy: `Count` (platform-exact), `ExactBytes` (exact, human-readable comment), `ApproxBytes` (tolerates cross-architecture layout noise by keeping the committed value within `APPROX_BYTES_TOLERANCE`, renamed from `ARENA_SIZE_TOLERANCE` since it is no longer specific to arena size).
- `write_snapshot` / `render_file_stats` are fully metric-driven; nothing downstream needs touching when a metric is added.

Adding e.g. a `sys deallocs` count is now: one `AtomicCounter` in `HeapTracker`, one field in `HeapCounters`, one `Metric::count(...)` entry. A new byte gauge (heap peak, arena capacity) is one gauge read plus one `Metric::approx_bytes(...)` entry.

Behavior is unchanged: the committed snapshots are byte-identical when regenerated with the refactored code.

🤖 Implemented with [Claude Code](https://claude.com/claude-code).
graphite-app Bot pushed a commit that referenced this pull request Jul 17, 2026
… peak growth (#24619)

Adds three heap-side metrics to the allocation snapshots, using the metric machinery from #24587:

- `sys deallocs` (exact count) — system deallocations per stage; read against `sys allocs` it shows whether a stage churns or retains heap.
- `sys alloc bytes` — cumulative bytes requested from the system allocator per stage (allocation sizes plus reallocation growth).
- `sys peak growth` — high-water mark of live heap during a stage relative to the live bytes at stage start, via live-bytes accounting in the `GlobalAlloc` wrapper. Captures transient heap a stage needs even when it frees it before finishing.

Arena chunk operations are excluded from all `sys` metrics at the source: `oxc_allocator` marks each chunk allocation/deallocation immediately before calling the system allocator (task-only `track_allocations` feature, compiled out everywhere else), and the task's tracking allocator consumes the marker and skips the call. This replaces the previous global chunk counter + per-stage subtraction: a subtraction can correct counts but not a high-water mark, and chunk sizes are quantized and platform-dependent (whether an arena needs one more chunk depends on byte totals that vary across architectures with type layout — the #22621 mechanism). The first CI run demonstrated this concretely: kitchen-sink's minifier peak differed deterministically by 720 kB between aarch64 and x86_64 because the mangler's temporary arena crossed a chunk-growth boundary only on x86_64; and antd's formatter "peak" was 762 MB of main-arena chunk-doubling history rather than formatter behavior (now 13.8 MB of actual heap use).

Cross-platform tolerances: the cumulative byte metrics still drift with the number of heap hashbrown tables allocated (their control-group width is architecture-dependent), so they use a relative tolerance (1%, 4 kB floor) via the committed-value comparison from #24565; `arena size` keeps its tight 1024-byte tolerance.

Sample values for checker.ts (2.9 MB source): parser needs only 2.4 kB of transient heap (fully arena-based), semantic peaks at 3.9 MB, the minifier is the pipeline's largest heap consumer at 10.2 MB, and the formatter requests 5.9 MB total while peaking at 3 MB (heavy alloc/free churn).

🤖 Implemented with [Claude Code](https://claude.com/claude-code).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants