Remove some AST `tokens` fields by nnethercote · Pull Request #158942 · rust-lang/rust

nnethercote · 2026-07-08T05:53:02Z

This commit removes/moves the tokens field from various AST node types, shrinking them. Details in individual commits.

nnethercote · 2026-07-08T05:53:15Z

@bors try @rust-timer queue

Remove some AST `tokens` fields

rust-bors · 2026-07-08T07:59:51Z

☀️ Try build successful (CI)
Build commit: 3f39560 (3f39560f5a1e9cce0b5cdb22caeda8dd2e7d5460)
Base parent: 5e91de6 (5e91de65d75d3c849c643f5079509b9e5985a5c0)

rust-timer · 2026-07-08T08:46:40Z

Finished benchmarking commit (3f39560): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.7%, -0.2%]	20
Improvements ✅ (secondary)	-1.0%	[-1.7%, -0.1%]	11
All ❌✅ (primary)	-0.4%	[-0.7%, -0.2%]	20

Max RSS (memory usage)

Results (primary -0.3%, secondary -0.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.3%	[0.5%, 2.7%]	4
Regressions ❌ (secondary)	2.8%	[2.1%, 3.5%]	2
Improvements ✅ (primary)	-6.6%	[-6.6%, -6.6%]	1
Improvements ✅ (secondary)	-4.4%	[-5.3%, -3.5%]	2
All ❌✅ (primary)	-0.3%	[-6.6%, 2.7%]	5

Cycles

Results (primary -2.4%, secondary -9.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.4%	[-2.9%, -1.7%]	6
Improvements ✅ (secondary)	-9.4%	[-11.7%, -5.0%]	6
All ❌✅ (primary)	-2.4%	[-2.9%, -1.7%]	6

Binary size

Results (primary 0.0%, secondary 0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.0%	[0.0%, 0.0%]	4
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[0.0%, 0.0%]	4

Bootstrap: 487.812s -> 487.974s (0.03%)
Artifact size: 388.49 MiB -> 388.90 MiB (0.11%)

nnethercote · 2026-07-08T10:43:41Z

Good icount results on a few benchmarks, even better cycles results.

r? @petrochenkov

rustbot · 2026-07-08T10:43:46Z

rustfmt is developed in its own repository. If possible, consider making this change to rust-lang/rustfmt instead.

cc @rust-lang/rustfmt

Some changes occurred in compiler/rustc_attr_parsing

cc @jdonszelmann, @JonathanBrouwer

clippy is developed in its own repository. If possible, consider making this change to rust-lang/rust-clippy instead.

cc @rust-lang/clippy

The parser was modified, potentially altering the grammar of (stable) Rust
which would be a breaking change.

cc @fmease

Some changes occurred in compiler/rustc_builtin_macros/src/autodiff.rs

cc @ZuseZ4

mejrs · 2026-07-08T11:46:30Z

-            span: self.span,
-            tokens: self.tokens.clone(),
-        })
+        ensure_sufficient_stack(|| Self { id: self.id, kind: self.kind.clone(), span: self.span })


Assuming that ensure_sufficient_stack was necessary for cloning deeply nested tokens, this Clone impl can be a #[derive(Clone)] instead.

View changes since the review

Interesting suggestion! But self.kind can be recursive, so it might also/instead for deeply nested types.

nnethercote · 2026-07-08T11:53:22Z

It would also be nice to remove tokens from items, statements, and (especially) expressions. But those cases are much more complicated, and aren't neatly chokepointed by parse_nonterminal.

petrochenkov · 2026-07-09T12:28:02Z

This is very nice, especially for Paths, because every tiny variable in every function body is a Path.

Left some comment nits.
@rustbot author

rustbot · 2026-07-09T12:28:06Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

These ones are always `None` and never read.

Specifically, from `Block`, `Pat`, `Path`, `Ty`, `Visibility`. This field is usually `None`. It only gets filled in for paths that might later be re-tokenized via `TokenStream::from_ast`, e.g. because of macro or cfg use. All such cases go through a single place: `parse_nonterminal`. This commit changes `parse_nonterminal` to store the non-`None` lazy attr token stream *next* to the node in `ParseNtResult` (using the new `WithTokens` type) instead of *inside* the parsed node. Along with some minor changes to the relevant `HasTokens` and `HasAttrs` impls, this is enough for things to work. These AST nodes all shrink by 8 bytes, as do various other nodes that contain nodes of these types. A guide to the changes: - `compiler/rustc_ast/src/tokenstream.rs` has the new `WithTokens` type. - `compiler/rustc_parse/src/parser/nonterminal.rs`, `compiler/rustc_expand/src/mbe/transcribe.rs` and `compiler/rustc_parse/src/parser/mod.rs` have minor changes due to the changes to `ParseNtResult`. - `compiler/rustc_ast/src/ast_traits.rs` has the `HasTokens`/`HasAttrs` changes. - Everything else is just mechanical changes for the removal of the `tokens` field.

The same basic idea from the previous commit applies: the tokens are only sometimes needed and we can store them next to the `AttrItem` (using `WithTokens`) instead of inside the `AttrItem`. The change is a bit more complex because more than just `parse_nonterminal` is involved (e.g. `parse_attr_item`, `parse_cfg_attr`, `expand_cfg_attr_item`).

rustbot · 2026-07-10T07:21:22Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

nnethercote · 2026-07-10T07:22:04Z

The job x86_64-gnu-nopt failed!

The PR got broken by another PR that landed and added some new code. I have fixed the breakage.

@bors r=petrochenkov

rust-bors · 2026-07-10T07:22:07Z

📌 Commit 351026d has been approved by petrochenkov

It is now in the queue for this repository.

rust-bors · 2026-07-10T23:17:39Z

☀️ Test successful - CI
Approved by: petrochenkov
Duration: 3h 24m 10s
Pushing 375b143 to main...

github-actions · 2026-07-10T23:20:48Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing ae705ae (parent) -> 375b143 (this PR)

Test differences

Show 42 test diffs

42 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 375b1431b7d89d1c2e2bc168c011848ae12b7d14 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

i686-msvc-1: 1h 39m -> 3h 4m (+85.3%)
x86_64-gnu-gcc-core-tests: 8m 18s -> 14m 46s (+77.8%)
x86_64-gnu-llvm-21: 56m 23s -> 1h 20m (+42.8%)
x86_64-gnu-llvm-21-3: 1h 24m -> 1h 57m (+38.9%)
dist-x86_64-illumos: 1h 22m -> 1h 46m (+29.6%)
dist-loongarch64-linux: 1h 28m -> 1h 50m (+25.2%)
i686-gnu-nopt-2: 2h 15m -> 1h 43m (-23.6%)
dist-sparcv9-solaris: 1h 13m -> 1h 29m (+23.0%)
dist-arm-linux-musl: 1h 16m -> 1h 32m (+21.0%)
optional-x86_64-gnu-parallel-frontend: 1h 50m -> 2h 12m (+20.1%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2026-07-10T23:59:03Z

Finished benchmarking commit (375b143): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.7%, -0.2%]	23
Improvements ✅ (secondary)	-1.1%	[-1.9%, -0.2%]	11
All ❌✅ (primary)	-0.4%	[-0.7%, -0.2%]	23

Max RSS (memory usage)

Results (primary 1.0%, secondary -0.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.0%	[1.0%, 2.9%]	2
Regressions ❌ (secondary)	3.6%	[3.6%, 3.6%]	1
Improvements ✅ (primary)	-0.9%	[-0.9%, -0.9%]	1
Improvements ✅ (secondary)	-2.4%	[-4.0%, -0.8%]	2
All ❌✅ (primary)	1.0%	[-0.9%, 2.9%]	3

Cycles

Results (primary -2.4%, secondary -5.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.4%	[-2.6%, -2.2%]	2
Improvements ✅ (secondary)	-5.4%	[-7.6%, -3.2%]	2
All ❌✅ (primary)	-2.4%	[-2.6%, -2.2%]	2

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 491.745s -> 490.777s (-0.20%)
Artifact size: 389.36 MiB -> 389.74 MiB (0.10%)

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 8, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jul 8, 2026

Auto merge of #158942 - nnethercote:rm-ast-tokens-fields, r=<try>

3f39560

Remove some AST `tokens` fields

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 8, 2026

nnethercote force-pushed the rm-ast-tokens-fields branch from d3bb72f to dbcb5a4 Compare July 8, 2026 10:39

nnethercote marked this pull request as ready for review July 8, 2026 10:43

rustbot assigned petrochenkov Jul 8, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 8, 2026

mejrs reviewed Jul 8, 2026

View reviewed changes

petrochenkov reviewed Jul 9, 2026

View reviewed changes

Comment thread compiler/rustc_parse/src/parser/mod.rs Outdated

Comment thread compiler/rustc_parse/src/parser/attr.rs Outdated

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 9, 2026

nnethercote force-pushed the rm-ast-tokens-fields branch from dbcb5a4 to 589a21e Compare July 9, 2026 13:43

This comment has been minimized.

Sign in to view

nnethercote added 4 commits July 10, 2026 17:11

Add a size assertion for NormalAttr

8d2b75a

Remove unnecessary tokens fields from AST nodes

0a9728b

These ones are always `None` and never read.

nnethercote force-pushed the rm-ast-tokens-fields branch from 589a21e to 351026d Compare July 10, 2026 07:21

rust-bors Bot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 10, 2026

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 10, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot added merged-by-bors This PR was explicitly merged by bors. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 10, 2026

rust-bors Bot merged commit 375b143 into rust-lang:main Jul 10, 2026
14 checks passed

rustbot added this to the 1.99.0 milestone Jul 10, 2026

nnethercote deleted the rm-ast-tokens-fields branch July 11, 2026 21:11

nnethercote mentioned this pull request Jul 13, 2026

Remove NormalAttr #159221

Closed

petrochenkov mentioned this pull request Jul 13, 2026

Make HasTokens a sub-trait of HasAttrs. #159160

Merged

Uh oh!

Uh oh!

Conversation

nnethercote commented Jul 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nnethercote commented Jul 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Jul 8, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jul 8, 2026

Overall result: ✅ improvements - no action needed

Uh oh!

nnethercote commented Jul 8, 2026

Uh oh!

rustbot commented Jul 8, 2026

Uh oh!

mejrs Jul 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nnethercote commented Jul 8, 2026

Uh oh!

Uh oh!

Uh oh!

petrochenkov commented Jul 9, 2026

Uh oh!

rustbot commented Jul 9, 2026

Uh oh!

This comment has been minimized.

rustbot commented Jul 10, 2026

Uh oh!

nnethercote commented Jul 10, 2026

Uh oh!

rust-bors Bot commented Jul 10, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jul 10, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 10, 2026

Test differences

Job duration changes

Uh oh!

rust-timer commented Jul 10, 2026

Overall result: ✅ improvements - no action needed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nnethercote commented Jul 8, 2026 •

edited by rustbot

Loading

mejrs Jul 8, 2026 •

edited by rustbot

Loading