Unicode data: reduce size of to_lower/to_upper tables by Kmeakin · Pull Request #152954 · rust-lang/rust

Kmeakin · 2026-02-22T05:21:29Z

Reduces the combined size of to_lower and to_upper from 25,364 bytes to 3,110 bytes. Explained in detail in the doc comments

rustbot · 2026-02-22T05:21:34Z

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: bootstrap
bootstrap expanded to 6 candidates
Random selection from Mark-Simulacrum, clubby789, jieyouxu

Mark-Simulacrum

Will need to review the actual implementation as well, but would like more context on the algorithm selection before doing so.

View changes since this review

rustbot · 2026-02-23T01:17:39Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

Kobzol · 2026-02-23T10:49:40Z

@bors try @rust-timer queue

Unicode data: reduce size of to_lower/to_upper tables

rust-bors · 2026-02-23T13:01:49Z

☀️ Try build successful (CI)
Build commit: dcce5a6 (dcce5a661ada2ae2006af256bf8e3b0d882272d7, parent: eeb94be79adc9df7a09ad0b2421f16e60e6d932c)

rust-timer · 2026-02-23T13:42:10Z

Finished benchmarking commit (dcce5a6): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (secondary -4.9%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.9%	[-5.3%, -4.5%]	2
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary -4.9%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.9%	[-4.9%, -4.9%]	1
All ❌✅ (primary)	-	-	0

Binary size

Results (primary -0.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.2%	[-0.3%, -0.2%]	4
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.2%	[-0.3%, -0.2%]	4

Bootstrap: 481.498s -> 480.969s (-0.11%)
Artifact size: 395.88 MiB -> 397.86 MiB (0.50%)

Kmeakin · 2026-03-05T01:18:31Z

@rustbot ready

Mark-Simulacrum

Probably can squash the commits at least a little? r=me with that and the nit fixed.

View changes since this review

Instead of generating a standalone executable to test `unicode_data`, generate normal tests in `coretests`. This ensures tests are always generated, and will be run as part of the normal testsuite. Also change the generated tests to loop over lookup tables, rather than generating a separate `assert_eq!()` statement for every codepoint. The old approach produced a massive (20,000 lines plus) file which took minutes to compile!

Add a doc-comment to the top of `case-mapping.rs`.

rustbot · 2026-03-08T22:40:31Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

Kmeakin · 2026-03-08T22:41:53Z

r=@Mark-Simulacrum

ognevny · 2026-03-09T05:48:37Z

r=@Mark-Simulacrum

that should be @bors r=Mark-Simulacrum :)

Kmeakin · 2026-03-09T19:05:24Z

@bors r=Mark-Simulacrum

rust-bors · 2026-03-09T19:09:18Z

@Kmeakin: 🔑 Insufficient privileges: not in review users

Mark-Simulacrum · 2026-03-10T02:11:06Z

@bors r+

rust-bors · 2026-03-10T02:11:09Z

📌 Commit 902199b has been approved by Mark-Simulacrum

It is now in the queue for this repository.

rust-bors · 2026-03-10T09:27:29Z

☀️ Test successful - CI
Approved by: Mark-Simulacrum
Duration: 3h 7m 8s
Pushing 3bc6ea5 to main...

github-actions · 2026-03-10T09:30:47Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 595f14b (parent) -> 3bc6ea5 (this PR)

Test differences

Show 32 test diffs

Stage 1

unicode::alphabetic: [missing] -> pass (J2)
unicode::case_ignorable: [missing] -> pass (J2)
unicode::cased: [missing] -> pass (J2)
unicode::grapheme_extend: [missing] -> pass (J2)
unicode::lowercase: [missing] -> pass (J2)
unicode::n: [missing] -> pass (J2)
unicode::to_lowercase: [missing] -> pass (J2)
unicode::to_uppercase: [missing] -> pass (J2)
unicode::uppercase: [missing] -> pass (J2)
unicode::white_space: [missing] -> pass (J2)

Stage 2

unicode::alphabetic: [missing] -> pass (J0)
unicode::case_ignorable: [missing] -> pass (J0)
unicode::cased: [missing] -> pass (J0)
unicode::grapheme_extend: [missing] -> pass (J0)
unicode::lowercase: [missing] -> pass (J0)
unicode::n: [missing] -> pass (J0)
unicode::to_lowercase: [missing] -> pass (J0)
unicode::to_uppercase: [missing] -> pass (J0)
unicode::uppercase: [missing] -> pass (J0)
unicode::white_space: [missing] -> pass (J0)
unicode::alphabetic: [missing] -> ignore (J1)
unicode::case_ignorable: [missing] -> ignore (J1)
unicode::cased: [missing] -> ignore (J1)
unicode::grapheme_extend: [missing] -> ignore (J1)
unicode::lowercase: [missing] -> ignore (J1)
unicode::n: [missing] -> ignore (J1)
unicode::to_lowercase: [missing] -> ignore (J1)
unicode::to_uppercase: [missing] -> ignore (J1)
unicode::uppercase: [missing] -> ignore (J1)
unicode::white_space: [missing] -> ignore (J1)

Additionally, 2 doctest diffs were found. These are ignored, as they are noisy.

Job group index

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 3bc6ea5673e3e292db1e5b6ecaaaa883ca5af0b8 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

pr-check-1: 33m 13s -> 27m 17s (-17.8%)
x86_64-gnu-llvm-20-3: 2h 1m -> 1h 44m (-14.3%)
dist-aarch64-apple: 1h 47m -> 1h 33m (-12.8%)
dist-aarch64-msvc: 1h 34m -> 1h 46m (+12.7%)
i686-gnu-2: 1h 42m -> 1h 30m (-12.3%)
x86_64-gnu-tools: 1h 2m -> 55m 29s (-11.5%)
pr-check-2: 43m 39s -> 38m 45s (-11.2%)
x86_64-rust-for-linux: 51m 54s -> 46m 22s (-10.7%)
aarch64-gnu-llvm-20-1: 59m 1s -> 53m 18s (-9.7%)
test-various: 2h 7m -> 1h 55m (-9.6%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2026-03-10T10:08:08Z

Finished benchmarking commit (3bc6ea5): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.2%	[0.0%, 0.3%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

Results (primary -0.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.5%	[-0.6%, -0.5%]	8
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.5%	[-0.6%, -0.5%]	8

Bootstrap: 478.928s -> 479.559s (0.13%)
Artifact size: 394.95 MiB -> 394.94 MiB (-0.00%)

rustbot assigned Mark-Simulacrum Feb 22, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Feb 22, 2026

Mark-Simulacrum requested changes Feb 23, 2026

View reviewed changes

Comment thread src/tools/unicode-table-generator/src/case_mapping.rs Outdated

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 23, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 23, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Feb 23, 2026

Auto merge of #152954 - Kmeakin:km/unicode-data/case-mapping, r=<try>

dcce5a6

Unicode data: reduce size of to_lower/to_upper tables

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 23, 2026

Kmeakin force-pushed the km/unicode-data/case-mapping branch 2 times, most recently from 51870c1 to abe23a5 Compare March 4, 2026 00:26

This comment has been minimized.

Sign in to view

Kmeakin force-pushed the km/unicode-data/case-mapping branch 2 times, most recently from 23fdbc4 to 3f4a072 Compare March 5, 2026 01:13

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 5, 2026

Kmeakin requested a review from Mark-Simulacrum March 6, 2026 13:23

Kmeakin force-pushed the km/unicode-data/case-mapping branch from 3f4a072 to eb10a54 Compare March 6, 2026 21:33

Mark-Simulacrum approved these changes Mar 8, 2026

View reviewed changes

Comment thread src/tools/unicode-table-generator/src/case_mapping.rs Outdated

Kmeakin added 2 commits March 8, 2026 22:38

Document the case-mapping lookup tables

724a36d

Add a doc-comment to the top of `case-mapping.rs`.

Compress case-mapping tables

902199b

Kmeakin force-pushed the km/unicode-data/case-mapping branch from eb10a54 to 902199b Compare March 8, 2026 22:40

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 10, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot added merged-by-bors This PR was explicitly merged by bors. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 10, 2026

rust-bors Bot merged commit 3bc6ea5 into rust-lang:main Mar 10, 2026
12 checks passed

rustbot added this to the 1.96.0 milestone Mar 10, 2026

This was referenced Mar 10, 2026

Add APIs for dealing with titlecase #122668

Merged

reland and fix RUST-147622 #151277

Draft

Kmeakin deleted the km/unicode-data/case-mapping branch March 10, 2026 13:58

Uh oh!

Conversation

Kmeakin commented Feb 22, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Feb 22, 2026

Uh oh!

Mark-Simulacrum left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rustbot commented Feb 23, 2026

Uh oh!

Kobzol commented Feb 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Feb 23, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Feb 23, 2026

Overall result: no relevant changes - no action needed

Uh oh!

This comment has been minimized.

Kmeakin commented Mar 5, 2026

Uh oh!

Mark-Simulacrum left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rustbot commented Mar 8, 2026

Uh oh!

Kmeakin commented Mar 8, 2026

Uh oh!

ognevny commented Mar 9, 2026

Uh oh!

Kmeakin commented Mar 9, 2026

Uh oh!

rust-bors Bot commented Mar 9, 2026

Uh oh!

Mark-Simulacrum commented Mar 10, 2026

Uh oh!

rust-bors Bot commented Mar 10, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Mar 10, 2026

Uh oh!

Uh oh!

github-actions Bot commented Mar 10, 2026

Test differences

Stage 1

Stage 2

Job duration changes

Uh oh!

rust-timer commented Mar 10, 2026

Overall result: ❌ regressions - no action needed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Kmeakin commented Feb 22, 2026 •

edited by rustbot

Loading

Mark-Simulacrum left a comment •

edited by rustbot

Loading

Mark-Simulacrum left a comment •

edited by rustbot

Loading