[Perf] Linux/arm64: 1 Regression on 7/22/2026 1:13:11 AM +00:00

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	17c839c4055469e3e8dc3e47a76b47e417b63af9
Compare	97e5cfb1cb0abe25865e96267845e44a75a6f5b7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Get

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
GetUInt64 - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	515.67 ns	899.09 ns	1.74	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'

Details

System.Text.Json.Tests.Perf_Get.GetUInt64

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

🔍 Automated Triage Analysis

Summary: Verified regression in GetUInt64 — caused by commit d0afd87 ("Modernize System.Text.Json product code (#130976)").

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: d0afd877c60eb177eb7454d6a3fe06dca6b8b312

🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 3 dry-run (rule R4_known_regression_new — see Full Analysis for the table). · 🦆 Re-analysis: ⚠️ disputes (overconfident)

📊 Full Analysis (click to expand)

Summary: A single confirmed regression: System.Text.Json.Tests.Perf_Get.GetUInt64 slowed from ~516 ns to ~899 ns (+74.4%, z=49.9) on Linux/arm64 — a clean step change against a very stable 299-run history. The most likely cause is d0afd877c60e ("Modernize System.Text.Json product code"), which directly rewrote Utf8JsonReader.TryGet.cs and Utf8JsonReader.cs (the exact TryGetUInt64 parsing path) by converting backing fields to field-backed properties. Finding confidence 4/5 (299 prior runs, unverified rerun); regression confidence 4/5 (direct code-path hit, sharp step change).

Issue Overview

Issue: [Perf] Linux/arm64: 1 Regression on 7/22/2026 1:13:11 AM +00:00 #131600 - [Perf] Linux/arm64: 1 Regression on 7/22/2026 1:13:11 AM +00:00
Date: 2026-07-22 | Queue: AmpereUbuntu (Ubuntu.2204.Arm64.Perf) | OS/Arch: ubuntu 22.04/arm64
Commits: 17c839c4 → 97e5cfb1 (46 commits)
Labels: untriaged, perf-regression, arch-arm64, os-linux, runtime-coreclr, ampere, kind-micro, compilationmode-tiered, runkind-micro
Historical Data: Available (299 prior runs; 23 post-change runs)
Runtime stack: Desktop + CoreCLR (arm64). The regressing code is the System.Text.Json BCL, exercised on top of the CoreCLR JIT.

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Text.Json.Tests.Perf_Get.GetUInt64	515.67 ns	899.09 ns	+74.4%	516.37	7.67	299	49.9	5%	N	Step change; new stable plateau ~888–901 ns

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 — very strong historical baseline (299 runs, CV=1.49%), unambiguous step change; not yet confirmed by an isolated rerun/bisection.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
d0afd877c60e	`src/libraries/System.Text.Json/.../Reader/Utf8JsonReader.TryGet.cs`, `Utf8JsonReader.cs`, `Utf8JsonReader.MultiSegment.cs`	Shared	BCL, but this is the exact code path under test (`TryGetUInt64`). Direct mechanism.
d8cc47693442	`src/coreclr/jit/` (fold FP compares w/ NaN in morph)	LessLikely	JIT change; no clear tie to integer JSON parsing.
ae6601d39db1	`src/coreclr/jit/` (cleanup/harden lowering)	LessLikely	JIT change; could affect arm64 codegen broadly but no specific link.
231d8f980ba7	`src/coreclr/jit/gentree.cpp` (HWIntrinsic miscompile fixes)	LessLikely	Fixes miscompiles; not a parsing path.
ce44c2396a24	arm64 intrinsic/commutative marking	LessLikely	arm64 codegen tweak; no direct tie to scalar UInt64 parse.

Confirmed Regressions

System.Text.Json.Tests.Perf_Get.GetUInt64 (+74.4%)

Delta: 515.67 ns → 899.09 ns (+74.4%)
Historical Context: μ=516.37 ns (σ=7.67, n=299); z-score=49.9; adaptive threshold=5%. CV=1.49%. All 23 post-change runs fall in [888.0, 901.6] ns, forming a new stable plateau.
Assessment: Step change — sudden, sustained ~74% jump with no pre-existing drift; two isolated historical spikes (543, 645 ns) did not shift the mean.
Finding Confidence: 4/5 — comprehensive history and clean step; unverified by rerun.
Regression Confidence: 4/5 — extreme outlier on a very stable series, and the leading candidate directly rewrote the exact reader path (TryGetUInt64).
Likely Cause: d0afd877c60e - "Modernize System.Text.Json product code (Modernize System.Text.Json product code #130976)"
- Affected Areas: Collections/Serialization — System.Text.Json Utf8JsonReader number parsing
- Stack Relevance: Shared (BCL) — but it is the precise code under test, so the mechanism is direct rather than cross-stack.
- Rationale: The GetUInt64 benchmark drives Utf8JsonReader.TryGetUInt64. This commit modified Reader/Utf8JsonReader.TryGet.cs, Reader/Utf8JsonReader.cs, and Reader/Utf8JsonReader.MultiSegment.cs, converting 33 single-use private backing fields to field-backed properties and several methods to expression-bodied members. Although described as behavior-preserving, such rewrites can change JIT inlining/field-access codegen in a hot tight loop and plausibly regress arm64 tiered codegen by ~74%. Timing aligns exactly (authored 2026-07-21, regression detected 2026-07-22), and it is the only commit in the range touching the failing test's code path.
- Candidates Considered:
  - d0afd877c60e - directly rewrites Utf8JsonReader.TryGet.cs/Utf8JsonReader.cs (the GetUInt64 path).
  - d8cc47693442 - JIT morph FP/NaN folding; no tie to integer parse.
  - ae6601d39db1 - JIT lowering cleanup; broad but no specific link.
  - 231d8f980ba7 - HWIntrinsic miscompile fixes in gentree.cpp.
Verification Status: Not verified — isolated rerun/bisection deferred (to be run separately). Recommended to confirm d0afd87 vs its parent 0cf0d0a.

Open Questions

Historical Data: Strong (299 prior runs, CV=1.49%); no gaps.
Confidence Limitations: 46-commit range; several JIT/arm64 codegen commits exist that could theoretically affect codegen, though none touch the parsing path. Attribution rests on code-path relevance, not yet on measurement.
Verification Needed: Isolated rerun of d0afd87 vs parent 0cf0d0a; if inconclusive, bisect the 46-commit range focusing on the two arm64/JIT commits as a secondary hypothesis.
Bisection Status: Not performed (deferred to a separate step).

Recommended Actions

Verification - Rerun System.Text.Json.Tests.Perf_Get.GetUInt64 at d0afd877c60e and its parent 0cf0d0a8b497 on arm64 to confirm the delta reproduces across this single commit.
Source review - Inspect the Utf8JsonReader.TryGet.cs / Utf8JsonReader.cs diff for field-backed-property conversions on hot fields (e.g., value span, position, consumed counters) that may have lost inlining or introduced extra indirection on arm64.
Bisection (fallback) - If the single-commit rerun is inconclusive, bisect 17c839c4...97e5cfb1 using RunWithBuildAtHash, treating the arm64/JIT commits (d8cc476, ae6601d, ce44c23, 231d8f9) as the secondary hypothesis.
Profiling - If confirmed, collect hardware counters (collectHardwareCounters=true) to characterize the extra work (branch mispredictions / instructions retired) in the UInt64 parse loop.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
GetUInt64	✅ Culprit found	d0afd877c60e	✅ RELATED	515.7→899.1 ns (+74%)	515.4→896.6 ns (+74%)	2605s

Total bisection time: 2605s
Success rate: 1/1 tests

System.Text.Json.Tests.Perf_Get.GetUInt64

Culprit: d0afd877c60e
Commit Message: Modernize System.Text.Json product code (Modernize System.Text.Json product code #130976)
Relevance: RELATED
Explanation: The commit rewrites System.Text.Json product source code (114 files, including JsonDocument.Parse, JsonElement, JsonHelpers, and JsonReader-adjacent types) — exactly the assembly and code path exercised by System.Text.Json.Tests.Perf_Get.GetUInt64. Although the PR claims "no behavior change" (replacing 33 backing fields with field-backed properties and converting methods to expression-bodied members), such source restructuring changes JIT codegen/inlining and clearly introduced the +74% step, cleanly isolated from its good parent 0cf0d0a. The very next commit (6d73f82) is docs/skill-only and provably cannot affect the binary, ruling it out as the culprit.

Helix jobs

Job ID	Submitted by bisection	Last status
`584dcc63-addd-43e4-b94d-5f5236fc0f7f`	✅	submitted
`06e7dd98-42f0-4bd5-a282-4aba4c7f19ba`	✅	submitted
`d72a963e-6962-4153-b315-3f50343bfb9e`	✅	completed

Per-commit measurements

Commit	Position	Measurements
`17c839c40554`	baseline	515.500 ns
`73197a56f4e1`	idx3, good	515.100 ns
`dd3e6bcde7bf`	idx6, good	515.000 ns
`0cf0d0a8b497`	idx9, last good / parent	515.400 ns
`6d73f82b4f10`	idx11, docs-only	890.100 ns
`8435557a42e0`	idx23	889.700 ns
`8aab0ce0c7a6`	idx35	894.800 ns
`97e5cfb1cb0a`	head	894.600 ns
`d0afd877c60e`	idx10, culprit	896.600 ns

Bisection details

…[earlier output omitted]

- Baseline Value: 515.4 ns
- Culprit Value: 896.6 ns
- Summary: Bisected the 46-commit range with a quadrant sweep (baseline good ~515 ns, quarter-point idx11 already regressed ~890 ns), narrowed to indices 1–11, tested idx3/6/9 (all ~515 ns good), then tested idx10 d0afd877 (896.6 ns bad) versus its parent idx9 0cf0d0a8 (515.4 ns good) to isolate the single culprit commit.

HELIX JOBS:
- 584dcc63-addd-43e4-b94d-5f5236fc0f7f
- 06e7dd98-42f0-4bd5-a282-4aba4c7f19ba
- d72a963e-6962-4153-b315-3f50343bfb9e

MEASUREMENTS:
- 17c839c4055469e3e8dc3e47a76b47e417b63af9 (baseline): 515.5 ns
- 73197a56f4e1244f500896e0fc593ead9cb17b2c (idx3, good): 515.1 ns
- dd3e6bcde7bfb753db6c9775971a24e4fcd10773 (idx6, good): 515.0 ns
- 0cf0d0a8b497ac770d4a3a93ad228655c1c9146b (idx9, last good / parent): 515.4 ns
- d0afd877c60eb177eb7454d6a3fe06dca6b8b312 (idx10, culprit): 896.6 ns
- 6d73f82b4f10a1b42e81a31481fabd7c8bacf8cc (idx11, docs-only): 890.1 ns
- 8435557a42e0812e71b97725f6b34a5a42c3631b (idx23): 889.7 ns
- 8aab0ce0c7a62a30edd82c4d7024033c781ee126 (idx35): 894.8 ns
- 97e5cfb1cb0abe25865e96267845e44a75a6f5b7 (head): 894.6 ns

Culprit identified with high confidence: d0afd87 ("Modernize System.Text.Json product code #130976") introduced the +74% regression in Perf_Get.GetUInt64, confirmed by Helix measurement of the culprit (896.6 ns) vs its parent (515.4 ns).

🤖 Proposed Next Actions

Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

Verified regression attributed to commit d0afd87 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

#	Action	Status	Details
1	Back-reference cause PR	⏭️ Skipped (dry-run)	on culprit PR for `d0afd877c60e`
2	Transfer issue	⏭️ Skipped (dry-run)	→ dotnet/runtime
3	Ping owners	⏭️ Skipped (dry-run)	ping eiriktsarpalis

🦆 Pre-bisection Re-analysis (advisory)

✅ Agrees with the pre-bisection plan — advisory only, not applied

Rationale: The single finding shows noise=False with a strong change confidence of 4/5 and a matching 4/5 finding confidence, so classifying it as an actionable regression and queuing exactly that one test for bisection is well-supported and wastes no slots.

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

⚠️ Disputes the analysis — advisory only, not applied

Flags: overconfident

Rationale: The bisection converged successfully on the culprit with no alignment warning, so the culprit is well-supported, but the aggregate 5/5 finding and regression scores overstate the single underlying finding's 4/5 signals, and the packet lacks z-score/variance/sustained-after evidence to justify raising them to maximal, so both should stay at 4.

Suggested confidence: Finding Confidence → 4/5; Change Confidence → 4/5 (advisory — not applied)

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

Generated by PerfTriageAgent on 2026-07-30 09:24 UTC

@eiriktsarpalis

Likely due to #130976, FYI @eiriktsarpalis.

@EgorBot

Running the exact bisection boundary and current main as a control for the source-stage comparison.

@EgorBot -ubuntu24_azure_ampere -commits 0cf0d0a,d0afd877c60eb177eb7454d6a3fe06dca6b8b312,a3ca578813e6c71b644fef5c1076af436800bdf1 --filter "System.Text.Json.Tests.Perf_Get.GetUInt64"

Note