[Perf] Linux/x64: 2 Regressions on 7/5/2026 4:58:39 PM +00:00 #130698

Activity

1. performanceautofiler

   added

   arch-x64

   os-linuxLinux OS (any supported distro)Linux OS (any supported distro)

   runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jul 14, 2026
2. LoopedBard3 commented on Jul 14, 2026

   

   LoopedBard3

   on Jul 14, 2026

   Last edited by LoopedBard3

   Member

   More actions

   🔍 Automated Triage Analysis

   Summary: Verified regression in Pin(Size: 512) — caused by commit 8f365c9 ("Publish GCHandle assignments with release semantics (#130213)"). 2 tests bisected to related culprit commits.

   Finding Confidence: 5/5 (analysis accuracy)
   Regression Confidence: 5/5 (likelihood of true regression)

   Likely Cause: 8f365c934088b8ae21a4dfce8073da0d3eb22a54

   🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 3 dry-run (rule R4_known_regression_new — see Full Analysis for the table). · 🦆 Re-analysis: ⚠️ disputes (overconfident-aggregate)

   📊 Full Analysis (click to expand)

   Summary: Two Memory.Pin regressions detected (+9.2% Char, +6.1% Byte) attributed to commit 8f365c9 which added VolatileStore release semantics to GCHandle assignment. Finding confidence 4/5 (excellent historical data, 299 runs, clear step change), regression confidence 4/5 (direct code path hit, z=3.16, precise temporal alignment with 4-commit range).

   Issue Overview

   • Issue: [Perf] Linux/x64: 2 Regressions on 7/5/2026 4:58:39 PM +00:00 #130698 - [Perf] Linux/x64: 2 Regressions on 7/5/2026 4:58:39 PM +00:00
   • Date: 2026-07-05 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux/x64
   • Commits: 58dfec78ed8d → ff279bafa8ca (4 commits)
   • Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
   • Historical Data: Available (297–299 runs per test, 45 post-change runs)

   Test Analysis

   Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
   System.Memory.Memory<Char>.Pin(Size: 512)	20.90 ns	22.83 ns	+9.22%	20.602	0.704	299	3.16	6.83%	N	Step change, confirmed regression
   System.Memory.Memory<Byte>.Pin(Size: 512)	20.65 ns	21.91 ns	+6.07%	20.306	0.681	297	2.35	6.71%	N	Step change, borderline (delta just below adaptive threshold but z > 2)

   Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

   Finding Confidence: 4/5 - Excellent historical data (297–299 runs), low CV (3.4%), clear step-change pattern sustained across 45 post-change runs. Minor downgrade from 5 due to a few outliers (26.4 ns spikes) slightly inflating σ.

   Candidate Stack Relevance

   Runtime stack: Desktop + CoreCLR (implicit)

   Commit	Touched Paths (summary)	Relevance	Notes
   8f365c934088	src/coreclr/gc/handletable.inl	Relevant	GC handle assignment — directly in CoreCLR VM
   e9914e44b440	src/libraries/System.Private.CoreLib/src/System/SpanHelpers.Packed.cs, String.Manipulation.cs	Shared	BCL string.Split optimization; touches SpanHelpers but for string-specific SIMD packing
   ff279bafa8ca	src/coreclr/tools/aot/, src/tests/Interop/	LessLikely	NativeAOT test enablement; no runtime code changes
   2210dd5a5133	eng/build.sh, docs/	LessLikely	Build script cleanup, no runtime impact

   Confirmed Regressions

   System.Memory.Memory<Char>.Pin(Size: 512) (+9.22%)

   • Delta: 20.90 ns → 22.83 ns (+9.22%)
   • Historical Context: μ=20.602 (σ=0.704, n=299); z-score=3.16; adaptive threshold=6.83%
   • Assessment: Step change — sustained elevation from ~20.6 ns to ~22.4–22.8 ns across 45 post-change runs
   • Finding Confidence: 4/5 - Excellent historical data, clear step pattern, low CV (3.4%)
   • Regression Confidence: 4/5 - z=3.16 exceeds threshold, direct GC handle code path, only 4 commits in range, precise temporal alignment
   • Likely Cause: 8f365c934088 - "Publish GCHandle assignments with release semantics (Publish GCHandle assignments with release semantics #130213)"
     • Affected Areas: GC / Memory pinning / GCHandle
     • Stack Relevance: Relevant
     • Rationale: This commit replaces plain pointer stores with VolatileStore (release semantics) in HndAssignHandle and HndAssignHandleGC in src/coreclr/gc/handletable.inl. The Memory<T>.Pin() benchmark creates a pinned GCHandle on each iteration. While x64 has strong store ordering (no hardware fence needed), VolatileStore prevents compiler reordering/optimization of the store, which can add measurable overhead in tight microbenchmark loops. The commit timestamp (2026-07-05T16:58:39) exactly matches the regression date. Both Char and Byte variants show the same pattern, confirming a shared code path (GC handle assignment).
     • Candidates Considered:
       • 8f365c934088 - Adds VolatileStore to GCHandle assignment; directly on the pin path
       • e9914e44b440 - String.Split optimization; changes SpanHelpers.Packed visibility (private→internal) but not logic; unlikely to affect Memory.Pin
       • ff279bafa8ca - NativeAOT test enablement; no runtime behavioral changes
   • Verification Status: Not verified

   System.Memory.Memory<Byte>.Pin(Size: 512) (+6.07%) — Borderline

   • Delta: 20.65 ns → 21.91 ns (+6.07%)
   • Historical Context: μ=20.306 (σ=0.681, n=297); z-score=2.35; adaptive threshold=6.71%
   • Assessment: Step change — sustained elevation mirroring the Char variant. Delta (6.07%) is just below adaptive threshold (6.71%) but z > 2 indicates statistical significance. Borderline classification due to conflicting threshold signals.
   • Finding Confidence: 4/5 - Same excellent data quality as Char variant
   • Regression Confidence: 3/5 - z=2.35 crosses significance but delta below adaptive threshold; same likely cause as Char variant but weaker signal
   • Likely Cause: 8f365c934088 - "Publish GCHandle assignments with release semantics (Publish GCHandle assignments with release semantics #130213)"
     • Affected Areas: GC / Memory pinning / GCHandle
     • Stack Relevance: Relevant
     • Rationale: Same root cause as Char variant — both tests exercise Memory<T>.Pin() which allocates/assigns a pinned GCHandle. The slightly smaller delta for Byte vs Char is likely measurement variance rather than a different code path (both use the same GCHandle infrastructure).
   • Verification Status: Not verified

   Benchmark Source

   Memory.cs#L27-L34:

   [Benchmark]
   public void Pin()
   {
       using (var pinned = _memory.Pin())
       {
           Consume(in pinned);
       }
   }

   The benchmark pins a Memory<T> of size 512 on each invocation, which internally creates a GCHandle of type Pinned. The VolatileStore change in HndAssignHandle is hit on every iteration.

   Open Questions

   • Confidence Limitations: The Byte variant is borderline — additional post-change data may resolve whether it crosses the adaptive threshold definitively.
   • Verification Needed: RunWithBuildAtHash on commits 8f365c9 (compare) vs 2210dd5 (baseline for that commit) would confirm the GCHandle change as the source of the ~2 ns overhead.
   • Design Question: The VolatileStore change fixes a correctness bug on ARM64 (weak ordering). On x64 the hardware overhead is zero, but the compiler barrier may be preventing optimizations. This may be an acceptable tradeoff (correctness > micro-perf), but the team should assess whether the x64 codegen can be tuned.

   Recommended Actions

   1. Verify attribution: Run benchmarks at commits 2210dd5a5133 and 8f365c934088 using RunWithBuildAtHash to confirm the VolatileStore change causes the regression.
   2. Assess tradeoff: This is a correctness fix for ARM64 race conditions in GCHandle publication. The x64 overhead (~2 ns per pin/unpin) may be acceptable. Discuss with GC team whether an #ifdef or platform-specific volatile can reduce x64 impact while preserving ARM64 correctness.
   3. Monitor Byte variant: Re-evaluate Memory<Byte>.Pin classification after 10+ additional runs to confirm sustained elevation.

   Bisection Results

   Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
   Pin(Size: 512)	✅ Culprit found	8f365c934088	✅ RELATED	20.9→22.8 ns (+9%)	20.9→21.8 ns (+4%)	765s
   Pin(Size: 512)	✅ Culprit found	8f365c934088	✅ RELATED	20.7→21.9 ns (+6%)	20.4→21.0 ns (+3%)	6798s

   Total bisection time: 7563s
   Success rate: 2/2 tests

   System.Memory.Memory.Pin(Size: 512)

   • Culprit: 8f365c934088
   • Commit Message: Publish GCHandle assignments with release semantics (Publish GCHandle assignments with release semantics #130213)
   • Relevance: RELATED
   • Explanation: This commit changes HndAssignHandle in handletable.inl to use VolatileStore instead of a plain store when assigning GC handle slots. Memory<Char>.Pin() creates a GCHandle internally, so this change directly affects the hot path of the benchmark. The VolatileStore adds a compiler barrier that prevents reordering optimizations, plausibly adding ~0.8ns overhead per Pin operation.

   Helix jobs

   Job ID	Submitted by bisection	Last status
   736dc4cc-8e32-4f29-a42e-60afe8300723	✅	submitted
   f2348508-cc17-4ad2-8605-63a06d675b24	✅	submitted

   Per-commit measurements

   Commit	Position	Measurements
   58dfec78ed8d	baseline	20.920 ns
   e9914e44b440	commit 1	20.920, 20.920 ns
   8f365c934088	culprit	21.820, 21.700 ns
   ff279bafa8ca	head	21.740 ns

   Bisection details

   …[earlier output omitted]

   ● Parse confirmation results (shell)
   │ cat /tmp/1784020884292-copilot-tool-output-hwd1i8.txt | python3 -c "
   │ import json, sys
   │ data = json.load(sys.stdin)
   │ report = json.loads(data['report'])
   │ benchmarks = report.get('Benchmarks', [])…
   └ 14 lines…

   Confirmation run is consistent:

   • e9914e4: 20.92 ns (same as first run)
   • 8f365c9: 21.70 ns (first run was 21.82)

   Both runs consistently show the step change at commit 8f365c9. The culprit is clear.

   BISECTION RESULT:
   - Culprit Commit: 8f365c934088b8ae21a4dfce8073da0d3eb22a54
   - Commit Message: Publish GCHandle assignments with release semantics (#130213)
   - Commits Tested: 5
   - Confidence: 3
   - Commit Relevance: RELATED
   - Relevance Explanation: This commit changes `HndAssignHandle` in `handletable.inl` to use `VolatileStore` instead of a plain store when assigning GC handle slots. `Memory<Char>.Pin()` creates a GCHandle internally, so this change directly affects the hot path of the benchmark. The VolatileStore adds a compiler barrier that prevents reordering optimizations, plausibly adding ~0.8ns overhead per Pin operation.
   - Baseline Value: 20.92 ns
   - Culprit Value: 21.82 ns
   - Summary: Bisected a 4-commit range between 58dfec78 and ff279baf. Two independent Helix runs consistently show performance stepping from ~20.92ns to ~21.7-21.8ns at commit 8f365c93. The measured delta (~4%) is smaller than the reported 9.2% but consistent across runs. Confidence capped at 3/5 because the single-sample delta (0.9ns) is within the 2σ noise floor (1.408ns), though two-sample consistency increases real-world confidence.
   
   HELIX JOBS:
   - 736dc4cc-8e32-4f29-a42e-60afe8300723
   - f2348508-cc17-4ad2-8605-63a06d675b24
   
   MEASUREMENTS:
   - 58dfec78ed8d (baseline): 20.92 ns
   - e9914e44b440 (commit 1): 20.92 ns
   - 8f365c93408 (culprit): 21.82 ns
   - ff279bafa8ca (head): 21.74 ns
   - e9914e44b440 (confirmation run): 20.92 ns
   - 8f365c93408 (confirmation run): 21.70 ns
   

   System.Memory.Memory.Pin(Size: 512)

   • Culprit: 8f365c934088
   • Commit Message: Publish GCHandle assignments with release semantics (Publish GCHandle assignments with release semantics #130213)
   • Relevance: RELATED
   • Explanation: This commit changes HndAssignHandle in src/coreclr/gc/handletable.inl to use VolatileStore instead of a plain pointer store. Memory<T>.Pin() allocates a pinned GCHandle via GCHandle.Alloc, which calls HndAssignHandle on the hot path. The VolatileStore adds a compiler barrier on x64 that prevents instruction reordering optimizations, directly impacting the benchmark's measured operation.

   Helix jobs

   Job ID	Submitted by bisection	Last status
   adeaa63d-0ba4-43c5-bf6b-2c98f92fa303	✅	submitted
   ab9d46f3-4222-4210-9c5e-13293380c46d	✅	completed
   7cc7114c-4a05-46e3-b7e0-07571e721e74	✅	submitted

   Per-commit measurements

   Commit	Position	Measurements
   58dfec78ed8d	baseline	20.530 ns
   8f365c934088	job1	21.135, 20.932 ns
   e9914e44b440	job1	20.543, 20.357 ns
   ff279bafa8ca	job1	21.158 ns
   (culprit)	culprit	21.030 ns

   Bisection details

   …[earlier output omitted]

   • Run 1: e9914e4 → 20.54 ns, 8f365c9 → 21.14 ns
   • Run 2: e9914e4 → 20.36 ns, 8f365c9 → 20.93 ns

   Consistent ~0.6 ns regression at commit 8f365c934088. Now let me get the commit details for the final report.

   ● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "8f365c93…
   └ {"index":null,"sha":"8f365c934088b8ae21a4dfce8073da0d3eb22a54","abbrevSha":"8...

   I have all the data needed. Here's the bisection result:

   BISECTION RESULT:
   - Culprit Commit: 8f365c934088b8ae21a4dfce8073da0d3eb22a54
   - Commit Message: Publish GCHandle assignments with release semantics (#130213)
   - Commits Tested: 5
   - Confidence: 3/5
   - Commit Relevance: RELATED
   - Relevance Explanation: This commit changes `HndAssignHandle` in `src/coreclr/gc/handletable.inl` to use `VolatileStore` instead of a plain pointer store. `Memory<T>.Pin()` allocates a pinned GCHandle via `GCHandle.Alloc`, which calls `HndAssignHandle` on the hot path. The VolatileStore adds a compiler barrier on x64 that prevents instruction reordering optimizations, directly impacting the benchmark's measured operation.
   - Baseline Value: 20.45 ns
   - Culprit Value: 21.03 ns
   - Summary: Bisected a 4-commit range with two independent Helix jobs confirming a consistent ~0.6 ns step change at commit 8f365c93. The delta is within the historical 2σ noise floor (1.36 ns) but reproduced consistently across both runs, and the commit directly modifies the GCHandle assignment code path exercised by Memory<T>.Pin(). Confidence capped at 3/5 due to noisy single-sample measurements relative to the historical variance.
   
   HELIX JOBS:
   - adeaa63d-0ba4-43c5-bf6b-2c98f92fa303
   - ab9d46f3-4222-4210-9c5e-13293380c46d
   - 7cc7114c-4a05-46e3-b7e0-07571e721e74
   
   MEASUREMENTS:
   - 58dfec78ed8d (baseline): 20.530 ns
   - e9914e44b440 (job1): 20.543 ns
   - e9914e44b440 (job2): 20.357 ns
   - 8f365c934088 (job1): 21.135 ns
   - 8f365c934088 (job2): 20.932 ns
   - ff279bafa8ca (job1): 21.158 ns
   

   🤖 Proposed Next Actions

   Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

   Verified regression attributed to commit 8f365c9 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

   #	Action	Status	Details
   1	Back-reference cause PR	⏭️ Skipped (dry-run)	on culprit PR for 8f365c934088
   2	Transfer issue	⏭️ Skipped (dry-run)	→ dotnet/runtime
   3	Ping owners	⏭️ Skipped (dry-run)	no owners resolved

   🦆 Pre-bisection Re-analysis (advisory)

   ✅ Agrees with the pre-bisection plan — advisory only, not applied

   Rationale: Both findings share the same likely commit (8f365c9), have relevant stack traces, and reasonably high change confidence (4/5 and 3/5). The candidate set is tight—two closely related Pin benchmarks differing only in type parameter—making bisection efficient and unlikely to waste slots on noise.

   This critique is advisory in the current configuration and did not change which tests were bisected.

   🦆 Re-analysis (advisory)

   ⚠️ Disputes the analysis — advisory only, not applied

   Flags: overconfident-aggregate

   Rationale: Bisection converged on the same commit for both findings with no alignment warnings, which strongly supports the culprit choice. However, the aggregate confidence scores (5/5 for both) exceed the per-finding scores (finding: 4/4, change: 4/3). The aggregate should not exceed its constituents — adjusting both to 4/5 to match the strongest individual signal.

   Suggested confidence: Finding Confidence → 4/5; Change Confidence → 4/5 (advisory — not applied)

   This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

   ---

   Generated by PerfTriageAgent on 2026-07-14 11:15 UTC
3. LoopedBard3 commented on Jul 14, 2026

   

   LoopedBard3

   on Jul 14, 2026

   Last edited by LoopedBard3

   Member

   More actions

   Seems like this may be due to #130213. This is clearer on the Byte pin test.
4. LoopedBard3

   transferred this issue fromdotnet/perf-autofiling-issueson Jul 14, 2026
5. github-actions

   added

   area-System.Memory

   on Jul 14, 2026
6. dotnet-policy-service commented on Jul 14, 2026

   

   dotnet-policy-serviceBot

   on Jul 14, 2026 – with Dotnet Policy Service

   Contributor

   More actions

   Tagging subscribers to this area: @dotnet/area-system-memory
   See info in area-owners.md if you want to be subscribed.
7. LoopedBard3

   added

   tenet-performancePerformance related issuePerformance related issue

   tenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark

   on Jul 14, 2026
8. adamsitnik commented on Jul 15, 2026

   

   adamsitnik

   on Jul 15, 2026

   Member

   More actions

   Seems like this may be due to #130213. This is clearer on the Byte pin test.

   This particular PR is a bug fix, so I would say it's by design.

   Leaving it up to @jkotas who reviewed the PR to make the call, as I don't know whether this arm64 fix could regress x64.
9. jkotas commented on Jul 15, 2026

   

   jkotas

   on Jul 15, 2026

   Member

   More actions

   The change regressed code produced by clang on x64:

   Before the change:

      0x7ffff7596e28:      test   %rbx,%rbx // if (value)
      0x7ffff7596e2b:      je     0x7ffff7596e38
      0x7ffff7596e2d:      mov    %r14,%rdi
      0x7ffff7596e30:      mov    %rbx,%rsi
      0x7ffff7596e33:      call   0x7ffff74e7880 // HndWriteBarrierWorker(handle, value);
      0x7ffff7596e38:      mov    %rbx,(%r14) // *(_UNCHECKED_OBJECTREF *)handle = value;
      0x7ffff7596e3b:      pop    %rbx
      0x7ffff7596e3c:      pop    %r14
      0x7ffff7596e3e:      pop    %rbp
      0x7ffff7596e3f:      ret
   

   After the change:

      0x7ffff76324b8:      test   %rbx,%rbx // if (value)
      0x7ffff76324bb:      je     0x7ffff76324ca
      0x7ffff76324bd:      mov    %r14,%rdi
      0x7ffff76324c0:      mov    %rbx,%rsi
      0x7ffff76324c3:      call   0x7ffff76295a0 // HndWriteBarrierWorker(handle, value);
      0x7ffff76324c8:      jmp    0x7ffff76324ca <---------- useless jump
      0x7ffff76324ca:      mov    %rbx,(%r14) // VolatileStore((_UNCHECKED_OBJECTREF *)handle, value);
      0x7ffff76324cd:      pop    %rbx
      0x7ffff76324ce:      pop    %r14
      0x7ffff76324d0:      pop    %rbp
      0x7ffff76324d1:      ret
   

   We should be able to tweak the sources to workaround this clang issue.

   👍React with 👍1Reacted by Adam Sitnik
10. jkotas

    added

    area-GC-coreclr

    on Jul 15, 2026
11. github-project-automation

    added this to AppModelon Jul 15, 2026
12. jkotas

    removed

    area-System.Memory

    on Jul 15, 2026
13. jkotas

    self-assigned this

    on Jul 15, 2026
14. dotnet-policy-service commented on Jul 15, 2026

    

    dotnet-policy-serviceBot

    on Jul 15, 2026 – with Dotnet Policy Service

    Contributor

    More actions

    Tagging subscribers to this area: @anicka-net, @dotnet/gc
    See info in area-owners.md if you want to be subscribed.
15. janvorli

    removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on Jul 23, 2026
16. janvorli

    added this to the 11.0.0 milestone on Jul 23, 2026
17. BradBarnich commented on Jul 27, 2026

    

    BradBarnich

    on Jul 27, 2026

    Contributor

    More actions

    @jkotas what do you think about #define VOLATILE_MEMORY_BARRIER() __atomic_signal_fence(__ATOMIC_SEQ_CST) for x64 and clang?

    From what I understand, the current value asm volatile ("" : : : "memory") creates an optimization block under clang that emits this extra jmp. Using __atomic_store like arm64 does (instead full barrier macro) also emitted extra instructions.

    some robot notes: https://gist.github.com/BradBarnich/e9b14acca0719a1d063e289da5a1f605#file-x64-volatile-md
18. janvorli commented on Jul 27, 2026

    

    janvorli

    on Jul 27, 2026

    Member

    More actions

    I've spent some time today on it and came up with the same conclusion as @BradBarnich. I've enabled llvm output using "-emit-llvm -S " options and then analyzed the output using copilot.
19. janvorli

    added a commit that references this issue on Aug 4, 2026

    Change VOLATILE_MEMORY_BARRIER for x64 Unix (#131807)

    ...

    This issue will close once commit 18f89a4 is merged into the 'main' branch.

    18f89a4
20. janvorli

    closed this as completedin #131807on Aug 4, 2026
21. github-actions

    locked and limited conversation to collaborators on Sep 4, 2026