Skip to content

Fix YieldProcessorMeasurement running continuously due to timestamp truncation#122293

Merged
agocke merged 3 commits intomainfrom
copilot/fix-high-cpu-usage
Dec 8, 2025
Merged

Fix YieldProcessorMeasurement running continuously due to timestamp truncation#122293
agocke merged 3 commits intomainfrom
copilot/fix-high-cpu-usage

Conversation

Copy link
Contributor

Copilot AI commented Dec 8, 2025 •
edited
Loading

  • Understand the issue: High CPU due to excessive YieldProcessorMeasurement
  • Identify the bug: Extra (unsigned int) cast at line 206 in yieldprocessornormalizedshared.cpp
  • Remove the incorrect cast to fix time comparison
  • Build and test the fix
  • Run code review
  • Run security scan
  • Add parentheses to clarify operator precedence per review feedback

Summary

Fixed high CPU utilization in .NET 10.0 by removing an incorrect (unsigned int) cast that was causing YieldProcessorMeasurement to run continuously instead of every 4 seconds. The variable s_previousNormalizationTimeMs is declared as int64_t but was being assigned with a truncated 32-bit value, leading to incorrect time comparisons and excessive measurements.

Added parentheses around subtraction operations to make operator precedence explicit.

Original prompt

This section details on the original issue you should resolve

<issue_title>High CPU utilization after update from net8 to net10</issue_title>
<issue_description>

Description

After updating asp net core application from net 8 to net10 noticed significantly increased CPU consumption. What is the reason for this?

Image Image Image

After the update, we noticed a significantly increased number of garbage collections.

Image

Configuration

Image: mcr.microsoft.com/dotnet/aspnet:10.0.0
OS: Ubuntu 24.04
Runtime: net10.0.0
Architecture: x64

ServerGarbageCollection: true
ConcurrentGarbageCollection: true
GarbageCollectionAdaptationMode: Not specified
</issue_description>

<agent_instructions>Delete the extra (unsigned) cast at https://github.com/dotnet/runtime/pull/115408/files#diff-1c8c26a1e7f6ab4505c185d7ba09c8b95dbac6a5da98d424bef94520bf94cdaeR206 that leads to this bug </agent_instructions>

Comments on the Issue (you are @copilot in this section)

@agocke @jestonedev We would need a before/after trace to be sure. A CPU graph alone doesn't give enough info to say which piece is spending more CPU. Setting GCDynamicAdaptationMode to 0 would also help as an easy way to determine if that's the cause. @jkotas The YieldProcessorMeasurement counter looks way too high. YieldProcessorMeasurement is done on finalizer thread, so the high finalizerstart/stop is a side-effect of that.

There is something wrong with the scheduling of YieldProcessorMeasurement</comment_new>
<comment_new>@jkotas

What is YieldProcessorMeasurement

The performance characteristics of processor yield instruction (pause on Intel) varies a lot between processors. It can even change while the process is running due to VM migrations in cloud. YieldProcessorMeasurement is expected to be done occasionally (every 4 seconds at most) by the runtime to measure the current performance characteristics of this instruction and retune the spinwaiting loops as needed.</comment_new>
<comment_new>@jkotas
Looks like a regression inroduced by https://github.com/dotnet/runtime/pull/115408/files#diff-1c8c26a1e7f6ab4505c185d7ba09c8b95dbac6a5da98d424bef94520bf94cdaeR206</comment_new>

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI changed the title [WIP] Fix high CPU utilization after update from net8 to net10 Fix YieldProcessorMeasurement running continuously due to timestamp truncation Dec 8, 2025
Copilot AI requested a review from jkotas December 8, 2025 15:32
@jkotas jkotas requested review from agocke and eduardo-vp December 8, 2025 15:49
agocke
agocke reviewed Dec 8, 2025
View reviewed changes
@agocke
Copy link
Member

agocke commented Dec 8, 2025

Is there any way to reasonably test this? I assume the timer would have to overflow, meaning we'd need to run the tests for ~50 days?

@jkotas
Copy link
Member

jkotas commented Dec 8, 2025

meaning we'd need to run the tests for ~50 days?

Right. Or artificially bump the timer so that it starts at more than UInt32.MaxValue.

Also, it would have to be a perf test. This fix does not affect functionality.

We have actually seen a regression that was attributed to this change #115408 (comment) . I suspect that it might have been caused by this bug. Nobody was able to repro the regression, and so it was won't fixed unfortunately.

@jkotas jkotas marked this pull request as ready for review December 8, 2025 21:30
Copilot AI review requested due to automatic review settings December 8, 2025 21:30
Copilot AI reviewed Dec 8, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a critical bug in YieldProcessorMeasurement where timestamp truncation caused continuous measurement execution instead of the intended 4-second interval, resulting in elevated CPU usage and increased GC pressure in ASP.NET Core applications on .NET 10.0.

Key Changes:

  • Removed incorrect cast from int64_t to unsigned int when assigning the return value of minipal_lowres_ticks() to s_previousNormalizationTimeMs

Copilot AI requested a review from agocke December 8, 2025 21:53
agocke
agocke approved these changes Dec 8, 2025
View reviewed changes
Copy link
Member

@agocke agocke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Also think it's appropriate for backport

@agocke
Copy link
Member

agocke commented Dec 8, 2025

/ba-g dead letter

@agocke agocke enabled auto-merge (squash) December 8, 2025 23:58
@agocke agocke merged commit 488d367 into main Dec 8, 2025
99 of 101 checks passed
@agocke agocke deleted the copilot/fix-high-cpu-usage branch December 8, 2025 23:58
@agocke
Copy link
Member

agocke commented Dec 8, 2025

/backport to release/10.0

@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2025

Started backporting to release/10.0 (link to workflow run)

@github-actions github-actions bot mentioned this pull request Dec 9, 2025
Merged
4 tasks
This was referenced Dec 9, 2025
Closed
Closed
steveisok added a commit that referenced this pull request Dec 10, 2025
@github-actions
@jkotas @agocke @steveisok
[release/10.0] Fix YieldProcessorMeasurement running continuously due…
0b4a62d 
… to timestamp truncation (#122317)

Backport of #122293 to release/10.0

/cc @agocke @Copilot

The background process responsible for periodic updates of processor
yield instruction costs consumes 100% of a single core on a machine with
50+ day uptime or non-standard OS timer configuration.

## Customer Impact

- [x] Customer reported
- [ ] Found internally

## Regression

- [x] Yes
- [ ] No

## Testing

Standard test pass, no testing specific to the fix - the bug requires
machine with 50 day uptime to test

## Risk

Low.

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: jkotas <[email protected]>
Co-authored-by: agocke <[email protected]>
Co-authored-by: Steve Pfister <[email protected]>
@maestro-local-dev maestro-local-dev bot mentioned this pull request Dec 10, 2025
Closed
@dotnet-maestro dotnet-maestro bot mentioned this pull request Dec 13, 2025
Merged
@github-actions github-actions bot locked and limited conversation to collaborators Jan 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code review Copilot Copilot left review comments

@agocke agocke agocke approved these changes

@eduardo-vp eduardo-vp Awaiting requested review from eduardo-vp

+1 more reviewer

@jkotas jkotas jkotas approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@jkotas jkotas

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

High CPU utilization after update from net8 to net10

4 participants

@agocke @jkotas