Skip to content

Use lock-free reads for ETW rundown versions - #132757

Merged
tommcdon merged 3 commits into
dotnet:mainfrom
tommcdon:dev/tommcdon/etw-rundown-fix
Sep 1, 2026
Merged

tommcdon merged 3 commits into
dotnet:mainfrom
tommcdon:dev/tommcdon/etw-rundown-fix

Conversation

@tommcdon

@tommcdon tommcdon commented Aug 25, 2026 •

Copy link
Copy Markdown
Member

The current ETW rundown code holds a CodeVersionManager lock across the full JIT method enumeration and ETW event emission. This can block JIT paths and delay code execution.

This code change removes rundown-wide CodeVersionManager lock and instead relies on lock-free reads of published native-code version state. It builds on PR #102298 and PR #107152, which established published version state and lock-free read-only version lookups. It is noteworthy that a concurrent ReJIT or tiering update may cause rundown to skip over a newly inserted node or skip a record when code-address validation fails (if it is in the middle of an update). The benefit of a CodeVersionManager lock-free rundown likely outweighs guaranteeing a globally atomic rundown snapshot.

Co-authored-by: Copilot [email protected]

Read published native code version state with acquire semantics and remove the rundown-wide CodeVersionManager lock. Raced or missing rundown records remain droppable while JIT helper resolution can proceed.

Co-authored-by: Copilot <[email protected]>

Copilot-Session: 08e6544d-6999-41a6-85b2-cbd9ea6fe071
@tommcdon tommcdon added this to the 12.0.0 milestone Aug 25, 2026
@tommcdon tommcdon self-assigned this Aug 25, 2026
Copilot AI lite review requested due to automatic review settings August 25, 2026 18:20
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces contention in CoreCLR ETW JIT method rundown by removing the CodeVersionManager lock that was held across full method enumeration and event emission, and by strengthening a couple of read paths to use lock-free published state.

Changes:

  • Remove the CodeVersionManager lock holder (and related debug assertion) from ETW JIT method enumeration/rundown.
  • Make NativeCodeVersionNode::GetNativeCode() and MethodDescVersioningState::GetFirstVersionNode() use VolatileLoad to support safe lock-free reads of published state.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/coreclr/vm/eventtrace.cpp Stops taking the CodeVersionManager lock during ETW JIT method rundown / event emission.
src/coreclr/vm/codeversion.cpp Uses VolatileLoad for native-code pointer and version-node head reads to better support lock-free consumers.
Suppressed comments (1)

src/coreclr/vm/eventtrace.cpp:5066

  • The comment inside the nativeCodeVersion.IsNull() path is now misleading: with lock-free lookups it can also be a transient mismatch during concurrent tiering/ReJIT publication, not only “hasn't been updated”. Updating the comment will make the intentional tradeoff clearer to future maintainers (especially since this path can continue and skip emitting rundown for that entry).
            nativeCodeVersion = pMD->GetCodeVersionManager()->GetNativeCodeVersion(pMD, codeStart);
            if (nativeCodeVersion.IsNull())
            {
                // The code version manager hasn't been updated with the jitted code
                if (codeStart != MethodAndStartAddressToEECodeInfoPointer(pMD, (PCODE)NULL))

@davidwrighton davidwrighton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also needs to have a VolatileLoad for the load the native code slot off of the MethodDesc itself, for the synthetic NativeCodeVersionNode scenario. Something like...


PCODE MethodDesc::GetNativeCodeVolatile()
{
    WRAPPER_NO_CONTRACT;
    SUPPORTS_DAC;
    _ASSERTE(!IsDefaultInterfaceMethod() || HasNativeCodeSlot());
    if (HasNativeCodeSlot())
    {
        PCODE pCode = VolatileLoad(GetAddrOfNativeCodeSlot());

#ifdef TARGET_ARM
        if (pCode != (PCODE)NULL)
            pCode |= THUMB_CODE;
#endif
        return pCode;
    }

    if (!HasStableEntryPoint() || HasPrecode())
        return (PCODE)NULL;

    return VolatileLoad(GetAddrOfSlot());
}

And then make sure that particular function is used instead of the existing MethodDesc::GetNativeCode(), OR modify MethodDesc::GetNativeCode to look like that.

@noahfalk noahfalk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modulo David's suggestion and a few inline comments, LGTM! I'm pleasantly surprised to see just how close we already were.

Comment thread src/coreclr/vm/codeversion.cpp Resolved
Comment thread src/coreclr/vm/eventtrace.cpp Resolved
Add a rundown-specific MethodDesc native-code reader and use it for synthetic version lookup and address validation without changing the global GetNativeCode behavior.

Co-authored-by: Copilot <[email protected]>

Copilot-Session: 08e6544d-6999-41a6-85b2-cbd9ea6fe071
Copilot AI review requested due to automatic review settings August 26, 2026 20:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Comment thread src/coreclr/vm/eventtrace.cpp Resolved
Comment thread src/coreclr/vm/eventtrace.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

src/coreclr/vm/codeversion.cpp:1367

  • ILCodeVersioningState::LinkILCodeVersionNode publishes m_pFirstVersionNode with VolatileStore, but GetFirstVersionNode() now uses VolatileLoadWithoutBarrier, which does not provide an acquire barrier for lock-free readers. Use VolatileLoad here to ensure readers that observe the published head also observe the initialized node contents.
PTR_ILCodeVersionNode ILCodeVersioningState::GetFirstVersionNode() const
{
    LIMITED_METHOD_DAC_CONTRACT;
    return VolatileLoadWithoutBarrier(&m_pFirstVersionNode);
}

Comment thread src/coreclr/vm/eventtrace.cpp Resolved
Comment thread src/coreclr/vm/codeversion.cpp Resolved
Preserve interpreter address mapping while reading published MethodDesc and version-list state without the rundown-wide CodeVersionManager lock.

Co-authored-by: Copilot <[email protected]>

Copilot-Session: 08e6544d-6999-41a6-85b2-cbd9ea6fe071
Copilot AI review requested due to automatic review settings August 29, 2026 00:42
@tommcdon
tommcdon force-pushed the dev/tommcdon/etw-rundown-fix branch from 2470c7f to a834ccf Compare August 29, 2026 00:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/coreclr/vm/codeversion.cpp:1292

  • m_pFirstVersionNode is published with VolatileStore (release), but this accessor uses VolatileLoadWithoutBarrier, which does not provide acquire semantics. On weak-memory architectures this can allow the iterator to observe a newly published head pointer while still seeing stale/uninitialized fields in the node. Use VolatileLoad here so lock-free reads correctly synchronize with the publish pattern.
PTR_NativeCodeVersionNode MethodDescVersioningState::GetFirstVersionNode() const
{
    LIMITED_METHOD_DAC_CONTRACT;
    return VolatileLoadWithoutBarrier(&m_pFirstVersionNode);
}

Comment thread src/coreclr/vm/codeversion.cpp Resolved
@tommcdon

Copy link
Copy Markdown
Member Author

/azp run runtime

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@tommcdon

Copy link
Copy Markdown
Member Author

@davidwrighton this is ready for your review again

@tommcdon
tommcdon merged commit 82d9412 into dotnet:main Sep 1, 2026
107 checks passed
@tommcdon
tommcdon deleted the dev/tommcdon/etw-rundown-fix branch September 1, 2026 20:39
@tommcdon

tommcdon commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

/backport to release/11.0

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Started backporting to release/11.0 (link to workflow run)

@tommcdon

tommcdon commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

/backport to release/10.0

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Started backporting to release/10.0 (link to workflow run)

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

@tommcdon backporting to release/10.0 failed, the patch most likely resulted in conflicts. Please backport manually!

git am output
$ git cherry-pick 82d9412f7b554f6ad2d4aae80a6e5438348bcfa5

Auto-merging src/coreclr/vm/codeversion.cpp
Auto-merging src/coreclr/vm/eventtrace.cpp
CONFLICT (content): Merge conflict in src/coreclr/vm/eventtrace.cpp
Auto-merging src/coreclr/vm/method.cpp
Auto-merging src/coreclr/vm/method.hpp
error: could not apply 82d9412f7b5... Use lock-free reads for ETW rundown versions (#132757)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"


$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: Use lock-free reads for ETW rundown versions
Applying: Use volatile MethodDesc reads during ETW rundown
Using index info to reconstruct a base tree...
M	src/coreclr/vm/codeversion.cpp
M	src/coreclr/vm/eventtrace.cpp
M	src/coreclr/vm/method.cpp
M	src/coreclr/vm/method.hpp
Falling back to patching base and 3-way merge...
Auto-merging src/coreclr/vm/codeversion.cpp
Auto-merging src/coreclr/vm/eventtrace.cpp
CONFLICT (content): Merge conflict in src/coreclr/vm/eventtrace.cpp
Auto-merging src/coreclr/vm/method.cpp
Auto-merging src/coreclr/vm/method.hpp
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0002 Use volatile MethodDesc reads during ETW rundown
Error: The process '/usr/bin/git' failed with exit code 128

Link to workflow output

steveisok pushed a commit that referenced this pull request Sep 2, 2026
Backport of #132757 to release/11.0

/cc @tommcdon

## Customer Impact

- [ ] Customer reported
- [x] Found internally

Reported by an internal partner team. Applications with large numbers of
JIT-compiled methods can spend several seconds performing ETW rundown
while holding the global code-versioning lock. JIT paths that need the
same lock can be delayed for the duration of rundown, blocking code
execution.

## Regression

- [ ] Yes
- [x] No

No. This fixes longstanding lock contention in the ETW rundown
implementation.

## Testing

- Validated with concurrent ReJIT, revert, tiered compilation, OSR, and
repeated CLR ETW rundown cycles.
- Validated both Release and Debug CoreCLR configurations without
assertions or access violations.
- Processed generated ETL traces programmatically with TraceEvent,
forcing payload materialization for every CLR rundown event; no
malformed payloads or decoding errors were found.

## Risk

- Low
- Diagnostics only. The primary behavioral tradeoff is that we might not
report a method that is in-progress of being JIT'd during rundown. This
is acceptable because rundown does not guarantee a globally atomic
snapshot.

Co-authored-by: Tom McDonald <[email protected]>
Copilot-Session: 08e6544d-6999-41a6-85b2-cbd9ea6fe071
tommcdon added a commit to tommcdon/runtime that referenced this pull request Sep 2, 2026
The current ETW rundown code holds a `CodeVersionManager` lock across
the full JIT method enumeration and ETW event emission. This can block
JIT paths and delay code execution.

This code change removes rundown-wide `CodeVersionManager` lock and
instead relies on lock-free reads of published native-code version
state. It builds on PR
established published version state and lock-free read-only version
lookups. It is noteworthy that a concurrent ReJIT or tiering update may
cause rundown to skip over a newly inserted node or skip a record when
code-address validation fails (if it is in the middle of an update). The
benefit of a `CodeVersionManager` lock-free rundown likely outweighs
guaranteeing a globally atomic rundown snapshot.

Co-authored-by: Copilot <[email protected]>

---------

Copilot-Session: 08e6544d-6999-41a6-85b2-cbd9ea6fe071
@dotnet-milestone-bot dotnet-milestone-bot Bot modified the milestones: 12.0.0, 12.0-preview1 Sep 2, 2026
steveisok pushed a commit that referenced this pull request Sep 3, 2026
Backport of #132757 to release/10.0

/cc @tommcdon

## Customer Impact

- [x] Customer reported
- [ ] Found internally

Reported by an internal partner team. Applications with large numbers of
JIT-compiled methods can spend several seconds performing ETW rundown
while holding the global code-versioning lock. JIT paths that need the
same lock can be delayed for the duration of rundown, blocking code
execution.

## Regression

- [ ] Yes
- [x] No

No. This fixes longstanding lock contention in the ETW rundown
implementation.

## Testing

- Validated the backport changes with concurrent ReJIT, revert, tiered
compilation, OSR, and repeated CLR ETW rundown cycles.
- Both Release and Debug stress runs completed with three observed ReJIT
compilations and a revert request
- Processed the generated Release and Debug ETL traces with TraceEvent,
forcing rundown events; no malformed payloads or decoding errors were
found.

## Risk

- Low
- Diagnostics only. The primary behavioral tradeoff is that we might not
report a method that is in-progress of being JIT'd during rundown. This
is acceptable because rundown does not guarantee a globally atomic
snapshot.

Copilot-Session: 08e6544d-6999-41a6-85b2-cbd9ea6fe071
Phantomical added a commit to Phantomical/mono that referenced this pull request Sep 10, 2026
Private_EventControlCallback fired on EVENT_CONTROL_CODE_ENABLE_PROVIDER
for the rundown provider and always emitted the DCEnd flavour of every
event, ignoring MatchAnyKeyword entirely. PerfView's default asks for
end rundown alone, which is what made this work by luck.

etw_rundown_pass () (etw-profiler.hpp) replaces that: a pure function of
the control code, MatchAnyKeyword and which provider fired, read against
CLR-ETW-Generated.h's own CLR_RUNDOWNSTART_KEYWORD (0x40) and
CLR_RUNDOWNEND_KEYWORD (0x100) - asserted equal to that header's values
where HOST_WIN32 makes them available, so the mirrored constants cannot
drift from the generated ones silently. 0x80 is CLR_ENDENUMERATION_KEYWORD,
a different keyword on the regular provider entirely, and is not one of
these; RundownPassIgnoresStaleKeyword gates the two are not confused.
Both keywords can be live at once, and on_attach () now runs one locked
walk that emits whichever flavour(s) each object's own pass asks for,
rather than one walk per flavour.

EVENT_CONTROL_CODE_CAPTURE_STATE is the control code a consumer sends to
ask a provider to re-emit its state without changing enablement, and ETW
hands it MatchAnyKeyword the same way it hands ENABLE_PROVIDER - the
currently-enabled keyword for that provider. So the same function answers
both, and CAPTURE_STATE on the rundown provider now gets a rundown too.

DCStartInit/DCStartComplete and DCEndInit/DCEndComplete now bracket the
walk, one pair per pass actually requested, so a consumer can tell a
finished rundown from a truncated one.

The walk still runs synchronously inside the callback, against Microsoft's
own PENABLECALLBACK advice, and that decision is argued at the
mono_unity_domain_foreach_locked () call site in on_attach (): the one
lock held for the whole walk is mono_domain_unload_mutex, which has
exactly one other call site (mono_domain_try_unload ()'s call to
mono_domain_free (), appdomain.c) and so excludes only an in-flight
domain unload - not an allocation, a JIT compile, or a thread attaching.
mono_jit_info_table_foreach_internal () itself takes no lock at all; it
reads through the same hazard pointers a concurrent publish uses. That is
a narrower hold than CoreCLR's own rundown, which dotnet/runtime#132757
found holding the code-versioning lock - one every JIT compile also takes
- for seconds on a method-heavy process, so kept synchronous.

method_jit_done () passes mono_domain_get (), and a tier-1/tier-2 publish
can run on a compile-worker thread, which attaches to the root domain.
Traced the call: MonoBackend::compile_bodies () (mono/llvm/runtime/backend.cpp)
wraps the whole publish, this raise included, in a DomainScope over the
body's own domain, so mono_domain_get () is already right by the time this
runs. Documented at the call site rather than changed.

Gtest coverage for etw_rundown_pass (): neither keyword, each alone, both
together, the regular provider, CAPTURE_STATE, DISABLE_PROVIDER, and the
0x80 confusion above. All EtwProfilerPure cases, so the existing gtest
filter in CMakeLists.txt already runs them with no registration change.

Claude-Session: https://claude.ai/code/session_01CTYKyM4Ud3ogeHAtEKXXdf
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 2, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants