Skip to content

[Mono][Windows] MiniJIT crashes across unrelated runtime tests with FAST_FAIL_SET_CONTEXT_DENIED #133702

New issue
New issue
Closed
#133774
Closed
[Mono][Windows] MiniJIT crashes across unrelated runtime tests with FAST_FAIL_SET_CONTEXT_DENIED#133702
#133774
Labels
Known Build ErrorUse this to report build issues in the .NET Helix tabarea-Codegen-JIT-monoblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

Description

@mdh1418
mdh1418
opened on Sep 11, 2026

The windows-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit leg is consistently crashing across many unrelated runtime test suites. The test names differ, but the processes all terminate with 0xC0000409; analysis of a collected dump identifies subcode 0x30, FAST_FAIL_SET_CONTEXT_DENIED.

This reproduced in rolling builds and is unrelated to the change in #133612. It appears that a Windows environment or security-policy change may have exposed a latent problem in Mono's generated-code unwind or context handling, but the exact trigger has not been established.

Affected builds

No src/mono product changes landed between the last observed passing build and the first affected build. Relevant intervening test changes also passed in their PR build, which makes a source regression in those tests unlikely.

Failure signature

Representative work-item logs, including GC and Regression_3, terminate with:

Exit Code: -1073740791
Expected: 100
Actual: -1073740791
END EXECUTION - FAILED

Analysis of corerun.exe.15456.dmp uploaded by the GC work item shows:

ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
Subcode: 0x30 FAST_FAIL_SET_CONTEXT_DENIED
PROCESS_NAME: corerun.exe
FAILURE_BUCKET_ID: FAIL_FAST_SET_CONTEXT_DENIED_c0000409_coreclr.dll!Unknown

The relevant stack is:

ntdll!RcContinueExit
ntdll!KiUserExceptionDispatch
<Mono JIT-generated code>
coreclr!mono_jit_search_all_backends_for_jit_info+0x193d
coreclr!mono_object_get_virtual_method+0x2ef
coreclr!mono_runtime_exec_managed_code+0x340
coreclr!mono_jit_exec+0x5c3
coreclr!mono_main+0x2da3
coreclr!monovm_execute_assembly+0xcf
corerun

Scope

The crash affects at least 46 work items at the process level, spanning GC, Loader, JIT, SIMD, exception, type-system, and regression test suites. Four additional work items continued after child processes failed with the same status and then published individual failed test results. The failures occurred on multiple Helix machines within approximately one minute of starting, while other work items passed, so this is not a single defective host or a timeout.

Suggested investigation

  1. Compare process mitigation, CET, and machine-image policy changes between the passing Helix job 7e3b9d05-ed46-4c39-8c22-b20461fefac0 and the first affected job 5da76109-d12c-405b-bb8f-9500d8d5896d.
  2. Inspect Mono's Windows generated-code unwind metadata and context restoration around mono_jit_search_all_backends_for_jit_info.
  3. Compare the current Mono path with the validation and context-handling changes made in Avoid reporting incorrect thread context to Watson #117522.
  4. Preserve and inspect the GC work item's corerun.exe.15456.dmp.

Known Issue Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

Activity

  1. mdh1418
    added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    area-Codegen-JIT-mono
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    on Sep 11, 2026
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on Sep 11, 2026

  3. dotnet-policy-service commented on Sep 11, 2026

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Sep 11, 2026 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @steveisok, @vitek-karas
    See info in area-owners.md if you want to be subscribed.

  4. lewing commented on Sep 11, 2026

    @lewing
    lewing
    on Sep 11, 2026
    Member

    Follow-up: onset, independent dump evidence, and resolution priorities

    A survey of 13 selected main runtime builds from September 4-11, targeted September 3 boundary checks, and the latest two runtime-extra-platforms builds separates this new Windows failure wave from earlier Mono regression-test failures.

    Onset and repeatability

    Times below are UTC queue times. Counts refer to Helix work items, not individual tests.

    Main build Queued Source SHA Windows MiniJIT result
    1589719 Sep 9, 20:00 bd707a42fb3b 81/81 passed
    1590497 Sep 10, 08:00 c307f70df227 46/82 process-level failures
    1591508 Sep 10, 20:00 9c868a69706d Same 46 process-level failures
    1592254 Sep 11, 08:00 abc1677630a1 Same 46, plus four groups publishing child-test failures

    The three failed-work-item sets are identical. The latest run's 46 process failures occurred on 46 different machines; the first affected run used 30 distinct failing machines. This is not a single defective host.

    The four additional groups in the latest run are baseservices-exceptions, Exceptions, JIT_others, and Regression_2. Their parent commands returned success, but they published failed child-test results. Successful submission jobs and zero parent exit codes must not be used as evidence that these tests passed.

    Small representative case and independent dump evidence

    JIT.CodeGenBringUpTests_d gives a useful passing/failing comparison:

    • Last-good console: machine a0003NL; div2_d.dll passed September 9 at 22:18:19 UTC with exit code 100.
    • First-bad console: machine a0003T3; div2_d.dll terminated September 10 at 09:09:58 UTC with -1073740791.
    • Latest console: machine a000541, queue windows.amd64.open.rt; passes ArrayExc_d, then terminates at div2_d.

    All report the same coarse OS string, Windows-11-10.0.26100-SP0. That does not establish identical servicing levels or mitigation policies.

    Independent parsing of the exception and module streams in the latest work item's corerun.exe.10496.dmp found:

    Exception code:        0xC0000409
Exception parameter 0: 0x30 (FAST_FAIL_SET_CONTEXT_DENIED)
Exception address:     0x7ffac95bf1e6 (ntdll.dll + 0x11f1e6)
ntdll.dll version:     10.0.26100.8972
KERNELBASE.dll version: 10.0.26100.9168

    This independently corroborates the subtype reported in the issue's different GC dump. It does not constitute a fully unwound native stack; native Windows debugging was not performed on the macOS investigation host. The generic "stack buffer overrun" description is not evidence of a buffer overwrite here.

    The baseservices-exceptions console also distinguishes ordinary managed exceptions from the hardware-exception path: the ordinary case passes, while mainhardware reports Wrong exit code: 0xC0000409, expected 0x00000001.

    Trigger: evidence versus hypothesis

    The last-good/first-bad source interval contains 12 commits, with no changes to src/mono, src/coreclr/hosts/corerun, global.json, or eng/Version.Details.xml.

    Mono's Windows x64 vectored exception handling in src/mono/mono/mini/exceptions-amd64.c handles hardware faults, modifies/restores the context, and returns EXCEPTION_CONTINUE_EXECUTION. Together with the dump subtype and repeatability, this makes a Windows servicing, image, or mitigation-policy change exposing a Mono compatibility problem the leading hypothesis.

    The actual triggering change is not yet established. These observations do not prove CET specifically, an image regression specifically, or a particular runtime PR.

    One refinement to the issue's suggested investigation: #117522 concerns CoreCLR Watson fatal-error context reporting. It may provide context, but it is not an established fix for Mono's normal hardware-exception continuation path.

    Earlier Mono failures are separate

    Recommended resolution path

    1. Compare last-good job 7e3b9d05-ed46-4c39-8c22-b20461fefac0 and first-bad job 5da76109-d12c-405b-bb8f-9500d8d5896d with Windows/Helix image owners: full OS servicing/UBR, native DLL versions, image deployment history, process context-IP validation, and CET/hardware stack-protection policy.
    2. Use div2_d and the hardware-exception case as small native-debugger targets. Inspect Mono's continuation targets, context restoration, and generated-code unwind metadata against Windows requirements.
    3. If an environment change is proved responsible, consider an approved known-good image/pool as temporary mitigation while fixing compatibility. Do not broadly weaken security settings or quarantine dozens of unrelated test groups as the permanent remedy.
    4. Preserve the existing narrow quarantines for the older Mono bugs. Fix the new socket-test portability issue separately: it catches the expected reset around ReceiveAsync, but AcceptAsync is outside that catch; expected reset handling must include that boundary while still requiring the healthy follow-up connection to succeed.

    No CI reruns or source modifications were made during this investigation. The immediate blocker has a tight onset window and a specific failure mechanism; identifying the environment/Mono interaction remains the next causal step.

    Note

    This analysis was generated with GitHub Copilot.

  5. steveisok commented on Sep 11, 2026

    @steveisok
    steveisok
    on Sep 11, 2026
    Member

    Any reason why we can't remove the windows run?

  6. lewing
    added a commit that references this issue on Sep 12, 2026

    Temporarily disable Windows x64 Mono MiniJIT runtime tests (#133747)

    77cc249
  7. tannergooding
    closed this as completedin #133774on Sep 12, 2026
  8. tannergooding
    added a commit that references this issue on Sep 12, 2026

    Fix Mono Windows x64 hardware exception continuations (#133774)

    37dd42c
  9. dotnet-policy-service
    removed
    untriagedNew issue has not been triaged by the area owner
    on Sep 12, 2026
  10. jtschuster
    added 2 commits that reference this issue on Sep 18, 2026

    Temporarily disable Windows x64 Mono MiniJIT runtime tests (dotnet#13…

    2c2ccff

    Fix Mono Windows x64 hardware exception continuations (dotnet#133774)

    42fd0cc
  11. github-actions
    locked and limited conversation to collaborators on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Known Build ErrorUse this to report build issues in the .NET Helix tabarea-Codegen-JIT-monoblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions