Server GC heap initialization fails with 0x8013200A ("affinitized to 0 CPUs") on legacy OpenVZ hosts due to mismatched CPU affinity reporting #133449

Activity

1. dotnet-policy-service

   added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Sep 8, 2026
2. github-actions

   added

   area-GC-coreclr

   on Sep 8, 2026
3. github-project-automation

   added this to AppModelon Sep 8, 2026
4. dotnet-policy-service commented on Sep 8, 2026

   

   dotnet-policy-serviceBot

   on Sep 8, 2026 – with Dotnet Policy Service

   Contributor

   More actions

   Tagging subscribers to this area: @anicka-net, @dotnet/gc
   See info in area-owners.md if you want to be subscribed.
5. lg2de commented on Sep 9, 2026

   

   lg2de

   on Sep 9, 2026

   Author

   More actions

   Root cause found: regression from #131740 (backport of #126763)

   I bisected this against the release tags. Between v10.0.11 and v10.0.12 there are 29 commits, and exactly one touches CPU/affinity code:

   1f69e1d0a23  [Release 10.0] Support hardware with more than 1024 CPUs (#131740)
   

   That PR backported #126763 + #127572 into release/10.0 on 2026-08-07, which is why 10.0.11 works and 10.0.12 does not.

   The mechanism

   The relevant part of the diff in src/coreclr/gc/unix/gcenv.unix.cpp:

   -    cpu_set_t cpuSet;
   -    int st = sched_getaffinity(getpid(), sizeof(cpu_set_t), &cpuSet);
   +    cpu_set_t* pCpuSet = CPU_ALLOC(configuredCpuCount);
   +    size_t cpuSetSize = CPU_ALLOC_SIZE(configuredCpuCount);
   +    int st = sched_getaffinity(getpid(), cpuSetSize, pCpuSet);

   Before, the mask was a fixed cpu_set_t — always 128 bytes / 1024 bits. Now it is sized from minipal_get_cpu_max_possible_count(), which reads /sys/devices/system/cpu/possible.

   The kernel rejects a mask buffer that is narrower than its own nr_cpu_ids:

   /* kernel/sched/core.c, SYSCALL_DEFINE3(sched_getaffinity, ...) */
   if ((len * BITS_PER_BYTE) < nr_cpu_ids)
           return -EINVAL;

   On this host, sysfs is virtualized per container but the syscall is not, so those two numbers disagree:

   	value
   /sys/devices/system/cpu/possible	0-3 → minipal_get_cpu_max_possible_count() = 4
   CPU_ALLOC_SIZE(4)	8 bytes = 64 bits (glibc rounds up to sizeof(unsigned long))
   /proc/self/status → Cpus_allowed_list	0-111 → kernel nr_cpu_ids = 112
   kernel check	64 < 112 → EINVAL

   And the error is swallowed in release builds (gcenv.unix.cpp#L208-L213):

   else
   {
       // We should not get any of the errors that the sched_getaffinity can return since none
       // of them applies for the current thread, so this is an unexpected kind of failure.
       assert(false);
   }

   So g_processAffinitySet stays empty, and interface.cpp#L265-L269 turns that into CLR_E_GC_BAD_AFFINITY_CONFIG = 0x8013200A.

   Additional diagnostics from the affected host

   /sys/devices/system/cpu/possible     0-3
   /sys/devices/system/cpu/present      0-3
   /sys/devices/system/cpu/online       0-3
   /sys/devices/system/cpu/kernel_max   5119
   stat -fc %T /sys                     sysfs
   getconf _NPROCESSORS_CONF            4
   getconf _NPROCESSORS_ONLN            4
   Cpus_allowed_list                    0-111
   

   The host kernel is built with NR_CPUS=5120 and runs with nr_cpu_ids=112, while the container's sysfs consistently reports 4 CPUs.

   Correcting my own original report: cgroups are not involved at all. /sys itself is a real sysfs and /sys/devices/system/cpu/possible exists and parses fine — it just reports the container's view. The tmpfs /sys/fs/cgroup I mentioned is a red herring; minipal_get_cpu_max_possible_count() never looks at cgroups.

   Why only Server GC is affected

   The same PR added a graceful fallback everywhere except in the GC:

   file	on sched_getaffinity failure
   pal/src/misc/sysinfo.cpp	if (nrcpus < 1) nrcpus = PAL_GetTotalCpuCount();
   nativeaot/Runtime/unix/PalUnix.cpp	if (count == 0) count = GetTotalProcessorCount();
   pal/src/thread/thread.cpp	"Treat failure to get the affinity mask in release build as non-fatal"
   gc/unix/gcenv.unix.cpp	assert(false) — set stays empty

   Workstation GC never reaches the affinity check, which is exactly why it starts fine on the identical host.

   Why none of the environment variables helped

   The IsEmpty() check at interface.cpp:265 runs before everything else:

   • DOTNET_PROCESSOR_COUNT / DOTNET_GCHeapCount are only consulted from line 280 onwards.
   • DOTNET_GCNoAffinitize is only read at line 298.
   • GCHeapAffinitizeRanges cannot help either: SetGCThreadsAffinitySet() only removes CPUs from g_processAffinitySet, it never adds any — an empty set can never become non-empty.

   Worth noting that the code just below already tolerates an empty set gracefully (interface.cpp#L302-L307 explicitly guards num_affinitized_processors != 0). The hard failure at line 265 is the only thing standing in the way.

   Proposed fix

   Give the GC path the same fallback the sibling files already got in that PR:

   else
   {
       // sched_getaffinity fails with EINVAL when the CPU mask the kernel maintains is wider
       // than the one we allocated. That happens when /sys/devices/system/cpu/possible reports
       // fewer CPUs than the kernel accounts for, as in legacy OpenVZ containers where sysfs is
       // virtualized per container but the syscall is not. Any other error is unexpected here,
       // since none of them applies for the current process.
       assert(errno == EINVAL);
   
       // Leaving the affinity set empty makes server GC initialization fail outright, so fall
       // back to assuming the process can run on all configured CPUs. This matches what the
       // !HAVE_SCHED_GETAFFINITY path below does.
       for (size_t i = 0; i < (size_t)configuredCpuCount; i++)
       {
           g_processAffinitySet.Add(i);
       }
   }

   This is a no-op on healthy systems (the branch is never taken there) and yields 4 heaps on 4 CPUs here. It touches no gc_heap code, so per garbage-collector-guidelines.md I do not believe a stress run is required — please correct me if you see it differently.

   Two follow-on paths I checked, both harmless: SetThreadAffinity() allocates an equally narrow mask, but sched_setaffinity tolerates a short buffer (get_user_cpu_mask clears the mask and copies only len bytes), and even on failure set_thread_affinity_for_heap (gc.cpp:2506-2512) only emits a dprintf.

   Separately, and probably as its own PR: the genuinely correct fix is to retry sched_getaffinity with a doubled buffer while errno == EINVAL, which is the pattern glibc documents for CPU_ALLOC. That would handle any case where the kernel's mask is wider than sysfs suggests, not just this one. I kept it out of the fix above to keep the change backportable.

   I have the change ready and am happy to open a PR against main (with a /backport to release/10.0 afterwards) if that sounds right to you. Caveat: I can only reason about it statically — I have no way to build a runtime and validate it on the affected host, and the branch is unreachable on normal hardware, so I could not exercise it locally.

   👍React with 👍1Reacted by Anton Lindgren
6. janvorli

   self-assigned this

   on Sep 9, 2026
7. janvorli

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Sep 9, 2026
8. janvorli

   added this to the 11.0.0 milestone on Sep 9, 2026
9. janvorli

   added a commit that references this issue on Sep 9, 2026

   Fix cpu affinity set size to have minimal size

   ...

   This issue will close once commit 4b9d4af is merged into the 'master' branch.

   4b9d4af
10. antonlindgren commented on Sep 9, 2026

    

    antonlindgren

    on Sep 9, 2026

    More actions

    Hi!

    What a pleasant report to read, kudos @lg2de :)

    We're seeing the same issue on our self-hosted github runners. It worked yesterday with 10.0.11 but not today with 10.0.12, and setting both DOTNET_gcServer=0 and DOTNET_GCNoAffinitize=1 also did the trick. Thanks!

    Though, and excuse me if this is a dumb question, I'm wondering about the milestone flag - does that mean dotnet10 will not be getting this fix, @janvorli?

    Thanks in advance!
11. agocke

    modified the milestones: 11.0.0, 10.0.x on Sep 9, 2026
12. janvorli

    closed this as completedin #133497on Sep 11, 2026
13. janvorli

    added a commit that references this issue on Sep 11, 2026

    Fix cpu affinity set size to have minimal size (#133497)

    ...

    This issue will close once commit b44cd90 is merged into the 'main' branch.

    b44cd90
14. github-actions

    locked and limited conversation to collaborators on Oct 12, 2026