I've got a deadlock where the GC is blocked acquiring the threadstore lock, and almost all the other threads are blocked on the GC. My specific instance has 310 active threads though when the process deadlocks.

I'm using .NET Core 3.1. The program reads different JSON files for every day for the past year or so, using a different thread for each one for performance (a poor man's in-proc Big Data solution). The process usually completes in 10 minutes, but it hung for about 9 hours with no obvious forward progress. Attaching VS, most threads are in various helpers like FastAllocateString, all blocked on the GC. This behavior just showed up today, meaning the app used one more thread - maybe I passed over some thread number limit with 310 active threads?

Stack traces from my crash dump (~*k):
DataExporterThreads.txt

Most threads are blocked in WKS::gc_heap::wait_for_gc_done. Thread 93 is doing a GC, but is blocked in SuspendEE trying to get the ThreadStore lock. In a cursory analysis, I was unable to determine which thread is holding the ThreadStore lock. There were several threads in JIT_PInvokeEndRarePath calling Thread::RareDisablePreemptiveGC, which oddly calls EnablePreemptiveGC and then blocks (I never got a good explanation from Vance or Chris Brumme on that one).

I was able to work around this at least once with the following code:
ThreadPool.SetMaxThreads(200, 50);

I'm attaching the stack trace from each thread. I have a dump captured by VS, with the debugger attached as both native & managed. Happy to upload it somehow, somewhere. 400 MB compressed.

Thread 93's portion back in native code:

ntdll!NtWaitForAlertByThreadId+0x14
ntdll!RtlpWaitOnAddressWithTimeout+0x43
ntdll!RtlpWaitOnAddress+0xb2
ntdll!RtlpWaitOnCriticalSection+0xdb
ntdll!RtlpEnterCriticalSectionContended+0xcc
ntdll!RtlEnterCriticalSection+0x40
coreclr!CrstBase::Enter+0x49 [f:\workspace_work\1\s\src\vm\crst.cpp @ 319]
coreclr!ThreadStore::Enter+0x10 [f:\workspace_work\1\s\src\vm\threads.cpp @ 5255]
coreclr!ThreadSuspend::LockThreadStore+0x7d [f:\workspace_work\1\s\src\vm\threadsuspend.cpp @ 2361]
coreclr!ThreadSuspend::SuspendEE+0xf7 [f:\workspace_work\1\s\src\vm\threadsuspend.cpp @ 6443]
coreclr!GCToEEInterface::SuspendEE+0x21 [f:\workspace_work\1\s\src\vm\gcenv.ee.cpp @ 25]
coreclr!WKS::GCHeap::GarbageCollectGeneration+0xff [f:\workspace_work\1\s\src\gc\gc.cpp @ 36545]
coreclr!WKS::gc_heap::trigger_gc_for_alloc+0x12 [f:\workspace_work\1\s\src\gc\gc.cpp @ 13832]
coreclr!WKS::gc_heap::try_allocate_more_space+0x24e [f:\workspace_work\1\s\src\gc\gc.cpp @ 13934]
coreclr!WKS::gc_heap::allocate_more_space+0x11 [f:\workspace_work\1\s\src\gc\gc.cpp @ 14369]
coreclr!WKS::gc_heap::allocate+0x58 [f:\workspace_work\1\s\src\gc\gc.cpp @ 14400]
coreclr!WKS::GCHeap::Alloc+0x88 [f:\workspace_work\1\s\src\gc\gc.cpp @ 35827]
coreclr!Alloc+0x12f [f:\workspace_work\1\s\src\vm\gchelpers.cpp @ 240]
coreclr!AllocateString+0x195 [f:\workspace_work\1\s\src\vm\gchelpers.cpp @ 996]
coreclr!FramedAllocateString+0x82 [f:\workspace_work\1\s\src\vm\jithelpers.cpp @ 2884]

A set of threads were blocked on the GC mode around a P/Invoke call to ReadFile.

[0x1] KERNELBASE!WaitForSingleObjectEx + 0x8f
[0x2] coreclr!CLREventWaitHelper2 + 0x6
[0x3] coreclr!CLREventWaitHelper + 0x20
[0x4] coreclr!CLREventBase::WaitEx + 0x12
[0x5] coreclr!CLREventBase::Wait + 0x12
[0x6] coreclr!Thread::WaitSuspendEventsHelper + 0xa9
[0x7] coreclr!Thread::WaitSuspendEvents + 0x8
[0x8] coreclr!Thread::RareEnablePreemptiveGC + 0x1342ec
[0x9] coreclr!Thread::EnablePreemptiveGC + 0x116
[0xa] coreclr!Thread::RareDisablePreemptiveGC + 0x1c6
[0xb] coreclr!JIT_PInvokeEndRarePath + 0x69
[0xc] System_Private_CoreLib!Interop+Kernel32.ReadFile(System.Runtime.InteropServices.SafeHandle, Byte*, Int32, Int32 ByRef, IntPtr)$##6000044 + 0xbf

Brian (who left the BCL team 8 years ago)