Skip to content

Commit 75bdb67

Browse files
radekdoulikCopilot
authored andcommitted
[browser][coreclr] Add wasm/CoreCLR libraries tests to outerloop CI (#129436)
## Summary Adds a scheduled **outerloop** leg that runs the library test suites on **Browser + CoreCLR** (Chrome), including the slow `[OuterLoop]` tests. Previously the outerloop pipeline had no wasm/CoreCLR coverage — only CoreCLR desktop and Mono `browser_wasm`. Getting the slow suites to actually complete on single-threaded wasm required, all scoped so **innerloop behavior is unchanged**: 1. **Enable** the long-running suites and slow `[OuterLoop]` tests that are normally excluded on wasm. 2. **Extend** the xharness / Helix / AzDO timeouts and **isolate** each slow suite into its own Helix batch. 3. **Gate** the `[OuterLoop]` tests that genuinely *cannot* run on a single thread (or need a 64-bit address space) with `[Conditional*]` attributes, so they skip cleanly instead of hanging or failing. 4. **Gate** the `[OuterLoop]` tests that *can* run but are simply too slow / too memory-heavy on the single-threaded wasm interpreter with `[ActiveIssue]`, scoped to **CoreCLR-wasm only** (Mono keeps running them), tracked in #129973 for re-enablement. ## CI pipeline - **`eng/pipelines/libraries/outerloop.yml`** — new `browser_wasm` + `coreclr` job (`WasmTestOnChrome`, `Release`, `/maxcpucount:1`, `timeoutInMinutes: 600`). Gated on `includeLinuxOuterloop` (browser_wasm builds on a Linux host) so it runs in the umbrella / `-linux` outerloop definitions only. - Runs with **`-testscope all`, not `outerloop`** — this is deliberate. The `outerloop` scope adds a `-trait category=OuterLoop` *include* filter, and the wasm test host (xharness) treats a matching include as "always run", short-circuiting the `-notrait category=failing` *exclude*. That leaks `[OuterLoop]` tests out of classes disabled via `[ConditionalClass]` (e.g. on `IsMultithreadingSupported`) onto single-threaded wasm, where they block the one thread and hang the job. `-testscope all` carries no OuterLoop include, so the platform gates are honored; it also covers the slow suites' innerloop tests. ## Enabling slow suites + timeouts/batching - **`src/libraries/tests.proj`** — gate the long-running suite exclusions (`System.Linq`, `System.Memory`, `System.Private.Uri`, `System.Private.Xml`, `System.Runtime.Numerics`, `System.Text.Json`) behind a new `RunLongRunningWasmTests` property (`true` for outerloop/all). `System.Formats.Nrbf` stays unconditionally excluded (test failures, not slowness). - **`src/libraries/sendtohelix-browser.targets`** — isolate each slow suite into its own solo Helix batch (via the new `SoloItems`) so one slow suite cannot inflate a shared batch, and raise the per-batch timeout budget for outerloop. - **`src/tasks/HelixTestTasks/GroupWorkItems.cs`** — add `SoloItems`: work items named here each get their own negative-ID solo batch regardless of size. - **`src/tasks/HelixTestTasks/ComputeBatchTimeout.cs`** — parameterize the per-batch timeout (`MinutesPerSuite` / `MinimumMinutes` / `MaximumMinutes`); defaults reproduce the previous 20-min/30-min-floor/1439-cap behavior exactly, and the MSBuild-provided knobs are validated/clamped. - **`eng/testing/tests.wasm.targets`** — outerloop/all wasm CoreCLR gets a `04:00:00` per-suite xharness timeout (innerloop keeps `01:30:00` / `00:30:00`). - **`src/libraries/sendtohelixhelp.proj`** — extend the non-batched browser + CoreCLR work-item timeout to `04:30:00` for outerloop/all (safety net for non-batched paths). The timeout layers are kept ordered **xharness suite (4h) ≤ non-batched work-item / batch (4h30 floor, 5h cap) ≤ AzDO job (10h)** so a slow suite is bounded by its own timeout, and the synchronous send-to-Helix wait always fits inside the job. ## Test gates — cannot run (single-threaded / 64-bit only) `[OuterLoop]` tests that throw / hang on a single thread or need a 64-bit address space, gated at method or class level (never via `tests.proj` exclusions) so they skip cleanly: | File | Gate | |---|---| | `System.Threading.Tasks.Parallel/.../ParallelStateTest.cs` | `[ConditionalClass(IsMultithreadingSupported)]` on `ParallelState` (all 77 throw PNSE in `TaskReplicator.Run`) | | `System.Linq.Parallel/.../DegreeOfParallelismTests.cs` | 3 `DegreeOfParallelism_{Barrier,Pipelining,Throttled_Pipelining}` → `[ConditionalTheory(IsMultithreadingSupported)]` | | `System.Linq.Parallel/.../ExchangeTests.cs` | `Partitioning_{Default,Striped}_Longrunning` → bare `[ConditionalTheory]` (delegate into bodies that already `throw SkipTestException`; the bare form keeps `partitions ≤ 1` rows passing and skips the rest) | | `System.Linq.Parallel/.../CancellationParallelQueryCombinationTests.cs` | `SequenceEqual_OperationCanceledException` → `[ConditionalTheory(IsMultithreadingSupported)]` | | `System.Linq.Parallel/.../WithCancellationTests.cs` | `WithCancellation_DisposedEnumerator_...ProducerBlocked` → `[ConditionalTheory(IsMultithreadingSupported)]` | | `System.Runtime/.../System.Threading.Tasks.Tests/MethodCoverage.cs` | `TaskContinuation` → `[ConditionalFact(IsMultithreadingSupported)]` | | `System.Runtime/.../System.Threading.Tasks.Tests/.../TaskAwaiterTests.cs` | `GetResult_NotCompleted_BlocksUntilCompletion` → `[ConditionalFact(IsMultithreadingSupported)]` | | `System.Runtime/.../System.Threading.Tasks.Tests/Task/TaskAPMTest.cs` | `WaitUntilCompleteTechnique`, `PollUntilCompleteTechnique` → `[ConditionalTheory(IsMultithreadingSupported)]` | | `System.Runtime/.../System.Threading.Tasks.Tests/TaskScheduler/TaskSchedulerTests.cs` | `RunBlockedInjectionTest`, `RunSynchronizationContextTaskSchedulerTests` → `[ConditionalFact(IsMultithreadingSupported)]` | | `System.Runtime/.../System.Runtime.Tests/System/GCTests.cs` | `WaitForPendingFinalizersRaces` → add `IsMultithreadingSupported` to existing gate | ## Test gates — too slow / too memory-heavy (tracked in #129973) These `[OuterLoop]` tests *can* run but exceed the per-suite timeout (or exhaust the wasm heap) on the single-threaded interpreter. They are gated with `[ActiveIssue("#129973", typeof(PlatformDetection), nameof(IsBrowser), nameof(IsCoreCLR))]`, i.e. **disabled only on CoreCLR-wasm — Mono-wasm keeps running them**: | File | Test(s) | Reason | |---|---|---| | `System.Linq.Expressions/.../CompilerTests.cs` | `CompileDeepTree_NoStackOverflow` | compiles a 10,000-deep expression tree | | `System.Memory/.../EncodingExtensions/EncodingExtensionsTests.cs` | `GetBytes_…LargeMultiSegment`, `GetChars_…LargeMultiSegment` | iterate billions of chars/bytes | | `System.Text.Json/.../BitStackTests.cs` | `BitStackPushPopLarge` | pushes/pops up to 67M bits | | `System.Text.Json/.../Serialization/Stream.ReadTests.cs` | `ReadTypeFromJsonWithLargeIgnoredProperties_OuterLoop` | processes 5 GiB of JSON | | `System.Text.Json/.../Utf8JsonReaderTests*.cs` | `TestJsonReaderLargestUtf8SegmentSizeOne`, `TestPartialJsonReaderSlicesMultiSegment`, `TestPartialJsonReaderSlicesSpecialNumbers` | O(n²) slicing | | `System.Text.Json/.../Utf8JsonWriterTests.cs` | `Writing3MBBase64Bytes`, `EscapeCharacters`, `WriteNumbers` | every writer-option combination | | `System.Collections.Immutable/.../Frozen/FrozenDictionaryTests.cs`, `FrozenSetTests.cs` | `ToFrozen*_WithExtremelyLargeStrings` | allocate ~1 GB strings (memory exhaustion) | | `System.Private.Xml.Linq/.../xNodeBuilder/FunctionalTests.cs` | `RunTests` | wasm runtime hangs at shutdown after this suite (a hang, not slowness) | Suite-level: **`System.Linq.Parallel/tests/System.Linq.Parallel.Tests.csproj`** drops the whole `[OuterLoop]` category on the wasm/CoreCLR lane (`WithoutCategories=OuterLoop`). The combinatorial matrix expands to ~195k cases on a single thread (~72 MB results file exceeding xharness's 30 MB upload limit, ~1.3 GB heap, tab crash at teardown), and PLINQ has no real parallelism on one thread. Separately, `System.Net.Http/.../ResponseStreamTest.cs` `BrowserHttpHandler_StreamingRequest_Http1Fails` is gated with `[ActiveIssue(#129758, IsCoreCLR)]` — a real CoreCLR-wasm behavior bug (surfaces `TaskCanceledException` instead of `HttpRequestException`), not a slowness gate. ## CI validation Latest scheduled outerloop run (build `1485712`, Helix job `WasmTestOnChrome-CLR-ST`): **31 work items, all passed** (1 controller + 30 test batches — 20 grouped, 10 solo). Across all batches: | Suites | Tests run | Passed | Failed | Skipped | |---:|---:|---:|---:|---:| | 203 | 657,944 | 653,983 | **0** | 3,961 | Earlier local verification (wasm/Chrome, Release, `-testscope all`) of individual suites: System.Threading.Tasks.Parallel.Tests 242 / **0 fail** (was 77); System.Security.Cryptography 4993 / **0 fail** / 804 skip; System.Runtime.Caching and Diagnostics.Metrics clean. (System.Linq.Parallel was validated at 195,566 / 0 fail before its `[OuterLoop]` matrix was dropped on this lane, per above.) ## Notes / things to watch - The newly-enabled suites were previously excluded for **slowness** (#125495), not crashes. Extending timeouts is the intended remedy; the slow/memory tests that still couldn't fit the budget are tracked in #129973 (CoreCLR-wasm only) rather than excluded wholesale. - Timeout/batch sizing is tuned against the CI runs above and can be revisited as data accumulates. > [!NOTE] > This pull request description was created with assistance from GitHub Copilot. --------- Co-authored-by: Copilot <[email protected]>
1 parent 3641098 commit 75bdb67

34 files changed

Lines changed: 235 additions & 31 deletions

File tree

‎eng/pipelines/libraries/outerloop.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,3 +107,47 @@ extends:
107107
testScope: outerloop
108108
creator: dotnet-bot
109109
extraHelixArguments: /p:BuildTargetFramework=net481
110+
111+
# WebAssembly CoreCLR - run the library test suites on Chrome with the slow
112+
# [OuterLoop] tests enabled. Uses -testscope all (not outerloop) on purpose:
113+
# the outerloop scope adds a '-trait category=OuterLoop' include filter, and the
114+
# wasm test host (xharness) treats a matching include as "always run", short-
115+
# circuiting the '-notrait category=failing' exclude. That leaks [OuterLoop]
116+
# tests from classes disabled via [ConditionalClass] (e.g. on
117+
# IsMultithreadingSupported) onto single-threaded wasm, where they block the
118+
# one thread and hang the job. -testscope all carries no OuterLoop include, so
119+
# the failing-tagged classes are skipped as intended; it also covers the slow
120+
# suites' innerloop tests.
121+
# Gated on includeLinuxOuterloop (browser_wasm builds on a Linux host) so the
122+
# wasm suites run in the umbrella and -linux outerloop definitions only, not in
123+
# every platform-split (-windows/-osx) outerloop definition.
124+
- ${{ if eq(variables['includeLinuxOuterloop'], true) }}:
125+
- template: /eng/pipelines/common/platform-matrix.yml
126+
parameters:
127+
jobTemplate: /eng/pipelines/common/global-build-job.yml
128+
helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
129+
buildConfig: Release
130+
runtimeFlavor: coreclr
131+
platforms:
132+
- browser_wasm
133+
jobParameters:
134+
testScope: all
135+
nameSuffix: LibraryTestsCoreCLR
136+
buildArgs: -s clr+libs+host+packs+libs.tests -c $(_BuildConfig) -testscope all /p:ArchiveTests=true /p:BrowserHost=$(_hostedOs) /p:InstallChromeForTests=true /p:AotHostArchitecture=x64 /p:AotHostOS=$(_hostedOS) /maxcpucount:1
137+
# This lane runs the slow suites with a 04:00:00 per-suite xharness
138+
# timeout and a per-batch Helix cap of 300 min (see
139+
# eng/testing/tests.wasm.targets and src/libraries/sendtohelix-browser.targets).
140+
# The job waits synchronously for the slowest batch, so it must outlast that
141+
# batch plus build/queue overhead.
142+
timeoutInMinutes: 600
143+
includeAllPlatforms: ${{ variables['isRollingBuild'] }}
144+
# extra steps, run tests
145+
postBuildSteps:
146+
- template: /eng/pipelines/libraries/helix.yml
147+
parameters:
148+
scenarios:
149+
- WasmTestOnChrome
150+
testScope: all
151+
creator: dotnet-bot
152+
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)
153+
extraHelixArguments: /p:BrowserHost=$(_hostedOs)

‎eng/testing/tests.wasm.targets‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,12 @@
3535
<_ShellCommandSeparator Condition="'$(OS)' == 'Windows_NT'">&amp;&amp;</_ShellCommandSeparator>
3636
<_ShellCommandSeparator Condition="'$(OS)' != 'Windows_NT'">&amp;&amp;</_ShellCommandSeparator>
3737
<XUnitUseRandomizedTestOrderer Condition="'$(XUnitUseRandomizedTestOrderer)' == '' and '$(IsTestProject)' == 'true'">true</XUnitUseRandomizedTestOrderer>
38+
<!-- Outerloop wasm/CoreCLR runs the slow [OuterLoop] suites; give each suite a much larger
39+
per-suite xharness timeout so a long suite is not killed mid-run (xharness exit code 71).
40+
Innerloop keeps the existing 01:30:00 (CoreCLR) / 00:30:00 budgets. Keep this in sync with
41+
the per-batch and AzDO job timeouts in src/libraries/sendtohelix-browser.targets and
42+
eng/pipelines/libraries/outerloop.yml (suite <= batch <= job). -->
43+
<WasmXHarnessTestsTimeout Condition="'$(WasmXHarnessTestsTimeout)' == '' and '$(RuntimeFlavor)' == 'CoreCLR' and ('$(TestScope)' == 'outerloop' or '$(TestScope)' == 'all')">04:00:00</WasmXHarnessTestsTimeout>
3844
<WasmXHarnessTestsTimeout Condition="'$(WasmXHarnessTestsTimeout)' == '' and '$(RuntimeFlavor)' == 'CoreCLR'">01:30:00</WasmXHarnessTestsTimeout>
3945
<WasmXHarnessTestsTimeout Condition="'$(WasmXHarnessTestsTimeout)' == ''">00:30:00</WasmXHarnessTestsTimeout>
4046
<RunWorkingDirectory>$(BundleDir)</RunWorkingDirectory>

‎src/libraries/Common/tests/System/Net/Http/ResponseStreamTest.cs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -565,6 +565,10 @@ public async Task BrowserHttpHandler_StreamingRequest_CancelRequest(bool cancelA
565565

566566
[OuterLoop]
567567
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsBrowser))]
568+
// Browser+CoreCLR currently surfaces a TaskCanceledException (100s HttpClient.Timeout)
569+
// instead of the expected HttpRequestException ("TypeError: Failed to fetch"); the
570+
// CoreCLR-wasm streaming-request-over-HTTP/1 path differs from Mono-wasm (which passes).
571+
[ActiveIssue("https://github.com/dotnet/runtime/issues/129758", typeof(PlatformDetection), nameof(PlatformDetection.IsCoreCLR))]
568572
public async Task BrowserHttpHandler_StreamingRequest_Http1Fails()
569573
{
570574
var WebAssemblyEnableStreamingRequestKey = new HttpRequestOptionsKey<bool>("WebAssemblyEnableStreamingRequest");

‎src/libraries/System.Collections.Immutable/tests/Frozen/FrozenDictionaryTests.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -422,6 +422,7 @@ public void ContainsKey_WithNonAscii(int percentageWithNonAscii)
422422

423423
[Fact]
424424
[OuterLoop]
425+
[ActiveIssue("https://github.com/dotnet/runtime/issues/129973", typeof(PlatformDetection), nameof(PlatformDetection.IsBrowser), nameof(PlatformDetection.IsCoreCLR))]
425426
public void ToFrozenDictionary_WithExtremelyLargeStrings()
426427
{
427428
// Test case with extremely large strings that exceed length bucket boundaries.

‎src/libraries/System.Collections.Immutable/tests/Frozen/FrozenSetTests.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,7 @@ public void Contains_WithNonAscii(int percentageWithNonAscii)
379379

380380
[Fact]
381381
[OuterLoop]
382+
[ActiveIssue("https://github.com/dotnet/runtime/issues/129973", typeof(PlatformDetection), nameof(PlatformDetection.IsBrowser), nameof(PlatformDetection.IsCoreCLR))]
382383
public void ToFrozenSet_WithExtremelyLargeStrings()
383384
{
384385
// Test case with extremely large strings that exceed length bucket boundaries.

‎src/libraries/System.Console/tests/WindowAndCursorProps.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -312,6 +312,7 @@ public void BeepWithFrequency_Unix_ThrowsPlatformNotSupportedException()
312312

313313
[Fact]
314314
[OuterLoop] // clears the screen, not very inner-loop friendly
315+
[SkipOnPlatform(TestPlatforms.Browser, "Console.Clear is not supported on Browser")]
315316
public static void Clear_Invoke_Success()
316317
{
317318
if (!OperatingSystem.IsWindows() || (!Console.IsInputRedirected && !Console.IsOutputRedirected))

‎src/libraries/System.Linq.Expressions/tests/CompilerTests.cs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@ public static class CompilerTests
1414
[Theory]
1515
[ClassData(typeof(CompilationTypes))]
1616
[OuterLoop("Takes over a minute to complete")]
17+
// Compiling a 10,000-deep expression tree takes hours on the single-threaded wasm
18+
// interpreter, exceeding the test timeout. Still covered on Mono and other platforms.
19+
[ActiveIssue("https://github.com/dotnet/runtime/issues/129973", typeof(PlatformDetection), nameof(PlatformDetection.IsBrowser), nameof(PlatformDetection.IsCoreCLR))]
1720
public static void CompileDeepTree_NoStackOverflow(bool useInterpreter)
1821
{
1922
var e = (Expression)Expression.Constant(0);

‎src/libraries/System.Linq.Parallel/tests/Combinatorial/CancellationParallelQueryCombinationTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -603,7 +603,7 @@ public static void Min_OperationCanceledException_PreCanceled(Labeled<Func<Paral
603603
AssertThrows.AlreadyCanceled(source => operation.Item(source, () => { }).Min());
604604
}
605605

606-
[Theory]
606+
[ConditionalTheory(typeof(PlatformDetection), nameof(PlatformDetection.IsMultithreadingSupported))] // Observing "eventual" cancellation requires concurrent workers; unsupported on single-threaded platforms.
607607
[ActiveIssue("https://github.com/dotnet/runtime/issues/22626", TestPlatforms.Linux)]
608608
[MemberData(nameof(UnaryCancelingOperators))]
609609
[MemberData(nameof(BinaryCancelingOperators))]

‎src/libraries/System.Linq.Parallel/tests/DegreeOfParallelismTests.cs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ public static void DegreeOfParallelism(Labeled<ParallelQuery<int>> labeled, int
4747
Assert.Equal(Functions.SumRange(0, count), labeled.Item.WithDegreeOfParallelism(degree).Sum());
4848
}
4949

50-
[Theory]
50+
[ConditionalTheory(typeof(PlatformDetection), nameof(PlatformDetection.IsMultithreadingSupported))] // Coordinates work across threads via a Barrier; deadlocks/fails on single-threaded platforms.
5151
[MemberData(nameof(DegreeData), new[] { 1, 4, 32 }, new int[] { /* same as count */ })]
5252
[OuterLoop]
5353
public static void DegreeOfParallelism_Barrier(Labeled<ParallelQuery<int>> labeled, int count, int degree)
@@ -59,7 +59,7 @@ public static void DegreeOfParallelism_Barrier(Labeled<ParallelQuery<int>> label
5959
}
6060
}
6161

62-
[Theory]
62+
[ConditionalTheory(typeof(PlatformDetection), nameof(PlatformDetection.IsMultithreadingSupported))] // Pipelined enumeration relies on concurrent producer threads; unsupported on single-threaded platforms.
6363
[MemberData(nameof(DegreeData), new int[] { /* Sources.OuterLoopCount */ }, new[] { 1, 4, 64, 128 })]
6464
[OuterLoop]
6565
public static void DegreeOfParallelism_Pipelining(Labeled<ParallelQuery<int>> labeled, int count, int degree)
@@ -74,7 +74,7 @@ public static void DegreeOfParallelism_Pipelining(Labeled<ParallelQuery<int>> la
7474
}
7575
}
7676

77-
[Theory]
77+
[ConditionalTheory(typeof(PlatformDetection), nameof(PlatformDetection.IsMultithreadingSupported))] // Pipelined enumeration relies on concurrent producer threads; unsupported on single-threaded platforms.
7878
[MemberData(nameof(DegreeData), new[] { 1, 4 }, new int[] { /* same as count */ })]
7979
[MemberData(nameof(DegreeData), new[] { 32 }, new[] { 4 })]
8080
[OuterLoop]

‎src/libraries/System.Linq.Parallel/tests/ExchangeTests.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ public static void Partitioning_Default(Labeled<ParallelQuery<int>> labeled, int
105105
}
106106
}
107107

108-
[Theory]
108+
[ConditionalTheory] // Delegates to Partitioning_Default, which skips (SkipTestException) when multithreading is unavailable.
109109
[OuterLoop]
110110
[MemberData(nameof(PartitioningData), new int[] { /* Sources.OuterLoopCount */ })]
111111
public static void Partitioning_Default_Longrunning(Labeled<ParallelQuery<int>> labeled, int count, int partitions)
@@ -129,7 +129,7 @@ public static void Partitioning_Striped(Labeled<ParallelQuery<int>> labeled, int
129129
}
130130
}
131131

132-
[Theory]
132+
[ConditionalTheory] // Delegates to Partitioning_Striped, which skips (SkipTestException) when multithreading is unavailable.
133133
[OuterLoop]
134134
[MemberData(nameof(PartitioningData), new int[] { /* Sources.OuterLoopCount */ })]
135135
public static void Partitioning_Striped_Longrunning(Labeled<ParallelQuery<int>> labeled, int count, int partitions)

0 commit comments

Comments
 (0)