Repository navigation
Commit 1654543
[browser][coreclr] Add wasm/CoreCLR libraries tests to outerloop CI (#129436)
## Summary
Adds a scheduled **outerloop** leg that runs the library test suites on
**Browser + CoreCLR** (Chrome), including the slow `[OuterLoop]` tests.
Previously the outerloop pipeline had no wasm/CoreCLR coverage — only
CoreCLR desktop and Mono `browser_wasm`.
Getting the slow suites to actually complete on single-threaded wasm
required, all scoped so **innerloop behavior is unchanged**:
1. **Enable** the long-running suites and slow `[OuterLoop]` tests that
are normally excluded on wasm.
2. **Extend** the xharness / Helix / AzDO timeouts and **isolate** each
slow suite into its own Helix batch.
3. **Gate** the `[OuterLoop]` tests that genuinely *cannot* run on a
single thread (or need a 64-bit address space) with `[Conditional*]`
attributes, so they skip cleanly instead of hanging or failing.
4. **Gate** the `[OuterLoop]` tests that *can* run but are simply too
slow / too memory-heavy on the single-threaded wasm interpreter with
`[ActiveIssue]`, scoped to **CoreCLR-wasm only** (Mono keeps running
them), tracked in #129973 for re-enablement.
## CI pipeline
- **`eng/pipelines/libraries/outerloop.yml`** — new `browser_wasm` +
`coreclr` job (`WasmTestOnChrome`, `Release`, `/maxcpucount:1`,
`timeoutInMinutes: 600`). Gated on `includeLinuxOuterloop` (browser_wasm
builds on a Linux host) so it runs in the umbrella / `-linux` outerloop
definitions only.
- Runs with **`-testscope all`, not `outerloop`** — this is deliberate.
The `outerloop` scope adds a `-trait category=OuterLoop` *include*
filter, and the wasm test host (xharness) treats a matching include as
"always run", short-circuiting the `-notrait category=failing`
*exclude*. That leaks `[OuterLoop]` tests out of classes disabled via
`[ConditionalClass]` (e.g. on `IsMultithreadingSupported`) onto
single-threaded wasm, where they block the one thread and hang the job.
`-testscope all` carries no OuterLoop include, so the platform gates are
honored; it also covers the slow suites' innerloop tests.
## Enabling slow suites + timeouts/batching
- **`src/libraries/tests.proj`** — gate the long-running suite
exclusions (`System.Linq`, `System.Memory`, `System.Private.Uri`,
`System.Private.Xml`, `System.Runtime.Numerics`, `System.Text.Json`)
behind a new `RunLongRunningWasmTests` property (`true` for
outerloop/all). `System.Formats.Nrbf` stays unconditionally excluded
(test failures, not slowness).
- **`src/libraries/sendtohelix-browser.targets`** — isolate each slow
suite into its own solo Helix batch (via the new `SoloItems`) so one
slow suite cannot inflate a shared batch, and raise the per-batch
timeout budget for outerloop.
- **`src/tasks/HelixTestTasks/GroupWorkItems.cs`** — add `SoloItems`:
work items named here each get their own negative-ID solo batch
regardless of size.
- **`src/tasks/HelixTestTasks/ComputeBatchTimeout.cs`** — parameterize
the per-batch timeout (`MinutesPerSuite` / `MinimumMinutes` /
`MaximumMinutes`); defaults reproduce the previous
20-min/30-min-floor/1439-cap behavior exactly, and the MSBuild-provided
knobs are validated/clamped.
- **`eng/testing/tests.wasm.targets`** — outerloop/all wasm CoreCLR gets
a `04:00:00` per-suite xharness timeout (innerloop keeps `01:30:00` /
`00:30:00`).
- **`src/libraries/sendtohelixhelp.proj`** — extend the non-batched
browser + CoreCLR work-item timeout to `04:30:00` for outerloop/all
(safety net for non-batched paths).
The timeout layers are kept ordered **xharness suite (4h) ≤ non-batched
work-item / batch (4h30 floor, 5h cap) ≤ AzDO job (10h)** so a slow
suite is bounded by its own timeout, and the synchronous send-to-Helix
wait always fits inside the job.
## Test gates — cannot run (single-threaded / 64-bit only)
`[OuterLoop]` tests that throw / hang on a single thread or need a
64-bit address space, gated at method or class level (never via
`tests.proj` exclusions) so they skip cleanly:
| File | Gate |
|---|---|
| `System.Threading.Tasks.Parallel/.../ParallelStateTest.cs` |
`[ConditionalClass(IsMultithreadingSupported)]` on `ParallelState` (all
77 throw PNSE in `TaskReplicator.Run`) |
| `System.Linq.Parallel/.../DegreeOfParallelismTests.cs` | 3
`DegreeOfParallelism_{Barrier,Pipelining,Throttled_Pipelining}` →
`[ConditionalTheory(IsMultithreadingSupported)]` |
| `System.Linq.Parallel/.../ExchangeTests.cs` |
`Partitioning_{Default,Striped}_Longrunning` → bare
`[ConditionalTheory]` (delegate into bodies that already `throw
SkipTestException`; the bare form keeps `partitions ≤ 1` rows passing
and skips the rest) |
|
`System.Linq.Parallel/.../CancellationParallelQueryCombinationTests.cs`
| `SequenceEqual_OperationCanceledException` →
`[ConditionalTheory(IsMultithreadingSupported)]` |
| `System.Linq.Parallel/.../WithCancellationTests.cs` |
`WithCancellation_DisposedEnumerator_...ProducerBlocked` →
`[ConditionalTheory(IsMultithreadingSupported)]` |
| `System.Runtime/.../System.Threading.Tasks.Tests/MethodCoverage.cs` |
`TaskContinuation` → `[ConditionalFact(IsMultithreadingSupported)]` |
|
`System.Runtime/.../System.Threading.Tasks.Tests/.../TaskAwaiterTests.cs`
| `GetResult_NotCompleted_BlocksUntilCompletion` →
`[ConditionalFact(IsMultithreadingSupported)]` |
| `System.Runtime/.../System.Threading.Tasks.Tests/Task/TaskAPMTest.cs`
| `WaitUntilCompleteTechnique`, `PollUntilCompleteTechnique` →
`[ConditionalTheory(IsMultithreadingSupported)]` |
|
`System.Runtime/.../System.Threading.Tasks.Tests/TaskScheduler/TaskSchedulerTests.cs`
| `RunBlockedInjectionTest`,
`RunSynchronizationContextTaskSchedulerTests` →
`[ConditionalFact(IsMultithreadingSupported)]` |
| `System.Runtime/.../System.Runtime.Tests/System/GCTests.cs` |
`WaitForPendingFinalizersRaces` → add `IsMultithreadingSupported` to
existing gate |
## Test gates — too slow / too memory-heavy (tracked in #129973)
These `[OuterLoop]` tests *can* run but exceed the per-suite timeout (or
exhaust the wasm heap) on the single-threaded interpreter. They are
gated with
`[ActiveIssue("#129973",
typeof(PlatformDetection), nameof(IsBrowser), nameof(IsCoreCLR))]`, i.e.
**disabled only on CoreCLR-wasm — Mono-wasm keeps running them**:
| File | Test(s) | Reason |
|---|---|---|
| `System.Linq.Expressions/.../CompilerTests.cs` |
`CompileDeepTree_NoStackOverflow` | compiles a 10,000-deep expression
tree |
| `System.Memory/.../EncodingExtensions/EncodingExtensionsTests.cs` |
`GetBytes_…LargeMultiSegment`, `GetChars_…LargeMultiSegment` | iterate
billions of chars/bytes |
| `System.Text.Json/.../BitStackTests.cs` | `BitStackPushPopLarge` |
pushes/pops up to 67M bits |
| `System.Text.Json/.../Serialization/Stream.ReadTests.cs` |
`ReadTypeFromJsonWithLargeIgnoredProperties_OuterLoop` | processes 5 GiB
of JSON |
| `System.Text.Json/.../Utf8JsonReaderTests*.cs` |
`TestJsonReaderLargestUtf8SegmentSizeOne`,
`TestPartialJsonReaderSlicesMultiSegment`,
`TestPartialJsonReaderSlicesSpecialNumbers` | O(n²) slicing |
| `System.Text.Json/.../Utf8JsonWriterTests.cs` |
`Writing3MBBase64Bytes`, `EscapeCharacters`, `WriteNumbers` | every
writer-option combination |
| `System.Collections.Immutable/.../Frozen/FrozenDictionaryTests.cs`,
`FrozenSetTests.cs` | `ToFrozen*_WithExtremelyLargeStrings` | allocate
~1 GB strings (memory exhaustion) |
| `System.Private.Xml.Linq/.../xNodeBuilder/FunctionalTests.cs` |
`RunTests` | wasm runtime hangs at shutdown after this suite (a hang,
not slowness) |
Suite-level:
**`System.Linq.Parallel/tests/System.Linq.Parallel.Tests.csproj`** drops
the whole `[OuterLoop]` category on the wasm/CoreCLR lane
(`WithoutCategories=OuterLoop`). The combinatorial matrix expands to
~195k cases on a single thread (~72 MB results file exceeding xharness's
30 MB upload limit, ~1.3 GB heap, tab crash at teardown), and PLINQ has
no real parallelism on one thread.
Separately, `System.Net.Http/.../ResponseStreamTest.cs`
`BrowserHttpHandler_StreamingRequest_Http1Fails` is gated with
`[ActiveIssue(#129758, IsCoreCLR)]` — a real CoreCLR-wasm behavior bug
(surfaces `TaskCanceledException` instead of `HttpRequestException`),
not a slowness gate.
## CI validation
Latest scheduled outerloop run (build `1485712`, Helix job
`WasmTestOnChrome-CLR-ST`): **31 work items, all passed** (1 controller
+ 30 test batches — 20 grouped, 10 solo). Across all batches:
| Suites | Tests run | Passed | Failed | Skipped |
|---:|---:|---:|---:|---:|
| 203 | 657,944 | 653,983 | **0** | 3,961 |
Earlier local verification (wasm/Chrome, Release, `-testscope all`) of
individual suites: System.Threading.Tasks.Parallel.Tests 242 / **0
fail** (was 77); System.Security.Cryptography 4993 / **0 fail** / 804
skip; System.Runtime.Caching and Diagnostics.Metrics clean.
(System.Linq.Parallel was validated at 195,566 / 0 fail before its
`[OuterLoop]` matrix was dropped on this lane, per above.)
## Notes / things to watch
- The newly-enabled suites were previously excluded for **slowness**
(#125495), not crashes. Extending timeouts is the intended remedy; the
slow/memory tests that still couldn't fit the budget are tracked in
#129973 (CoreCLR-wasm only) rather than excluded wholesale.
- Timeout/batch sizing is tuned against the CI runs above and can be
revisited as data accumulates.
> [!NOTE]
> This pull request description was created with assistance from GitHub
Copilot.
---------
Co-authored-by: Copilot <[email protected]>1 parent 8489f32 commit 1654543
34 files changed
Lines changed: 235 additions & 31 deletions
File tree
- eng
- pipelines/libraries
- testing
- src
- libraries
- Common/tests/System/Net/Http
- System.Collections.Immutable/tests/Frozen
- System.Console/tests
- System.Linq.Expressions/tests
- System.Linq.Parallel/tests
- Combinatorial
- System.Memory/tests/EncodingExtensions
- System.Net.Http/tests/FunctionalTests
- System.Net.WebSockets.Client/tests
- System.Private.Xml.Linq/tests/xNodeBuilder
- System.Runtime/tests
- System.Runtime.Tests/System
- System.Threading.Tasks.Tests
- System.Runtime.CompilerServices
- TaskScheduler
- Task
- System.Threading.Timer.Tests
- System.Text.Json/tests/System.Text.Json.Tests
- Serialization
- System.Threading.Tasks.Parallel/tests
- tasks/HelixTestTasks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
38 | 44 | | |
39 | 45 | | |
40 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
565 | 565 | | |
566 | 566 | | |
567 | 567 | | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
568 | 572 | | |
569 | 573 | | |
570 | 574 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
422 | 422 | | |
423 | 423 | | |
424 | 424 | | |
| 425 | + | |
425 | 426 | | |
426 | 427 | | |
427 | 428 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
379 | 379 | | |
380 | 380 | | |
381 | 381 | | |
| 382 | + | |
382 | 383 | | |
383 | 384 | | |
384 | 385 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
312 | 312 | | |
313 | 313 | | |
314 | 314 | | |
| 315 | + | |
315 | 316 | | |
316 | 317 | | |
317 | 318 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
17 | 20 | | |
18 | 21 | | |
19 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
603 | 603 | | |
604 | 604 | | |
605 | 605 | | |
606 | | - | |
| 606 | + | |
607 | 607 | | |
608 | 608 | | |
609 | 609 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
| 50 | + | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
| 62 | + | |
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
| 77 | + | |
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | | - | |
| 108 | + | |
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
| |||
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
132 | | - | |
| 132 | + | |
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
| |||
0 commit comments