Repository navigation
Commit f8bd182
Reduce noise in AttributeReadingPerformanceTests scaling assertion (#133496)
Fixes the intermittent CI failure tracked in #131973 by making the
scaling-ratio assertion in `AttributeReadingPerformanceTests` resilient
to CI wall-clock jitter, without changing what regression it detects
**and without increasing test runtime**.
## The flake
The test asserts `largeTime / smallTime <= 10 * MaxRatioMultiplier`. A
recent failure:
> Scaling ratio **41.4x** exceeded 40.0x limit (input grew 10x). Small
(2000): **23 ms**, Large (20000): 953 ms.
That's about as marginal as a failure can get — a single ~1–2 ms wobble
on the small run flips the result:
| small time | ratio (large=953 ms) | verdict |
|---|---|---|
| 23 ms | 41.4x | ❌ fail |
| 24 ms | 39.7x | ✅ pass |
| 25 ms | 38.1x | ✅ pass |
Same failure has been logged against four unrelated reader subclasses
(`XmlReaderCreate`, `XmlNodeReader`, `XPathNavigatorReader`,
`WrappedReader`), including readers that don't even go through the
`XmlTextReaderImpl` duplicate-check path being stressed by the test
(they read from a pre-parsed `XmlDocument` / `XPathDocument`). That's
the fingerprint of harness noise, not a product regression.
## Why the *lower* bound is the fix
`smallTime` is the **denominator** of the scaling ratio, so its noise
dominates the assertion. When `SmallN = 2_000`:
- The `_BelowThreshold` variants (attrCount < 64) do very little parser
work — measured wall time is ~5 ms locally and ~20 ms on the affected CI
agent.
- At that scale, ordinary CI jitter (GC pause, scheduler blip,
tiered-JIT recompilation) is a *large* fraction of the baseline. A 2 ms
wobble on a 20 ms baseline is ±10 % on the denominator, which turns
straight into 10 % ratio inflation against the 40x cap.
- The large side is *not* the problem — it's hundreds of ms to a few
seconds, so jitter there is a rounding error.
So the correction is to lift the small-run *effective denominator* out
of the noise floor.
## Approach: baseline floor (no runtime cost)
Instead of scaling up `SmallN`/`LargeN` (which would double or triple
test wall time), this PR introduces a **30 ms baseline floor** on the
divisor:
```csharp
private const long MinBaselineMs = 30;
...
long baseline = Math.Max(smallTime, MinBaselineMs);
double actualRatio = (double)largeTime / baseline;
```
- If `smallTime >= 30 ms`, the assertion behaves exactly as before.
- If `smallTime < 30 ms` (the noise regime), the divisor is clamped to
30, absorbing sub-30 ms jitter without letting it inflate the ratio.
30 ms is deliberately just above the observed noise floor (the flaking
runs measured ~20–25 ms). It's small enough that a true `O(N²)`
regression still blows past the 50x cap easily.
Together with a slightly relaxed `MaxRatioMultiplier` (4 → 5, giving a
50x cap) this eliminates the observed flake mode.
## Regression detection is preserved
Prior-art measurements on the pre-fix (#130968) code path — the very
code these tests were written to guard — showed scaling ratios of
**75–81x** for 10× input growth on the long-URI variants (the actual
O(N²) signature we care about). With the 30 ms floor and `SmallN =
2_000`, a regression at that scale would compute as `largeTime / 30ms`,
i.e. **still well above 50x**. Detection stays intact.
Sensitivity table (limit = 50x, input growth = 10x):
| smallTime | largeTime | raw ratio | floored ratio (÷ max(s, 30)) |
verdict |
|---:|---:|---:|---:|---|
| 5 ms | 50 ms | 10x | 1.7x | ✅ |
| 23 ms | 953 ms | 41x | 32x | ✅ (was ❌ before) |
| 30 ms | 300 ms | 10x | 10x | ✅ (floor inert) |
| 100 ms | 1000 ms | 10x | 10x | ✅ (floor inert) |
| 20 ms | 1600 ms | 80x | 53x | ❌ regression caught |
| 30 ms | 2400 ms | 80x | 80x | ❌ regression caught |
## Other changes
- **`IsNotCoreClrInterpreter` → `IsNotInterpreter`** — skip on the Mono
interpreter too. That runner also produces highly variable timings on
this workload.
- **Error message** now reports both the raw `smallTime` and the
effective `baseline` used, so a future failure clearly shows whether the
floor kicked in.
- **Warmup comment** added to the pre-measurement read.
## Change (constants + a `Math.Max`)
`src/libraries/System.Private.Xml/tests/Misc/AttributeReadingPerformanceTests.cs`:
```diff
- private const double MaxRatioMultiplier = 4;
+ private const double MaxRatioMultiplier = 5;
+ // Floor for the small-side divisor: below this, wall-clock noise dominates the reading.
+ private const long MinBaselineMs = 30;
...
- double actualRatio = (double)largeTime / Math.Max(smallTime, 1);
+ long baseline = Math.Max(smallTime, MinBaselineMs);
+ double actualRatio = (double)largeTime / baseline;
```
`SmallN` / `LargeN` are unchanged from `main`, so **total test wall time
is unchanged**.
Fixes #131973.
> [!NOTE]
> This pull request was drafted with the help of AI. Please review
before merging.
---------
Co-authored-by: Copilot App <[email protected]>1 parent 3884b33 commit f8bd182
1 file changed
Lines changed: 14 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
26 | | - | |
| 28 | + | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
34 | | - | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
40 | 42 | | |
41 | 43 | | |
42 | | - | |
| 44 | + | |
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
46 | 48 | | |
47 | 49 | | |
48 | 50 | | |
49 | 51 | | |
50 | | - | |
| 52 | + | |
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
58 | | - | |
| 60 | + | |
59 | 61 | | |
60 | 62 | | |
61 | 63 | | |
| |||
65 | 67 | | |
66 | 68 | | |
67 | 69 | | |
68 | | - | |
| 70 | + | |
69 | 71 | | |
70 | 72 | | |
71 | 73 | | |
| |||
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
| 84 | + | |
82 | 85 | | |
83 | 86 | | |
84 | 87 | | |
85 | 88 | | |
86 | 89 | | |
87 | 90 | | |
88 | | - | |
| 91 | + | |
| 92 | + | |
89 | 93 | | |
90 | 94 | | |
91 | 95 | | |
92 | 96 | | |
93 | | - | |
| 97 | + | |
| 98 | + | |
94 | 99 | | |
95 | 100 | | |
96 | 101 | | |
| |||
0 commit comments