Add support for collapsed error messages by natebosch · Pull Request #2651 · dart-lang/test

natebosch · 2026-05-20T01:58:58Z

Checks against a subject are commonly nested against some other subject,
for example the check for an exact list length is in practice an
equals expectation checked against an integer which is the "has
length" subject, nested under the subject holding the list instance.
Failure messages have a rigid structure that represents this nesting
pattern and always indent the "clauses" (the descriptions of what was
checked in expectations) following the "label" which describes the
subject.

For very simple conditions the noise of the structure is greater than
the signal of the description output by the specific failing
expectation. For instance a trivial check for a future completing to an
expected value looks like:

  Expected: a Future<List<int>> that:
    completes to a value that:
      is not empty
  Actual: a Future<List<int>> that:
    completes to a value that:
    Actual: []
    Which: is empty

Add support for collapsing this to:

  Expected: a Future<List<int>> that completes to a non-empty iterable
  Actual: a Future<List<int>> that completes to []
  Which: is empty

Add a predicateNoun callback argument to expect, expectAsync, and
expectUnawaited. Extensions must pass a callback which provides the
"clause" representation like "is empty". Now they may also pass a
callback which provides a predicate noun formed by combining the clause
with a noun for the subject like "an empty iterable".

Add an addPredicate callback argument to nest, and nestAsync.
Extensions must pass a callback which provides a label for the value
they are deriving from the original subject like "completes to a value".
Now they may also pass a callback which takes a collapsed form of the
noun as described by nested conditions, and attaches the predicate it
contributes.

Check for possibilities to collapse failure messages down to predicate
nouns when generating failure messages. When full single-line collapsing
is not possible, child clauses and Actual: lines still collapse where
supported within the multi-line format.

Add predicate noun callbacks to existing extensions where sensible.
Tweak some wording where I noticed inconsistencies.

Use of these new arguments is completely optional for extension
implementations and impacts only the failure message formatting which is
not considered a breaking change.

The user impact is that many simple expectation failures will be more
compact and readable, but the position of the failing expectation in a
series of cascaded expectations may cause a significantly different
output for the same checks usage. It has always been the case that
expectations which follow a failure have no impact on the output,
because they are never reached to run. Now when the first expectation in
a series has a collapsible representation, its failure triggers an
overall collapsed format which will read differently from the expanded
format if any condition other than the first fails.

github-actions · 2026-05-20T01:59:57Z

PR Health

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

Checks against a subject are commonly nested against some other subject, for example the check for an exact list length is in practice an `equals` expectation checked against an integer which is the "has length" subject, nested under the subject holding the list instance. Failure messages have a rigid structure that represents this nesting pattern and always indent the "clauses" (the descriptions of what was checked in expectations) following the "label" which describes subject. For very simple conditions the noise of the structure is greater than the signal of the description output by the specific failing expectation. For instance a trivial check for an future completing to an expected value looks like: ``` Expected: a Future<List<int>> that: completes to a value that: is not empty Actual: a Future<List<int>> that: completes to a value that: Actual: [] Which: is empty ``` Add support for collapsing this to. ``` Expected: a Future<List<int>> that completes to a non-empty iterable Actual: a Future<List<int>> that completes to [] Which: is empty ``` Add a `predicateNoun` callback argument to `expect`, `expectAsync`, and `expectUnawaited`. Extensions must pass a callback which provides the "clause" representation like "is empty". Now they may also pass a callback which provides a predicate noun formed by combining the clause with a noun for the subject like "an empty iterable". Add a `addPredicate` callback argument to `nest`, and `nestAsync`. Extensions must pass a callback which provides a label for the value they are deriving from the original subject like "completes to a value". Now they may also pass a callback which takes a collapsed form of the noun as described by nested conditions, and attaches the predicate it contributes. Check for possibilities to collapse failure messages down to predicate nouns when generating failure messages. Add predicate noun callbacks to existing extensions where sensible. Tweak some wording where I noticed inconsistencies. Use of these new arguments is completely optional for extension implementations and impacts only the failure message formatting which is not considered a breaking change. The user impact is that many simple expectation failures will be more compact and readable, but the position of the failing expectation in a series of cascaded expectations may cause a significantly different output for the same checks usage. It has always been the case that expectations which follow a failure have no impact on the output, because they are never reached to run. Now when the first expectation in a series has a collapsible representation it's failure triggers an overall collapsed format which will read differently to the expanded format if any condition other than the first fails.

brianquinlan

This looks goods if you are convinced that the complexity is worth it.

brianquinlan · 2026-08-18T22:52:32Z

+/// expectation modifying the noun (often just a representation of the expected
+/// value) to be used when collapsing simple expectations.
+///
+/// Examples of predicate nouns:


Very nice! Love the examples.

lrhn

(Can't claim to actually be able to follow the computations, but it looks plausible and it's tested!)

lrhn · 2026-08-19T08:50:10Z

-    });
+    context.expect(
+      () => prefixFirst('equals ', literal(other)),
+      predicateNoun: () => literal(other).singleOrNull,


Probably not worth it, but:

Doing literal(other) twice, which calls the potentially expensive prettyPrint.
(It returns an Iterable which is always a List. Just change the return type to List, instead of risking code that depends on it being cheap to iterate and safe to iterate twice, without actually being promised that.)

Could do List<String>? otherLiteral; before the context.expect call and () => prefixFirst('equals ', otherLiteral ??= literal(other)), and
predicateNoun: () => (otherLiteral ??= literal(other)).singleOrNull.

I know this only gets called if a test fails, and it's not like any of the other list-of-string operations are optimized, so it's probably not performance critical. I just don't like seing duplicate work.

It happens a lot that there is an other and both callbacks call literal on it, but it's not every context.expect that has an other value.
(Could have a helper extension function context.expectWithOther(other, (otherLiteral) => prefixFirst('equals ', literal(other)), predicateName: (otherLiteral) => otherLiteral.singleOrNull, ..., or a LazyLiteral(other) with a caching lazyOtherLiteral.literal.)

I think we can address this later and it will be non-breaking. For now I'll keep them separate since the callbacks are called (hopefully) rarely, and a memoization approach add complexity.

lrhn · 2026-08-19T08:53:37Z

+      predicateNoun: () => literal(other).singleOrNull,
+      (actual) {
+        if (actual == other) return null;
+        return Rejection(which: ['is not equal']);


I'd keep the are.

Being equal is a binary relation. It needs either two subjects (so s plural "are") or a to <other> subject complement (a term I just looked up, so I hope it's correct).

Unless the "to" gets added by the predicateNoun handling, but I don't really nuderstand the flow flow, and there is no documentation!

This failure looks like:

Expected: <2> Actual: <1> Which: is not equal

I think initially I had this longer and shortened it because it felt redundant:

Expected: <2> Actual: <1> Which: is not equal to <2>

I think the "is" reads better, but we can adjust this in a followup. We don't consider is breaking to change the particular wording of failure messages.

lrhn · 2026-08-19T09:00:38Z

+      predicateNoun: () => literal(other).singleOrNull,
+      (actual) {
+        if (identical(actual, other)) return null;
+        return Rejection(which: ['is not identical']);


"are", unless to $other gets added somewhere.
(And I doubt that, because won't always be "to" that's the right word, so unless it's a default, this function should still mention the "to".)

lrhn · 2026-08-19T09:41:38Z

+    () => ['has last element'],
+    addPredicate: (predicateNoun) => 'has last element: $predicateNoun',
+    (actual) {
+      final iterator = actual.iterator;


(Unrelated: Could this be using the actual.last instead. It might be more efficient.
It'd have to catch a StateError and recognize that as meaning "no element", which isn't great, or iterate twice, which is also not great.
_I tried testing an iterable, and was surprised when check(myIteralble).last.equals(x) didn't call my last code. I guess you have to do check(() => myIterable.last)..., putting the actual code you want to check inside the check, to be sure. Might need to be documented somewhere.)

lrhn · 2026-08-19T10:50:01Z

+    Rejection? rejection, {
+    required bool thisContextFailed,
+  }) {
+    final clause = _clauses.singleOrNull;


(Wouldn't it be awesome if return was an expression:

final clause = _clauses.singleOrNull ?? (return null);

😁 )

It's not the first time I've wanted that, but it's close 😄

lrhn · 2026-08-19T10:52:12Z

+      if (clause.isLeaf) {
+        expandedCollapsed = (childExpected, childActual);
+      } else {
+        final rootLabel = _label().single;


(Confusing to read, could _label be named _labelFactory or _labelFunction?)

How about _labelCallback?

Done.

lrhn · 2026-08-19T10:52:54Z

+    );
+  }
+
+  /// Computes the uncollapsed, multi-line [FailureDetail] for this context and


Noun-phrased function, drop the "Computes". (Which is better than "Returns", but not more necessary.)

lrhn · 2026-08-19T11:15:45Z

-      FailureDetail(_expected(), -1, -1);
+  FailureDetail detail(_TestContext failingContext, Rejection? rejection) {
+    final collapsedExpected = _predicateNoun?.call();
+    final collapsedActualString = collapsedExpected != null


(Remove String from name to match the others. And to not repeat type in name.)

lrhn · 2026-08-19T11:19:05Z

+        : null;
+    final collapsedDetails = collapsedActualString != null
+        ? (collapsedExpected!, collapsedActualString)
+        : null;


Hard to follow the logic. Maybe nesting would be better?

(String, String)? collapsedDetails; if (_predicateNoun?.call() case final collapsedExpected?) { if (rejection?.actual.singleOrNull case final collapsedActual?) { collapsedDetails = (collapsedExpected, collapsedActual); } }

I think it's the same computation, but I find it much easier to read.
(And it doesn't need !, which is a sign that the code has the same logic as the programmer.)

Support collapsing for clauses other than the one that failed.

This reverts commit 5a6813d. I think this is still useful in some cases

Don't need to consider `thisContextFailed` for the `_collapsedDetails` implementation. Either the clause is on the failing path, or the rejection will be null. Add 2 asserts documenting guarantees Fix a bug where non-failing child context with a failing parent context were treated as failing themselves.

Revisions updated by `dart tools/rev_sdk_deps.dart test`. test (https://github.com/dart-lang/test/compare/55b186d..18ef597): 18ef5978 Fri Sep 4 10:13:39 2026 -0700 Nate Bosch Add support for collapsed error messages (dart-lang/test#2651) 6783445c Wed Sep 2 16:46:04 2026 -0700 Nate Bosch Prepare to publish (dart-lang/test#2737) 8de693e1 Tue Sep 1 03:54:15 2026 +0000 dependabot[bot] Bump the github-actions group with 2 updates (dart-lang/test#2744) 575e4dad Mon Aug 31 17:01:46 2026 -0700 Nate Bosch Fix deadlock waiting on serverSocket (dart-lang/test#2740) bd9acf32 Thu Aug 27 22:53:54 2026 -0700 Nate Bosch Unskip a test on windows (dart-lang/test#2741) b575eda9 Thu Aug 27 17:16:16 2026 -0700 Nate Bosch Remove redundant tag (dart-lang/test#2738) d21b06cb Thu Aug 27 16:38:17 2026 -0700 dependabot[bot] Bump the github-actions group across 1 directory with 5 updates (dart-lang/test#2709) 773bc56c Thu Aug 27 16:11:00 2026 -0700 Nate Bosch Use read permissions in test_descriptor and test_process workflows (dart-lang/test#2736) 73e85f1b Wed Aug 26 06:36:41 2026 +0700 herdiyanitdev Precompiled browser tests: stop serving files from outside the precompiled root (dart-lang/test#2724) bbe8f9bf Tue Aug 18 09:02:24 2026 -0700 Kevin Moore Export TestFailure from package:test_api/scaffolding.dart (dart-lang/test#2723) TAG=agy CONV=e6fd1c32-ded3-443f-83e5-2e15b4ffaccb [email protected] Change-Id: Iabc67d46c413470f0d2e5341e719d4edf702a94b Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/546020 Commit-Queue: Nate Bosch <[email protected]> Auto-Submit: Nate Bosch <[email protected]> Reviewed-by: Kevin Moore <[email protected]> Commit-Queue: Kevin Moore <[email protected]>

github-actions Bot added the package:checks Issues related to pkg:checks label May 20, 2026

natebosch force-pushed the collapse-failure-messages branch 6 times, most recently from dccf79c to ccbd58e Compare May 20, 2026 03:14

natebosch force-pushed the collapse-failure-messages branch from ccbd58e to bd2f609 Compare May 20, 2026 03:16

natebosch mentioned this pull request Jun 16, 2026

package:checks stable release burndown #2672

Open

natebosch added 8 commits July 18, 2026 04:27

Refactor to reduce duplication

e609893

Merge branch 'master' into collapse-failure-messages

a7c4ece

Rename and add another doc comment

7451a62

Update merged in test to new format

cc87daf

Merge branch 'master' into collapse-failure-messages

63ed54b

Tweaks and more detailed docs

be566d9

More singleOrNull

992942c

Tweaks

fae0e20

natebosch marked this pull request as ready for review August 18, 2026 22:14

natebosch requested a review from a team as a code owner August 18, 2026 22:14

natebosch requested review from brianquinlan and lrhn August 18, 2026 22:14

brianquinlan approved these changes Aug 18, 2026

View reviewed changes

lrhn approved these changes Aug 19, 2026

View reviewed changes

natebosch added 5 commits September 4, 2026 00:21

Merge branch 'master' into collapse-failure-messages

e86bb2d

More consistent flow

afe29eb

Shorter throws Object label

fce33c3

Doc tweaks

a927d0b

_label -> _labelCallback

f9e41f7

natebosch added 15 commits September 4, 2026 01:25

More comment tweak, rename field

bdf3ff1

Changed wording in test

486f52d

Refactor chained conditional ternaries

26b4e88

Check isEmpty after contains

673d5f4

Drop some redudnant which

5a6813d

format

14f0885

Unnecessary escape

93008aa

Collapse passing clauses

11d3a12

Support collapsing for clauses other than the one that failed.

Revert "Drop some redudnant which"

b0ab1e6

This reverts commit 5a6813d. I think this is still useful in some cases

Drop atSameLevel clauses when collapsing

8325c49

Collapse for async throws and emits

f037c96

Format

3708984

Expand context doc

b1fea6d

Add collapsing in Actual: lines

1e7f36d

natebosch merged commit 18ef597 into master Sep 4, 2026
71 checks passed

natebosch deleted the collapse-failure-messages branch September 4, 2026 17:13

Conversation

natebosch commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Health

Uh oh!

brianquinlan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lrhn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

natebosch commented May 20, 2026 •

edited

Loading

github-actions Bot commented May 20, 2026 •

edited

Loading