Skip to content

Commit 8556037

Browse files
feat(collect-tests): expand .each and report accurate per-status counts (#16259)
1 parent f6dee87 commit 8556037

20 files changed

Lines changed: 808 additions & 150 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
- `[@jest/expect-utils, jest-mock]` Add `mockFn.whenCalledWith(...args)` for configuring return values per argument list, with first-class asymmetric-matcher support ([#16053](https://github.com/jestjs/jest/pull/16053))
66
- `[@jest/expect-utils]` Export `AsymmetricMatcher` and `FunctionParameters` types (previously private to `expect`) ([#16053](https://github.com/jestjs/jest/pull/16053))
7+
- `[jest-circus, jest-core, jest-jasmine2, jest-test-result, jest-types]` `--collectTests` now expands `test.each`/`describe.each` cases and reports per-status counts (skipped/todo via the new `wouldRun` flag for selected tests) plus a summary line that match a real run, including under `--testNamePattern` and `.only`/`fdescribe` focus on both the circus and jasmine2 runners ([#16259](https://github.com/jestjs/jest/pull/16259))
78
- `[jest-haste-map]` Replace `NodeWatcher` and `FSEventsWatcher` with `@parcel/watcher` for the non-watchman watch path ([#16188](https://github.com/jestjs/jest/pull/16188))
89
- `[jest-resolve]` Bump `unrs-resolver` to 1.12.1, remove `jest-pnp-resolver` and unnecessary checks ([#15721](https://github.com/jestjs/jest/pull/15721))
910

‎docs/CLI.md‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -150,16 +150,28 @@ A glob pattern relative to `rootDir` matching the files that coverage info needs
150150

151151
Discover and print all test suites and test names without executing them. Jest loads each test file, evaluates the top-level `describe` blocks to register tests, then exits before running any test code or lifecycle hooks.
152152

153-
Output is a tree of file paths with their nested describe and test names:
153+
Output is a tree of file paths with their nested describe and test names, followed by a summary of the total counts. Skipped and todo tests are annotated, and any file that throws while loading is reported with its error (and the exit code is non-zero):
154154

155155
```
156156
path/to/my.test.ts
157157
My suite
158158
passes
159-
fails
159+
skips [skipped]
160+
write later [todo]
161+
162+
Test suites: 1
163+
Tests: 3 total, 1 runnable, 1 skipped, 1 todo
160164
```
161165

162-
Use `--json` to get machine-readable output instead.
166+
Parametrized tests declared with [`test.each`](GlobalAPI.md#testeachtablename-fn-timeout) / `describe.each` are expanded to one entry per case, and each collected test is categorized exactly as a real run would categorize it:
167+
168+
- a test that would run is reported in the `passed` bucket and flagged `wouldRun: true` (it was selected but never executed);
169+
- [`test.skip`](GlobalAPI.md#testskipname-fn), tests inside a skipped `describe`, tests deselected by `test.only`, and tests excluded by `--testNamePattern` are reported as `pending`;
170+
- [`test.todo`](GlobalAPI.md#testtodoname) is reported as `todo`.
171+
172+
Because every test is accounted for in the same bucket an actual run would use, the counts reported by `--collectTests` match those of a run in which every selected test passes — so it can be used to count tests without executing them, including under `--testNamePattern`.
173+
174+
Use `--json` to get machine-readable output instead. The JSON uses the same shape as a normal run, including `numTotalTests`, `numPassedTests`, `numPendingTests` and `numTodoTests`, and each assertion carries the `wouldRun` flag described above.
163175

164176
### `--colors`
165177

‎e2e/__tests__/collectTests.test.ts‎

Lines changed: 106 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -7,21 +7,41 @@
77

88
import runJest from '../runJest';
99

10+
const COUNT_KEYS = [
11+
'numFailedTests',
12+
'numPassedTests',
13+
'numPendingTests',
14+
'numTodoTests',
15+
'numTotalTestSuites',
16+
'numTotalTests',
17+
] as const;
18+
19+
type Counts = Record<(typeof COUNT_KEYS)[number], number>;
20+
21+
const counts = (result: Counts): Counts =>
22+
Object.fromEntries(COUNT_KEYS.map(key => [key, result[key]])) as Counts;
23+
1024
describe('jest --collect-tests', () => {
11-
test('lists test names without executing test bodies', () => {
25+
// `test.failing` (xfail) collection parity is only implemented for the
26+
// default circus runner, so tests that rely on the kitchenSink fixture (which
27+
// uses `test.failing`) run on circus only. Focus parity is supported on both
28+
// runners and is exercised by a dedicated test below.
29+
const testOnCircus = process.env.JEST_JASMINE ? test.skip : test;
30+
31+
test('prints a tree and summary without executing tests', () => {
1232
const {exitCode, stdout} = runJest('each', [
1333
'--collect-tests',
1434
'--testPathPatterns=success',
1535
]);
1636

1737
expect(exitCode).toBe(0);
18-
expect(stdout).toContain("The word red contains the letter 'e'");
19-
expect(stdout).toContain('passes one row expected true == true');
20-
expect(stdout).toContain('passes all rows expected true == true');
2138
expect(stdout).toContain('success.test.js');
39+
expect(stdout).toContain("The word red contains the letter 'e'");
40+
expect(stdout).toMatch(/Test suites:\s+1/);
41+
expect(stdout).toMatch(/Tests:\s+15 total/);
2242
});
2343

24-
test('produces valid JSON with --json', () => {
44+
test('emits JSON with per-test status and the wouldRun flag', () => {
2545
const {exitCode, stdout} = runJest('each', [
2646
'--collect-tests',
2747
'--json',
@@ -32,39 +52,53 @@ describe('jest --collect-tests', () => {
3252
const json = JSON.parse(stdout);
3353
expect(json.success).toBe(true);
3454
expect(json.numTotalTestSuites).toBe(1);
35-
expect(json.numPendingTests).toBeGreaterThan(0);
3655

3756
const testFile = json.testResults[0];
3857
expect(testFile.name).toContain('success.test.js');
58+
// Every test in this fixture would run, so all collect to `passed` and are
59+
// flagged `wouldRun` (selected, but never executed).
3960
for (const assertion of testFile.assertionResults) {
40-
expect(assertion.status).toBe('pending');
61+
expect(assertion.status).toBe('passed');
62+
expect(assertion.wouldRun).toBe(true);
4163
}
4264
});
4365

44-
test('does not execute tests (failing tests still exit 0)', () => {
45-
const {exitCode, stdout} = runJest('each', [
66+
testOnCircus('annotates skipped and todo tests in the tree', () => {
67+
const {exitCode, stdout} = runJest('collect-tests', [
4668
'--collect-tests',
47-
'--testPathPatterns=failure',
69+
'--testPathPatterns=kitchenSink',
4870
]);
4971

5072
expect(exitCode).toBe(0);
51-
expect(stdout).toContain('failure.test.js');
52-
expect(stdout).toContain('fails');
73+
expect(stdout).toContain('explicitly skipped [skipped]');
74+
expect(stdout).toContain('write this later [todo]');
75+
expect(stdout).toContain('deeply nested passes\n'); // runnable: unannotated
76+
});
77+
78+
test('surfaces test files that fail to load', () => {
79+
const {exitCode, stdout} = runJest('collect-tests-load-error', [
80+
'--collect-tests',
81+
]);
82+
83+
// A file that throws while loading cannot be collected: it is reported with
84+
// its error, and the exit code is non-zero.
85+
expect(exitCode).not.toBe(0);
86+
expect(stdout).toContain('boom while loading this test file');
87+
expect(stdout).toMatch(/test suite\(s\) failed to load/);
88+
expect(stdout).toContain('this one collects fine'); // healthy file still listed
5389
});
5490

55-
test('filters correctly with --testNamePattern', () => {
91+
test('does not execute test bodies (failing tests still exit 0)', () => {
5692
const {exitCode, stdout} = runJest('each', [
5793
'--collect-tests',
58-
'--testPathPatterns=success',
59-
'--testNamePattern=one row',
94+
'--testPathPatterns=failure',
6095
]);
6196

6297
expect(exitCode).toBe(0);
63-
expect(stdout).toContain('passes one row expected');
64-
expect(stdout).not.toContain("The word red contains the letter 'e'");
98+
expect(stdout).toContain('fails');
6599
});
66100

67-
test('exits 0 even when no tests match', () => {
101+
test('exits 0 when no tests match', () => {
68102
const {exitCode, stdout} = runJest('each', [
69103
'--collect-tests',
70104
'--testPathPatterns=nonexistent',
@@ -73,4 +107,58 @@ describe('jest --collect-tests', () => {
73107
expect(exitCode).toBe(0);
74108
expect(stdout).toContain('No tests found');
75109
});
110+
111+
// The fixture's runnable tests all pass, so the per-status breakdown collected
112+
// without execution matches a real run exactly — including `.each` expansion,
113+
// skips, todos, xfail, focus, and `--testNamePattern` deselection.
114+
testOnCircus.each([
115+
['unfiltered', [] as Array<string>],
116+
['--testNamePattern', ['--testNamePattern=passes']],
117+
])('reports the same counts as a real run (%s)', (_label, args) => {
118+
const real = JSON.parse(
119+
runJest('collect-tests', ['--json', ...args]).stdout,
120+
);
121+
const collected = JSON.parse(
122+
runJest('collect-tests', ['--collect-tests', '--json', ...args]).stdout,
123+
);
124+
125+
expect(real.numFailedTests).toBe(0);
126+
expect(counts(collected)).toEqual(counts(real));
127+
expect(collected.numPassedTests).toBeGreaterThan(0);
128+
expect(collected.numPendingTests).toBeGreaterThan(0);
129+
});
130+
131+
// Focus (`test.only`/`fdescribe`) deselects the other tests in the file. Both
132+
// runners must collect the same per-status counts a real run produces; the
133+
// focused fixture has no `test.failing`, so jasmine parity holds here too.
134+
test('reports the same counts as a real run with focused tests', () => {
135+
const real = JSON.parse(
136+
runJest('collect-tests', ['--json', '--testPathPatterns=focused']).stdout,
137+
);
138+
const collected = JSON.parse(
139+
runJest('collect-tests', [
140+
'--collect-tests',
141+
'--json',
142+
'--testPathPatterns=focused',
143+
]).stdout,
144+
);
145+
146+
expect(real.numFailedTests).toBe(0);
147+
expect(real.numPassedTests).toBe(1);
148+
expect(real.numPendingTests).toBeGreaterThan(0);
149+
expect(counts(collected)).toEqual(counts(real));
150+
});
151+
152+
testOnCircus('marks xfail (test.failing) entries as failing', () => {
153+
const {stdout} = runJest('collect-tests', [
154+
'--collect-tests',
155+
'--json',
156+
'--testPathPatterns=kitchenSink',
157+
]);
158+
const xfail = JSON.parse(stdout).testResults[0].assertionResults.find(
159+
(assertion: {title: string}) =>
160+
assertion.title === 'known broken passes when it throws',
161+
);
162+
expect(xfail.failing).toBe(true);
163+
});
76164
});
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
/**
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
*
4+
* This source code is licensed under the MIT license found in the
5+
* LICENSE file in the root directory of this source tree.
6+
*/
7+
8+
throw new Error('boom while loading this test file');
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
/**
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
*
4+
* This source code is licensed under the MIT license found in the
5+
* LICENSE file in the root directory of this source tree.
6+
*/
7+
8+
test('this one collects fine', () => {
9+
expect(true).toBe(true);
10+
});
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"jest": {
3+
"testEnvironment": "node"
4+
}
5+
}
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
/**
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
*
4+
* This source code is licensed under the MIT license found in the
5+
* LICENSE file in the root directory of this source tree.
6+
*/
7+
8+
/* eslint-disable jest/no-focused-tests */
9+
10+
// When a file focuses tests with `.only`, the unfocused tests are reported as
11+
// skipped (pending) in a real run. `--collect-tests` must reflect the same.
12+
13+
test.only('focused passes', () => {
14+
expect(true).toBe(true);
15+
});
16+
17+
test('unfocused becomes skipped', () => {
18+
throw new Error('should never run');
19+
});
20+
21+
test.todo('todo survives focus');
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
/**
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
*
4+
* This source code is licensed under the MIT license found in the
5+
* LICENSE file in the root directory of this source tree.
6+
*/
7+
8+
// Exercises every shape that affects test counts: plain tests, `.each`
9+
// expansion (array + template), nested describes, `.skip`, `describe.skip`,
10+
// `.todo` and `.failing` (xfail). The companion e2e test asserts that
11+
// `--collect-tests` reports the same per-status counts as a real run.
12+
13+
const rows = Array.from({length: 30}, (_, index) => index);
14+
15+
describe('plain and each', () => {
16+
for (const index of rows) {
17+
test(`passes ${index}`, () => {
18+
expect(index).toBe(index);
19+
});
20+
}
21+
22+
test.each(rows)('array case %i', value => {
23+
expect(value).toBeGreaterThanOrEqual(0);
24+
});
25+
26+
it.each`
27+
left | right
28+
${1} | ${1}
29+
${2} | ${2}
30+
`('template case $left == $right', ({left, right}) => {
31+
expect(left).toBe(right);
32+
});
33+
34+
describe.each([1, 2])('describe.each %i', value => {
35+
test('deeply nested passes', () => {
36+
expect(value).toBe(value);
37+
});
38+
});
39+
});
40+
41+
describe('skips', () => {
42+
test.skip('explicitly skipped', () => {
43+
throw new Error('should never run');
44+
});
45+
46+
it.skip.each([1, 2])('skipped each %i', () => {
47+
throw new Error('should never run');
48+
});
49+
});
50+
51+
describe.skip('entirely skipped describe', () => {
52+
test('child one', () => {
53+
throw new Error('should never run');
54+
});
55+
describe('nested under skip', () => {
56+
test('grandchild', () => {
57+
throw new Error('should never run');
58+
});
59+
});
60+
});
61+
62+
describe('todos', () => {
63+
test.todo('write this later');
64+
test.todo('and this one');
65+
});
66+
67+
test.failing('known broken passes when it throws', () => {
68+
expect(1).toBe(2);
69+
});

‎e2e/collect-tests/package.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"jest": {
3+
"testEnvironment": "node"
4+
}
5+
}

0 commit comments

Comments
 (0)