Skip to content

Commit e247a61

Browse files
avivkelleraduh95
avivkeller
authored and
aduh95
committedApr 29, 2024
test_runner: add --test-skip-pattern cli option
PR-URL: #52529 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
1 parent c8c6173 commit e247a61

File tree

13 files changed

+185
-28
lines changed

13 files changed

+185
-28
lines changed
 

Diff for: ‎doc/api/cli.md

+17
Original file line numberDiff line numberDiff line change
@@ -1984,6 +1984,9 @@ A regular expression that configures the test runner to only execute tests
19841984
whose name matches the provided pattern. See the documentation on
19851985
[filtering tests by name][] for more details.
19861986

1987+
If both `--test-name-pattern` and `--test-skip-pattern` are supplied,
1988+
tests must satisfy **both** requirements in order to be executed.
1989+
19871990
### `--test-only`
19881991

19891992
<!-- YAML
@@ -2052,6 +2055,20 @@ node --test --test-shard=2/3
20522055
node --test --test-shard=3/3
20532056
```
20542057

2058+
### `--test-skip-pattern`
2059+
2060+
<!-- YAML
2061+
added:
2062+
- REPLACEME
2063+
-->
2064+
2065+
A regular expression that configures the test runner to skip tests
2066+
whose name matches the provided pattern. See the documentation on
2067+
[filtering tests by name][] for more details.
2068+
2069+
If both `--test-name-pattern` and `--test-skip-pattern` are supplied,
2070+
tests must satisfy **both** requirements in order to be executed.
2071+
20552072
### `--test-timeout`
20562073

20572074
<!-- YAML

Diff for: ‎doc/api/test.md

+15-8
Original file line numberDiff line numberDiff line change
@@ -298,12 +298,15 @@ describe.only('a suite', () => {
298298

299299
## Filtering tests by name
300300

301-
The [`--test-name-pattern`][] command-line option can be used to only run tests
302-
whose name matches the provided pattern. Test name patterns are interpreted as
303-
JavaScript regular expressions. The `--test-name-pattern` option can be
304-
specified multiple times in order to run nested tests. For each test that is
305-
executed, any corresponding test hooks, such as `beforeEach()`, are also
306-
run. Tests that are not executed are omitted from the test runner output.
301+
The [`--test-name-pattern`][] command-line option can be used to only run
302+
tests whose name matches the provided pattern, and the
303+
[`--test-skip-pattern`][] option can be used to skip tests whose name
304+
matches the provided pattern. Test name patterns are interpreted as
305+
JavaScript regular expressions. The `--test-name-pattern` and
306+
`--test-skip-pattern` options can be specified multiple times in order to run
307+
nested tests. For each test that is executed, any corresponding test hooks,
308+
such as `beforeEach()`, are also run. Tests that are not executed are omitted
309+
from the test runner output.
307310

308311
Given the following test file, starting Node.js with the
309312
`--test-name-pattern="test [1-3]"` option would cause the test runner to execute
@@ -327,8 +330,8 @@ test('Test 4', async (t) => {
327330

328331
Test name patterns can also be specified using regular expression literals. This
329332
allows regular expression flags to be used. In the previous example, starting
330-
Node.js with `--test-name-pattern="/test [4-5]/i"` would match `Test 4` and
331-
`Test 5` because the pattern is case-insensitive.
333+
Node.js with `--test-name-pattern="/test [4-5]/i"` (or `--test-skip-pattern="/test [4-5]/i"`)
334+
would match `Test 4` and `Test 5` because the pattern is case-insensitive.
332335

333336
To match a single test with a pattern, you can prefix it with all its ancestor
334337
test names separated by space, to ensure it is unique.
@@ -349,6 +352,9 @@ only `some test` in `test 1`.
349352

350353
Test name patterns do not change the set of files that the test runner executes.
351354

355+
If both `--test-name-pattern` and `--test-skip-pattern` are supplied,
356+
tests must satisfy **both** requirements in order to be executed.
357+
352358
## Extraneous asynchronous activity
353359

354360
Once a test function finishes executing, the results are reported as quickly
@@ -3153,6 +3159,7 @@ Can be used to abort test subtasks when the test has been aborted.
31533159
[`--test-only`]: cli.md#--test-only
31543160
[`--test-reporter-destination`]: cli.md#--test-reporter-destination
31553161
[`--test-reporter`]: cli.md#--test-reporter
3162+
[`--test-skip-pattern`]: cli.md#--test-skip-pattern
31563163
[`--test`]: cli.md#--test
31573164
[`MockFunctionContext`]: #class-mockfunctioncontext
31583165
[`MockTimers`]: #class-mocktimers

Diff for: ‎doc/node.1

+5-1
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,11 @@ option set.
442442
.
443443
.It Fl -test-shard
444444
Test suite shard to execute in a format of <index>/<total>.
445-
445+
.
446+
.It Fl -test-skip-pattern
447+
A regular expression that configures the test runner to skip tests
448+
whose name matches the provided pattern.
449+
.
446450
.It Fl -test-timeout
447451
A number of milliseconds the test execution will fail after.
448452
.

Diff for: ‎lib/internal/test_runner/runner.js

+22-2
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ function filterExecArgv(arg, i, arr) {
113113
!ArrayPrototypeSome(kFilterArgValues, (p) => arg === p || (i > 0 && arr[i - 1] === p) || StringPrototypeStartsWith(arg, `${p}=`));
114114
}
115115

116-
function getRunArgs(path, { forceExit, inspectPort, testNamePatterns, only }) {
116+
function getRunArgs(path, { forceExit, inspectPort, testNamePatterns, testSkipPatterns, only }) {
117117
const argv = ArrayPrototypeFilter(process.execArgv, filterExecArgv);
118118
if (forceExit === true) {
119119
ArrayPrototypePush(argv, '--test-force-exit');
@@ -124,6 +124,9 @@ function getRunArgs(path, { forceExit, inspectPort, testNamePatterns, only }) {
124124
if (testNamePatterns != null) {
125125
ArrayPrototypeForEach(testNamePatterns, (pattern) => ArrayPrototypePush(argv, `--test-name-pattern=${pattern}`));
126126
}
127+
if (testSkipPatterns != null) {
128+
ArrayPrototypeForEach(testSkipPatterns, (pattern) => ArrayPrototypePush(argv, `--test-skip-pattern=${pattern}`));
129+
}
127130
if (only === true) {
128131
ArrayPrototypePush(argv, '--test-only');
129132
}
@@ -448,7 +451,7 @@ function watchFiles(testFiles, opts) {
448451
function run(options = kEmptyObject) {
449452
validateObject(options, 'options');
450453

451-
let { testNamePatterns, shard } = options;
454+
let { testNamePatterns, testSkipPatterns, shard } = options;
452455
const {
453456
concurrency,
454457
timeout,
@@ -514,6 +517,22 @@ function run(options = kEmptyObject) {
514517
throw new ERR_INVALID_ARG_TYPE(name, ['string', 'RegExp'], value);
515518
});
516519
}
520+
if (testSkipPatterns != null) {
521+
if (!ArrayIsArray(testSkipPatterns)) {
522+
testSkipPatterns = [testSkipPatterns];
523+
}
524+
525+
testSkipPatterns = ArrayPrototypeMap(testSkipPatterns, (value, i) => {
526+
if (isRegExp(value)) {
527+
return value;
528+
}
529+
const name = `options.testSkipPatterns[${i}]`;
530+
if (typeof value === 'string') {
531+
return convertStringToRegExp(value, name);
532+
}
533+
throw new ERR_INVALID_ARG_TYPE(name, ['string', 'RegExp'], value);
534+
});
535+
}
517536

518537
const root = createTestTree({ __proto__: null, concurrency, timeout, signal });
519538
root.harness.shouldColorizeTestFiles ||= shouldColorizeTestFiles(root);
@@ -537,6 +556,7 @@ function run(options = kEmptyObject) {
537556
signal,
538557
inspectPort,
539558
testNamePatterns,
559+
testSkipPatterns,
540560
only,
541561
forceExit,
542562
};

Diff for: ‎lib/internal/test_runner/test.js

+33-17
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ const {
8484
forceExit,
8585
sourceMaps,
8686
testNamePatterns,
87+
testSkipPatterns,
8788
testOnlyFlag,
8889
} = parseCommandLine();
8990
let kResistStopPropagation;
@@ -137,6 +138,19 @@ function stopTest(timeout, signal) {
137138
return deferred.promise;
138139
}
139140

141+
function testMatchesPattern(test, patterns) {
142+
const matchesByNameOrParent = ArrayPrototypeSome(patterns, (re) =>
143+
RegExpPrototypeExec(re, test.name) !== null,
144+
) || (test.parent && testMatchesPattern(test.parent, patterns));
145+
if (matchesByNameOrParent) return true;
146+
147+
const testNameWithAncestors = StringPrototypeTrim(test.getTestNameWithAncestors());
148+
149+
return ArrayPrototypeSome(patterns, (re) =>
150+
RegExpPrototypeExec(re, testNameWithAncestors) !== null,
151+
);
152+
}
153+
140154
class TestContext {
141155
#test;
142156

@@ -300,8 +314,7 @@ class Test extends AsyncResource {
300314
ownAfterEachCount: 0,
301315
};
302316

303-
if ((testNamePatterns !== null && !this.matchesTestNamePatterns()) ||
304-
(testOnlyFlag && !this.only)) {
317+
if (this.willBeFiltered()) {
305318
this.filtered = true;
306319
this.parent.filteredSubtestCount++;
307320
}
@@ -408,18 +421,16 @@ class Test extends AsyncResource {
408421
}
409422
}
410423

411-
matchesTestNamePatterns() {
412-
const matchesByNameOrParent = ArrayPrototypeSome(testNamePatterns, (re) =>
413-
RegExpPrototypeExec(re, this.name) !== null,
414-
) ||
415-
this.parent?.matchesTestNamePatterns();
416-
417-
if (matchesByNameOrParent) return true;
424+
willBeFiltered() {
425+
if (testOnlyFlag && !this.only) return true;
418426

419-
const testNameWithAncestors = StringPrototypeTrim(this.getTestNameWithAncestors());
420-
if (!testNameWithAncestors) return false;
421-
422-
return ArrayPrototypeSome(testNamePatterns, (re) => RegExpPrototypeExec(re, testNameWithAncestors) !== null);
427+
if (testNamePatterns && !testMatchesPattern(this, testNamePatterns)) {
428+
return true;
429+
}
430+
if (testSkipPatterns && testMatchesPattern(this, testSkipPatterns)) {
431+
return true;
432+
}
433+
return false;
423434
}
424435

425436
/**
@@ -987,8 +998,8 @@ class TestHook extends Test {
987998
getRunArgs() {
988999
return this.#args;
9891000
}
990-
matchesTestNamePatterns() {
991-
return true;
1001+
willBeFiltered() {
1002+
return false;
9921003
}
9931004
postRun() {
9941005
const { error, loc, parentTest: parent } = this;
@@ -1016,7 +1027,7 @@ class Suite extends Test {
10161027
constructor(options) {
10171028
super(options);
10181029

1019-
if (testNamePatterns !== null && !options.skip) {
1030+
if (testNamePatterns !== null && testSkipPatterns !== null && !options.skip) {
10201031
this.fn = options.fn || this.fn;
10211032
this.skipped = false;
10221033
}
@@ -1050,7 +1061,12 @@ class Suite extends Test {
10501061
// tests that it contains - in case of children matching patterns.
10511062
this.filtered = false;
10521063
this.parent.filteredSubtestCount--;
1053-
} else if (testOnlyFlag && testNamePatterns == null && this.filteredSubtestCount === this.subtests.length) {
1064+
} else if (
1065+
testOnlyFlag &&
1066+
testNamePatterns == null &&
1067+
testSkipPatterns == null &&
1068+
this.filteredSubtestCount === this.subtests.length
1069+
) {
10541070
// If no subtests are marked as "only", run them all
10551071
this.filteredSubtestCount = 0;
10561072
}

Diff for: ‎lib/internal/test_runner/utils.js

+5
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,7 @@ function parseCommandLine() {
200200
let destinations;
201201
let reporters;
202202
let testNamePatterns;
203+
let testSkipPatterns;
203204
let testOnlyFlag;
204205

205206
if (isChildProcessV8) {
@@ -240,6 +241,9 @@ function parseCommandLine() {
240241
testNamePatternFlag,
241242
(re) => convertStringToRegExp(re, '--test-name-pattern'),
242243
) : null;
244+
const testSkipPatternFlag = getOptionValue('--test-skip-pattern');
245+
testSkipPatterns = testSkipPatternFlag?.length > 0 ?
246+
ArrayPrototypeMap(testSkipPatternFlag, (re) => convertStringToRegExp(re, '--test-skip-pattern')) : null;
243247
}
244248

245249
globalTestOptions = {
@@ -250,6 +254,7 @@ function parseCommandLine() {
250254
sourceMaps,
251255
testOnlyFlag,
252256
testNamePatterns,
257+
testSkipPatterns,
253258
reporters,
254259
destinations,
255260
};

Diff for: ‎src/node_options.cc

+3
Original file line numberDiff line numberDiff line change
@@ -667,6 +667,9 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
667667
"run test at specific shard",
668668
&EnvironmentOptions::test_shard,
669669
kAllowedInEnvvar);
670+
AddOption("--test-skip-pattern",
671+
"run tests whose name do not match this regular expression",
672+
&EnvironmentOptions::test_skip_pattern);
670673
AddOption("--test-udp-no-try-send", "", // For testing only.
671674
&EnvironmentOptions::test_udp_no_try_send);
672675
AddOption("--throw-deprecation",

Diff for: ‎src/node_options.h

+1
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,7 @@ class EnvironmentOptions : public Options {
177177
bool test_only = false;
178178
bool test_udp_no_try_send = false;
179179
std::string test_shard;
180+
std::vector<std::string> test_skip_pattern;
180181
bool throw_deprecation = false;
181182
bool trace_atomics_wait = false;
182183
bool trace_deprecation = false;

Diff for: ‎test/fixtures/test-runner/output/name_and_skip_patterns.js

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
// Flags: --test-skip-pattern=disabled --test-name-pattern=enabled
2+
'use strict';
3+
const common = require('../../../common');
4+
const {
5+
test,
6+
} = require('node:test');
7+
8+
test('disabled', common.mustNotCall());
9+
test('enabled', common.mustCall());
10+
test('enabled disabled', common.mustNotCall());

Diff for: ‎test/fixtures/test-runner/output/name_and_skip_patterns.snapshot

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
TAP version 13
2+
# Subtest: enabled
3+
ok 1 - enabled
4+
---
5+
duration_ms: *
6+
...
7+
1..1
8+
# tests 1
9+
# suites 0
10+
# pass 1
11+
# fail 0
12+
# cancelled 0
13+
# skipped 0
14+
# todo 0
15+
# duration_ms *

Diff for: ‎test/fixtures/test-runner/output/skip_pattern.js

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Flags: --test-skip-pattern=disabled --test-skip-pattern=/no/i
2+
'use strict';
3+
const common = require('../../../common');
4+
const {
5+
describe,
6+
it,
7+
test,
8+
} = require('node:test');
9+
10+
test('top level test disabled', common.mustNotCall());
11+
test('top level skipped test disabled', { skip: true }, common.mustNotCall());
12+
test('top level skipped test enabled', { skip: true }, common.mustNotCall());
13+
it('top level it enabled', common.mustCall());
14+
it('top level it disabled', common.mustNotCall());
15+
it.skip('top level skipped it disabled', common.mustNotCall());
16+
it.skip('top level skipped it enabled', common.mustNotCall());
17+
describe('top level describe', common.mustCall());
18+
describe.skip('top level skipped describe disabled', common.mustNotCall());
19+
describe.skip('top level skipped describe enabled', common.mustNotCall());
20+
test('this will NOt call', common.mustNotCall());

Diff for: ‎test/fixtures/test-runner/output/skip_pattern.snapshot

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
TAP version 13
2+
# Subtest: top level skipped test enabled
3+
ok 1 - top level skipped test enabled # SKIP
4+
---
5+
duration_ms: *
6+
...
7+
# Subtest: top level it enabled
8+
ok 2 - top level it enabled
9+
---
10+
duration_ms: *
11+
...
12+
# Subtest: top level skipped it enabled
13+
ok 3 - top level skipped it enabled # SKIP
14+
---
15+
duration_ms: *
16+
...
17+
# Subtest: top level describe
18+
ok 4 - top level describe
19+
---
20+
duration_ms: *
21+
type: 'suite'
22+
...
23+
# Subtest: top level skipped describe enabled
24+
ok 5 - top level skipped describe enabled # SKIP
25+
---
26+
duration_ms: *
27+
type: 'suite'
28+
...
29+
1..5
30+
# tests 3
31+
# suites 2
32+
# pass 1
33+
# fail 0
34+
# cancelled 0
35+
# skipped 2
36+
# todo 0
37+
# duration_ms *

Diff for: ‎test/parallel/test-runner-output.mjs

+2
Original file line numberDiff line numberDiff line change
@@ -119,8 +119,10 @@ const tests = [
119119
process.features.inspector ? { name: 'test-runner/output/lcov_reporter.js', transform: lcovTransform } : false,
120120
{ name: 'test-runner/output/output.js' },
121121
{ name: 'test-runner/output/output_cli.js' },
122+
{ name: 'test-runner/output/name_and_skip_patterns.js' },
122123
{ name: 'test-runner/output/name_pattern.js' },
123124
{ name: 'test-runner/output/name_pattern_with_only.js' },
125+
{ name: 'test-runner/output/skip_pattern.js' },
124126
{ name: 'test-runner/output/unfinished-suite-async-error.js' },
125127
{ name: 'test-runner/output/unresolved_promise.js' },
126128
{ name: 'test-runner/output/default_output.js', transform: specTransform, tty: true },

0 commit comments

Comments
 (0)