Skip to content

test: Stryker Jest runner with config workarounds#208

Closed
brody2consult wants to merge 4 commits into
xmldom:masterfrom
brody2consult:stryker-jest-runner
Closed

test: Stryker Jest runner with config workarounds#208
brody2consult wants to merge 4 commits into
xmldom:masterfrom
brody2consult:stryker-jest-runner

Conversation

@brody2consult

@brody2consult brody2consult commented Mar 31, 2021 •
edited
Loading

Copy link
Copy Markdown
Member

updated: I reconfigured by removing stryker.conf.json from my work area, running npx stryker init again, and re-patching stryker.conf.json with some of the configuration from the master branch. The reconfiguration gave me the coverageAnalysis entry changed to perTest.

I was hoping to improve the speed on GitHub. I tried with and without this update on cloud instances with 2 vCPU, for 2 test runner processes, which seems to reflect what happens on GitHub. But it took about half minute longer with the Jest runner, which is insignificant considering that it takes almost 2 hours on GitHub with 2 test runner processes.

I had to update jest.config.js to configure testMatch instead of testRegex. From the debug tracing command, as directed when Stryker fails, it looked to me like with the Jest runner it was messing up the configuration of testRegex. I suspect that this is why the jest-runner did not work in PR #140 (update to Stryker 4). I think it would be ideal if we could get this fixed on Stryker at some point. My changes may need cleanup, and I am not sure if we need to configure testPathIgnorePatterns or not.

Test time was almost the same, maybe half a minute longer with the Jest runner. See stryker-mutator/stryker-js#1514 (comment):

Mutation switching by itself will not have a significant impact on Jest projects unfortunately. You might have some performance improvement because Jest would be able to cache build results, but definitely not an order of magnitude. However, it does open up the way for coverage analysis in jest. Hot reload should theoretically also be possible with Jest, but not with their current public api alone. We'll be focussing on jest next, after we've released v4.

End results with command runner

Mutation testing  [==================================================] 100% (elapsed: ~1h 53m, remaining: n/a) 2014/2014 tested (695 survived, 41 timed out)
[...]
Ran 0.98 tests per mutant on average.
---------------|---------|----------|-----------|------------|----------|---------|
File           | % score | # killed | # timeout | # survived | # no cov | # error |
---------------|---------|----------|-----------|------------|----------|---------|
All files      |   65.49 |     1278 |        41 |        695 |        0 |       0 |
 dom-parser.js |   80.00 |      156 |         0 |         39 |        0 |       0 |
 dom.js        |   65.81 |      590 |        24 |        319 |        0 |       0 |
 entities.js   |    7.85 |       19 |         0 |        223 |        0 |       0 |
 sax.js        |   82.30 |      513 |        17 |        114 |        0 |       0 |
---------------|---------|----------|-----------|------------|----------|---------|
[...]
23:51:59 (1131) INFO MutationTestExecutor Done in 113 minutes 37 seconds.

End results with Jest runner

Mutation testing  [==================================================] 100% (elapsed: ~1h 53m, remaining: n/a) 1832/1832 tested (513 survived, 92 timed out)
[...]
Ran 183.07 tests per mutant on average.
---------------|---------|----------|-----------|------------|----------|---------|
File           | % score | # killed | # timeout | # survived | # no cov | # error |
---------------|---------|----------|-----------|------------|----------|---------|
All files      |   65.42 |     1223 |        92 |        513 |      182 |       4 |
 dom-parser.js |   80.00 |      149 |         7 |         24 |       15 |       0 |
 dom.js        |   65.81 |      585 |        29 |        168 |      151 |       0 |
 entities.js   |    7.85 |       19 |         0 |        223 |        0 |       0 |
 sax.js        |   82.19 |      470 |        56 |         98 |       16 |       4 |
---------------|---------|----------|-----------|------------|----------|---------|
[...]
23:55:19 (1135) INFO MutationTestExecutor Done in 114 minutes 11 seconds.

I would also like to investigate the 4 errors on sax.js at some point.

@brody2consult brody2consult requested a review from karfau March 31, 2021 04:43
brody2consult
brody2consult commented Mar 31, 2021
View reviewed changes
Comment thread stryker.conf.json
brody2consult
brody2consult commented Apr 18, 2021
View reviewed changes
Comment thread jest.config.js
@brody2consult brody2consult changed the title test: configure Stryker to use Jest runner test: configure to use Stryker 4.5.1 with Jest runner Apr 18, 2021
@brody2consult

brody2consult commented Apr 18, 2021

Copy link
Copy Markdown
Member Author

As I said in PR #220 Stryker 4.6.0 update seems to run much slower, proposing to use Jest runner with Stryker 4.5.1 for now at least.

@brody2consult brody2consult mentioned this pull request May 25, 2021
Merged
1 task
@karfau karfau changed the title test: configure to use Stryker 4.5.1 with Jest runner test: configure to use Stryker with Jest runner Jun 9, 2021
@brody2consult brody2consult changed the title test: configure to use Stryker with Jest runner test: Stryker Jest runner with config workarounds Aug 24, 2021
@brody2consult

brody2consult commented Aug 24, 2021

Copy link
Copy Markdown
Member Author

In terms of status: I have tested this with Stryker 4, not sure if it will work with Stryker 5 or not. I am also not sure whether or not these workarounds are needed with Stryker 5.x as well.

In case we do still need these workarounds with Stryker 5.x, I think it would be good to investigate why.

@brody2consult brody2consult mentioned this pull request Aug 24, 2021
Merged
@karfau

karfau commented Jul 20, 2023

Copy link
Copy Markdown
Member

We are no longer using stryker mutation testing, so this PR is obsolete

@karfau karfau closed this Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@karfau karfau Awaiting requested review from karfau

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@brody2consult @karfau