TestBed.configureTestingModule Performance Issue

I'm submitting a ... (check one with "x")

[ ] bug report 
[x] feature request
[ ] support request

Current behavior

Importing modules into TestBed.configureTestingModule can slow tests down substantially. We have a 'shared module' with parts, that lots of our components use, including a third party library or two. It is very convenient to import this module as a rather than cherry pick in the tests, however testing a small component yesterday, I saw test bootstrap taking two whole seconds and had to cherry-pick dependencies to make tests run reliably.

Expected behavior

TestBed.configureTestingModule should be performant.

Minimal reproduction of the problem with instructions

In this plunker

go to the src/simple.spec.ts and comment and un-comment the SharedModule import into configureTestingModule.
Observe time it takes for test to run.

I am seeing a jump from 0.079s to 0.241s. (~= 3x slower). Multiply that by 5 cases and this simple test takes a whole second. Make the shared module larger and you have real problems.

What is the motivation / use case for changing the behavior?

Slow tests hurt TDD workflow
Slow enough tests disconnect karma from the browser and fail the suite.

Please tell us about your environment:

Win10, VsCode, Webpack,

Angular version: 2.0.X

Yes.
Browser: Locally, PhantomJS. Plunk tested in Chrome.
Language: TypeScript (though tests are transpiled to ES5 prior to running, naturally)

Note: Even as I write this, I wonder if it's a reasonable expectation for large modules to go through TestBed quickly? Maybe I'm just doing something wrong architecturally, (use multiple, smaller shared modules, not one big one, etc.) However, tests are time-consuming enough without having to track down every individual dependency of a more complex component just to get started.

Seems like the goal is really to have a fresh component instance each time, and for the most part the testing module is going to be static for a particular .spec file. It also seems like configuring the module at each iteration is the part that is really adding on the time. A better pattern could be configuring the module once, then testModule.createComponent multiple times.

TestBed.configureTestingModule once and calling TestBed.createComponent on that module multiple times doesn't work, though... maybe I'm missing an obvious way to preserve module state between iterations?

Edit: FWIW I tried logging the timestamp and it looks like most of the time passes during TestBed.createComponent, not TestBed.configureTestingModule - however I still suspect it's related to module configuration, since adding things to imported modules adds to the time, whether or not those imports are used in the component being tested.

I have a noticed unit test execution performance issues that seem directly related to this issue.

I have an ionic 2 / angular 2 app. When trying to unit test my components using TestBed and ComponentFixture classes, there is easily 1-2 seconds spent in the setup of each test that seems directly related to the imports: [IonicModule.forRoot(MyComponent)] that I've added to my testing module via the TestBed.configureTestingModule({...}) method.

When I remove the import, my tests run much faster, but they obviously fail because my component's template no longer compiles.

@vmandy

@vmandy I've been working around this by doing one or more of:

spending more time curating just what I need in the test module
mocking the component dependencies that are time-consuming to bootstrap
overriding the template of the component I'm testing so it doesn't depend on much.

All of which are time consuming and sub-optimal 👎 (except from the point of view that I am really forced to isolate the unit I am testing?)

It seems like there are some theoretical optimizations to this at least, because I know these components are not taking 1-2 secs to bootstrap at run-time.

Iam having similar problems. Thought it was caused by wallaby, but it seems to be the TestBed.

wallabyjs/public#885

I am seeing the same issue. Any unit test with component compilation is taking close to 1sec...

Unfortunately I was pulled off the Angular 2 project and haven't looked at this in quite some time. It seems like a lot of the complications of TestBed (test-module setup, generally, not just performance) could be avoided by an approach similar to the enzyme.shallow render approach in the React ecosystem. - where just one unit is rendered and none of it's dependencies - the same thing can be accomplished by over-riding an Angular component's template in the .spec, but that is often tedious/time-consuming as well.

we have close to 2000 test cases getting executed just under 1 min in Angular 1 where Angular 2 takes 1min to execute just 100 test cases. It would be nice if we can get a resolution for this issue as it hinders the testing capabilities in large applications.

@mlakmal

@mlakmal I think there are lots of folks relying heavily on e2e tests, or just testing functional logic in component classes

let component = new ComponentClass(mockDep1, mockDep2);
expect(component.doAddition(2,2)).toBe(4);

Which doesn't test any of the template rendering (template logic)... but it does allow easy testing of class logic at least

@ollwenjones

@ollwenjones thanks, i am going to try that out. i really doesn't want to test any template logic since most of that gets covered by our automated tests, so just testing the component class code should be enough.

@mlakmal

@mlakmal sweet! Makes me happy to contribute something that actually helps someone. 🍻 - FWIW I'm often torn about template-rendering unit tests, because it's so easy to slip into testing the framework.

@ollwenjones

I'm in the process of upgrading a pretty big project from rc.4 to 2.3, so it's the first time we're using the NgModule approach.

We have ~1000 unit tests, with maybe 1/2 of those testing components. Prior to upgrade, tests run pretty quickly - around 30s.

After upgrade, they take at leat twice as long, but the worst thing is that they seem kind of "flaky" - Karma will periodically lost the connection to Chrome and often I will need to refresh Chrome multiple times. I suspect that something is timing out but haven't isolated it yet.

@ollwenjones regarding "shallow" component tests, did you know about the NO_ERRORS_SCHEMA that you can set in the testing module?

import { NO_ERRORS_SCHEMA} from '@angular/core';
// ...
TestBed.configureTestingModule({
  declarations: [ /*... whatever */ ],
  schemas: [NO_ERRORS_SCHEMA]
});

This means that the compiler just ignores any elements it does not recognize, meaning you do not need to declare all the components used in the template of the component under test.

Found the same issue in Angular's tests: #13500
I was able to improve the situation by creating bespoke testing modules for each test.

@michaelbromley

@michaelbromley 😮 I had no idea about NO_ERRORS_SCHEMA would have / will probably save me hours - also an opportunity to hopefully speed up some of these tests, as the leaner the test-module the faster TestBed seems to go.

@MatMercer

@MatMercer

Since I'm using 5.2.0, I don't have the "recent" features.

Be aware that angular versions are only supported for a limited amount of time [docs]
And also, if the Angular team improves the test setup performance like you want, it would be on a
newer version, so you would have to upgrade anyways, there's really not workaround*.

On another Issue about something similar, someone mentioned that the next Angular release with Ivy and R3SomethingTestBed made test setup run much faster even without AOT.
angular/angular-cli#6650

I found a workaround, it's not pretty, basically you use something like git diff HEAD^ --name-only and process the data to generate something that you can pass to const context = require.context(... in the test.ts file, but that means you have to be regenerating the HTML file every time (which I was doing using a gulp task watching changes for TS/HTML files)

(Disclaimer: I'm not a member of Angular team)

@giniedp

I'm currently refactoring my component tests (49 files) to use ng-bullet that I've added manually from here into my project as @giniedp said. The snippet @giniedp gave is a little bit outdated so I'm giving a warning to the newcomers: use the ng-bullet from here, it's updated and worked with my angular 5.2.0.

@Goodwine Indeed, what you have said is right, if they will give performance improvements in the next releases there is no "workarounds", we will need to upgrade it anyway.

After I finish the refactor I'll share the results here, I'll probably use karma-parallel too.

it is actually the very same implementation as the latest version of ng-bullet, except that i choose to skip await TestBed.compileComponents().

@giniedp

@giniedp

The signature of your snippet that gave me compilation errors:

export function configureTestSuite(configureAction: () => Promise<any>) {

The signature of the ng-bullet project:

export const configureTestSuite = (configureAction?: () => void) => {

Probably the way we used them in our tests is different (I'm using based in the ng-bullet npm page), that's why your snippet didn't worked for me.

you are right. That is another minor difference in how we adopted from ng-bullet. The change on your side would have been

-   configureTestSuite(() => {
+   configureTestSuite(async () => {
        TestBed.configureTestingModule({
            // ...
        })
    });

for us it was important not to take away the flow control from the developer, so he is still able to do async stuff inside configureTestSuite

    configureTestSuite(async () => {
        await TestBed.configureTestingModule({
            // ...
        }).compileComponents()
        await moreAsyncStuff()
    });

@giniedp

@giniedp @Goodwine

That's interesting @giniedp. In my use-case we don't care about the flow control.

I have to say thanks! I successfully added ng-bullet in the project and the tests execution went from 5m:30s to 30s. The problem I had with the peer dependencies before was because my node version was too outdated.

Adding karma-parallel didn't did much (28s instead of 30s) and broke a test so I didn't used it.

@Goodwine

@Goodwine

On another Issue about something similar, someone mentioned that the next Angular release with Ivy and R3SomethingTestBed made test setup run much faster even without AOT.

Yes. enabledIvy: true (it's tsconfig.json's option, see https://angular.io/guide/ivy#opting-into-angular-ivy) configures to use TestBedRender3 as the TestBed implementation. It's TestBed for Render 3, a.k.a. Ivy. And Ivy JiT compiler has caching mechanism for compiled modules and components. ( See https://github.com/angular/angular/blob/8.2.14/packages/core/src/render3/jit/module.ts#L49-L66 ).

So if you turn Ivy on, TestBed.configureTestingModule(...).compileComponents() gets faster because the second or later compilation for the same components are skipped.

With Ivy, the TestBed infrastructure has been overhauled and no longer requires recompilation of testing modules. I have seen significant speedups (8x) in tests because of this change.

@JoostK

@JoostK Good to hear!

Would nice to have an overview of the performance differences between:

Angular 8 with Karma
Angular 8 with Jest
Angular 9 with Karma (+Ivy)
Angular 9 with Jest (+Ivy)

@JoostK

Hello everyone,

As @JoostK mentioned, Ivy TestBed was overhauled to avoid unnecessary recompilations between tests, which helped improve overall performance. We did some measurements to compare ViewEngine and Ivy TestBed versions using Material Components test suite, Angular.io app and Angular Framework unit tests - performance improvement was around 30-40%. We also received feedback from developers that they see even better results on their projects.

I’m closing this ticket and feel free to create new ones if you notice TestBed performance issues after switching to Ivy.

Thank you.

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

Navigation Menu

Repository navigation

TestBed.configureTestingModule Performance Issue #12409

95 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TestBed.configureTestingModule Performance Issue #12409

Description

Activity

ollwenjones commented on Oct 28, 2016

vmandy commented on Nov 15, 2016

ollwenjones commented on Nov 15, 2016

FlemmingBehrend commented on Dec 1, 2016

mlakmal commented on Dec 15, 2016

ollwenjones commented on Dec 15, 2016

mlakmal commented on Dec 15, 2016

ollwenjones commented on Dec 15, 2016

mlakmal commented on Dec 15, 2016

ollwenjones commented on Dec 15, 2016

michaelbromley commented on Dec 15, 2016

marclaval commented on Dec 15, 2016

ollwenjones commented on Dec 15, 2016

95 remaining items

Goodwine commented on Dec 4, 2019

MatMercer commented on Dec 4, 2019

giniedp commented on Dec 4, 2019

MatMercer commented on Dec 4, 2019

giniedp commented on Dec 4, 2019

MatMercer commented on Dec 4, 2019

Quramy commented on Dec 5, 2019

JoostK commented on Mar 15, 2020

krokofant commented on Mar 15, 2020

AndrewKushnir commented on May 29, 2020

angular-automatic-lock-bot commented on Jun 30, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions