For the past few weeks our nightly firefox tests have been failing. At first glance which tests are failing is flakey and not consistent, but the cause is always a test timeout. I dug into what was going on and from my findings I discovered that running karma and firefox is the cause.
For some reason, the karma integration is adding a 2-3 second delay to our tests. I have logged performance metrics everywhere in our code and no where causes a 2 second delay. Furthermore, opening the tests in an actual Firefox browser using npm run test:debug removes the 2 second delay. Additionally, when I tested moving to a different test runner such as playwright and Web Test Runner in Firefox the 2 second delay was gone as well.
The test failure problem is that Nightly Firefox has increased that delay to 4-6 seconds, which exceeds our test timeout.
To fix this we're going to have to migrate off Karma. Karma itself is deprecated and recommends Web Test Runner to run tests in an actual browser (as opposed to jest which run in JSDOM). A quick test shows it could work for us. However, one problem I foresee is that WTR uses Playwright as the default runner, and Playwright does not support Nightly Firefox. We'll probably need to use their Webdriverio/js runner instead just so we can run that.
For the past few weeks our nightly firefox tests have been failing. At first glance which tests are failing is flakey and not consistent, but the cause is always a test timeout. I dug into what was going on and from my findings I discovered that running karma and firefox is the cause.
For some reason, the karma integration is adding a 2-3 second delay to our tests. I have logged performance metrics everywhere in our code and no where causes a 2 second delay. Furthermore, opening the tests in an actual Firefox browser using
npm run test:debugremoves the 2 second delay. Additionally, when I tested moving to a different test runner such as playwright and Web Test Runner in Firefox the 2 second delay was gone as well.The test failure problem is that Nightly Firefox has increased that delay to 4-6 seconds, which exceeds our test timeout.
To fix this we're going to have to migrate off Karma. Karma itself is deprecated and recommends Web Test Runner to run tests in an actual browser (as opposed to jest which run in JSDOM). A quick test shows it could work for us. However, one problem I foresee is that WTR uses Playwright as the default runner, and Playwright does not support Nightly Firefox. We'll probably need to use their Webdriverio/js runner instead just so we can run that.