Skip to content

GitHub Actions: Try to fix storybook smoke tests #71284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
t-hamano merged 7 commits into from
Aug 23, 2025
Merged

GitHub Actions: Try to fix storybook smoke tests #71284

t-hamano merged 7 commits into from
Aug 23, 2025

Conversation

@t-hamano
Copy link
Contributor

@t-hamano t-hamano commented Aug 21, 2025 •
edited
Loading

What?

We re-enable the storybook smoke test, but somehow, the CI started to fail again with the following error:

Details 
Run npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
  npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
  "npx http-server ./storybook/build --port 50240 --silent" \
  "npx wait-on tcp:127.0.0.1:50240 && \
  NODE_PATH=./node_modules \
  npx --package=@storybook/[email protected] -- \
  test-storybook --url http://localhost:50240 --config-dir ./storybook"
  shell: /usr/bin/bash -e {0}
[SB] npm warn exec The following package was not found and will be installed: [email protected]
[TEST] npm warn exec The following package was not found and will be installed: @storybook/[email protected]
[TEST] npm warn deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
[TEST] npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
[TEST] npm warn deprecated [email protected]: Glob versions prior to v9 are no longer supported
[SB] (node:6758) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
[SB] (Use `node --trace-deprecation ...` to show where the warning was created)
[TEST] Test Suites: 0 of 157 total
[TEST] Tests:       0 total
[TEST] Snapshots:   0 total
[TEST] Time:        0.216 s
[TEST] Ran all test suites.
[TEST] Error: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium_headless_shell-1187/chrome-linux/headless_shell
[TEST] ╔═════════════════════════════════════════════════════════════════════════╗
[TEST] ║ Looks like Playwright Test or Playwright was just installed or updated. ║
[TEST] ║ Please run the following command to download new browsers:              ║
[TEST] ║                                                                         ║
[TEST] ║     npx playwright install                                              ║
[TEST] ║                                                                         ║
[TEST] ║ <3 Playwright Team                                                      ║
[TEST] ╚═════════════════════════════════════════════════════════════════════════╝ Failed to launch browser.
[TEST]     at executablePathOrDie (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/registry/index.js:517:15)
[TEST]     at Object.executablePathOrDie (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/registry/index.js:548:45)
[TEST]     at Chromium._prepareToLaunch (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:184:39)
[TEST]     at async Chromium._launchProcess (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:196:22)
[TEST]     at async Chromium._innerLaunch (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:112:70)
[TEST]     at async Chromium._innerLaunchWithRetries (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:99:14)
[TEST]     at async /home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/browserServerImpl.js:77:18
[TEST]     at async ProgressController.run (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/progress.js:78:22)
[TEST]     at async BrowserServerLauncherImpl.launchServer (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/browserServerImpl.js:68:17)
[TEST]     at async BrowserType.launchServer (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/client/browserType.js:73:12)
[TEST] npx wait-on tcp:127.0.0.1:50240 && NODE_PATH=./node_modules npx --package=@storybook/[email protected] -- test-storybook --url http://localhost:50240 --config-dir ./storybook exited with code 1
--> Sending SIGTERM to other processes..
[SB] npx http-server ./storybook/build --port 50240 --silent exited with code null
Error: Process completed with exit code 1.

CI example: https://github.com/WordPress/gutenberg/actions/runs/17112981527/job/48538253515
Actions: https://github.com/WordPress/gutenberg/actions/workflows/storybook-check.yml

The CI failures started around the same time as the new Playwright v1.55.0 was released, so I'm sure that the Playwright update is affecting us.

Why? How?

I believe we pinned all libraries to local versions with this PR, but maybe I missed something. Updating the Playwright version may solve the problem, but it won't be a fundamental solution.

I'm testing what happens if we explicitly install Chromium: #71284

As this comment says, the underlying problem may be that the Playwright version, and therefore the Chromium version, is different.

Installing the test runner first seems to resolve the Chromium version differences.

Note: In the trunk branch, the Playwright version has been updated to 1.55, so this issue does not occur (See: #71285). The purpose of this PR is to attempt to fix the fundamental problem, and the Playwright version in this PR is still 1.54.2.

@t-hamano t-hamano mentioned this pull request Aug 21, 2025
Merged
@t-hamano t-hamano added Storybook Storybook and its stories for components GitHub Actions Pull requests that update GitHub Actions code labels Aug 21, 2025
@t-hamano t-hamano mentioned this pull request Aug 21, 2025
Closed
@t-hamano t-hamano added [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. and removed GitHub Actions Pull requests that update GitHub Actions code labels Aug 21, 2025
@t-hamano t-hamano self-assigned this Aug 21, 2025
@github-actions
Copy link

github-actions bot commented Aug 21, 2025 •
edited
Loading

Flaky tests detected in 7127ab8.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/17115321910
📝 Reported issues:

@Mamaduka
Copy link
Member

Mamaduka commented Aug 21, 2025 •
edited
Loading

Here's the playwright update PR #71285.

Update: Storybook CI is passing on Playwright PR - https://github.com/WordPress/gutenberg/actions/runs/17116365106/job/48548184628?pr=71285. I think the problem was a browser version mismatch.

Screenshot

CleanShot 2025-08-21 at 07 49 39

@t-hamano t-hamano mentioned this pull request Aug 21, 2025
Merged
@t-hamano
Copy link
Contributor Author

t-hamano commented Aug 21, 2025

I suspect there may be a version difference between the Playwright installed in the Gutenberg project and the one installed by the test-runner.

I'd like to test to see if we can match the two versions.

@t-hamano
Copy link
Contributor Author

t-hamano commented Aug 21, 2025

I push an additional commit, but somehow it's not yet applied to this PR..

image

It might be a GitHub issue itself, so I'd like to wait for a moment: https://www.githubstatus.com/

@t-hamano t-hamano marked this pull request as ready for review August 21, 2025 08:33
@t-hamano t-hamano requested a review from desrosj as a code owner August 21, 2025 08:33
@github-actions
Copy link

github-actions bot commented Aug 21, 2025 •
edited
Loading

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: t-hamano <[email protected]>
Co-authored-by: tyxla <[email protected]>
Co-authored-by: Mamaduka <[email protected]>
Co-authored-by: priethor <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@priethor
Copy link
Contributor

priethor commented Aug 21, 2025

@t-hamano it seems to be working now 🎉

tyxla
tyxla approved these changes Aug 22, 2025
View reviewed changes
Copy link
Member

@tyxla tyxla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good 👍 Thank you for resolving it @t-hamano 🙌

@t-hamano
Copy link
Contributor Author

t-hamano commented Aug 23, 2025

@tyxla Thanks for the review!

Note: The failing CI is a known issue and has been fixed in trunk: #71293

@t-hamano t-hamano merged commit 9b0a3be into trunk Aug 23, 2025
97 of 102 checks passed
@t-hamano t-hamano deleted the storybook-install-chromium branch August 23, 2025 03:54
@github-actions github-actions bot added this to the Gutenberg 21.6 milestone Aug 23, 2025
@maxschmeling maxschmeling mentioned this pull request Oct 7, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tyxla tyxla tyxla approved these changes

@desrosj desrosj Awaiting requested review from desrosj desrosj is a code owner

Assignees

@t-hamano t-hamano

Labels

Storybook Storybook and its stories for components [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.

Projects

None yet

Milestone

Gutenberg 21.6

Development

Successfully merging this pull request may close these issues.

5 participants

@t-hamano @Mamaduka @priethor @tyxla