Skip to content

Upgrade dev dependencies (vitest 5, Playwright 1.63) and fix the collector for Playwright 1.63 - #18

Merged
benjithompson merged 3 commits into
mainfrom
chore/upgrade-deps
Sep 22, 2026
Merged

benjithompson merged 3 commits into
mainfrom
chore/upgrade-deps

Conversation

@benjithompson

@benjithompson benjithompson commented Sep 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR upgrades the dev dependencies. It also fixes the collector, because Playwright 1.63 broke it. After the upgrade, npm audit finds 0 vulnerabilities (before: 7, with 1 critical and 2 high).

Package Before After
vitest (root) 2.1.9 5.0.1
tsx (root) 4.23.1 4.23.15
@types/node (dashboard) 22.20.1 22.20.4
@playwright/test (collector, dev) 1.61.1 1.63.0
typescript (root) 5.9.3 5.9.3 (no change)

The package-lock.json file is regenerated. The old lockfile pinned vite 5, and vitest 5 needs vite 6 or later.

Why Playwright 1.63 broke the collector

Cause (measured). Playwright 1.63 no longer turns off Chromium's RenderDocument feature (microsoft/playwright#41303). With RenderDocument on, each navigation gives the new document a new frame. A binding call from the old document's pagehide then never reaches Node.

  • A minimal probe with only exposeBinding and addInitScript receives 4 of 4 pagehide calls on 1.61.1 and 0 of 4 on 1.63.0.
  • Playwright 1.63 with the older Chromium build also receives 0. So the Playwright driver causes the change, not the browser.
  • With --disable-features=RenderDocument, 1.63 receives all 4 again.

Effect. 18 collector tests failed. Intermediate Samples were lost, INP was missing, and routes landed on the wrong Navigation. Users who run the collector on Playwright 1.63 get the same losses. The peer range >=1.40.0 <2 includes 1.63.

Why the collector does not turn RenderDocument off. Playwright does not merge a user's --disable-features flag with its own list, so a second flag can remove Playwright's entries, for example PaintHolding. Also, RenderDocument is now how Chromium works.

The fix (packages/collector/src/index.ts)

  • Test-driven navigations flush first. The collector already overrides the page fixture. It now wraps goto, reload, goBack and goForward, and flushes the current document before each one. The flush uses the same bounded paint wait as teardown. It also waits for the latest input's event-timing entry, for at most 300 ms after that input. That entry exists only after a presented frame, which comes a few ms after the paint (measured).
  • Link-click and script navigations flush at beforeunload. At that time, the old document is still the frame's current document. pagehide stays as a later reading on engines that keep the frame, such as Firefox.
  • One Sample per document. A document can now send more than one reading. Node merges them by docId. It writes the Sample at the fixture-driven flush, or 400 ms after the first reading. The kill test confirms that an intermediate Sample is on disk before a SIGKILL.
  • INP is recalculated on every drain, so a later, worse interaction still counts.

Additional result: the cross-site limitation is gone. The limitation in the upload kit has the same cause, a frame swap. The new test/crosssite.test.ts checks both Samples and their routes for a cross-site goto and a cross-site link click. It fails on the old collector (4 of 6) and passes with the fix.

Test fixture fix (separate commit)

The fixture page inserted its layout-shifting element 50 ms after its script started. Under 1.63 with parallel load, the first paint sometimes came 380 to 728 ms later (measured), so no shift occurred, and the test waited until its timeout. The same problem occurred with the original collector code. The shift now happens 50 ms after first-contentful-paint.

Docs

  • Root README: I removed the cross-site bullet and added a note about the flush before navigation.
  • Upload kit README and example spec: I updated the cross-site notes.

Test results (measured on macOS, Node 23.11)

Check Result
npm run typecheck passes
Collector on Playwright 1.63.0 181 passed, 2 skipped. 7 full runs in a row, no failures.
Collector on Playwright 1.61.1 175 passed, 2 skipped. 2 of 2 full runs pass (before the 6 cross-site tests were added).
Dashboard 157 passed, 7 skipped
npm audit 0 vulnerabilities

Notes

  • vitest 5 supports Node 22.12+, 24 and 26+. Node 23 gives an EBADENGINE warning, but the tests pass.
  • Behavior change for users: each goto, reload, goBack and goForward now waits for the flush of the current page first. The wait is usually a few ms. It is bounded by the FCP wait (1.5 s, only when the page never painted), the paint wait (300 ms) and the input wait (300 ms).
  • I did not test the fix on a BlazeMeter Engine. That needs a live run with the upload kit.

🤖 Generated with Claude Code

benjithompson and others added 3 commits September 22, 2026 14:52
- vitest 2.1.9 -> 5.0.1 (fixes the critical vitest and high vite/nanoid advisories)
- tsx 4.23.1 -> 4.23.15
- @types/node 22.20.1 -> 22.20.4 (dashboard)
- package-lock.json regenerated: the old lock pinned vite 5, which vitest 5 cannot peer

@playwright/test stays on 1.61.1. Playwright 1.63.0 fails 18 collector
tests (INP capture, pagehide flush, route(), Sample counts per Navigation);
that needs its own investigation.

npm audit: 7 vulnerabilities -> 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Content inserted before anything is painted moves nothing, so the browser
honestly reports no shift. Under Playwright 1.63 with parallel load, first
paint landed 380-728ms after the script ran in about 1 of 12 page loads
(measured), so vitals.spec's wait for a layout-shift entry hung to its
timeout. The shift now fires 50ms after first-contentful-paint, with a 1s
backstop.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…cument)

Playwright 1.63 no longer disables Chromium's RenderDocument (PR
microsoft/playwright#41303), so every navigation swaps the document's frame.
A binding call from the old document's pagehide never reaches Node, and 18
collector tests failed: intermediate Samples, INP and routes were lost.

- Test-driven goto/reload/goBack/goForward flush the current document first,
  with the teardown flush's bounded paint wait, plus a bounded wait for the
  latest input's event-timing entry (it needs a presented frame).
- Link-click and script navigations flush at beforeunload, while the document
  is still the frame's current one. pagehide stays for engines that keep the
  frame (Firefox, Chromium without RenderDocument).
- A document may report more than once. Node merges its readings by docId
  into one Sample and writes it at the fixture-driven flush or 400ms after
  the first reading.
- INP is recomputed on every drain, so a later, worse interaction still wins.

This also lifts the documented cross-site limitation (renderer-process swap,
same mechanism): new crosssite.test.ts pins both Samples and their routes for
cross-site goto and link-click journeys. It fails on the old collector.

@playwright/test devDependency: 1.61.1 -> 1.63.0. The suite passes on both
1.61.1 and 1.63.0.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@benjithompson benjithompson changed the title Upgrade dev dependencies: vitest 5, clear npm audit findings Upgrade dev dependencies (vitest 5, Playwright 1.63) and fix the collector for Playwright 1.63 Sep 22, 2026
@benjithompson
benjithompson merged commit ae95442 into main Sep 22, 2026
1 check passed
@benjithompson
benjithompson deleted the chore/upgrade-deps branch September 22, 2026 22:28
benjithompson added a commit that referenced this pull request Sep 22, 2026
…ght 1.63 fix (#21)

PR #18 fixed the collector for Playwright 1.63 but kept version 0.2.0.
npm 0.2.0 was published before that fix. Without a bump, the v0.4.0
release file and npm 0.2.0 would be two different builds with one
version number. 0.2.1 gives the fixed build its own version.

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant