feat(automation): Add workflow to vendor and update Test262 tests [pt 4/5] by jcscottiii · Pull Request #56843 · web-platform-tests/wpt

jcscottiii · 2025-12-18T03:36:53Z

This commit introduces a GitHub Actions workflow to automate the vendoring of the Test262 test suite into third_party/test262. This automation is the final piece of the integration, ensuring that the tests can be kept up-to-date with the upstream tc39/test262 repository.

The workflow is configured to run weekly or can be triggered manually. It performs the following steps:

Checks out both the WPT and tc39/test262 repositories.
Selectively copies a subset of the Test262 tests into WPT.
Copies the essential Test262 harness files.
Creates a vendored.toml file at the root of the vendored directory, recording the specific commit SHA from tc39/test262 that the files were sourced from. This ensures traceability, as described in the RFC.
Commits the updated files and opens a pull request with the changes.

Initially, only a subset of Test262 tests related to Interop 2026 features (Temporal and top-level-await) are being imported. This is a temporary measure to manage the volume of new test results and to avoid overloading wpt.fyi, which currently has limitations in processing the full Test262 result set (see wpt-fyi/issues/4681). The workflow can be expanded to include more tests in the future.

This work directly supports the integration of Test262 into WPT as detailed in the RFC: web-platform-tests/rfcs#229

This commit is part of a series of smaller PRs split from the larger, original implementation in #55997.

DanielRyanSmith · 2025-12-18T18:56:13Z

.github/workflows/update-test262-tests.yml

+        run: |
+          LATEST_SHA=$(git -C test262-spec rev-parse HEAD)
+          echo "Latest remote Test262 SHA: $LATEST_SHA"
+
+          rm -rf wpt/third_party/test262/test/
+          mkdir -p wpt/third_party/test262/test/
+          mkdir -p wpt/third_party/test262/harness/
+
+          # Selectively copy only Interop 2026 feature tests for now
+          # See: https://github.com/web-platform-tests/wpt.fyi/issues/4681
+          # Mapping of web feature to test directory can be found at: https://github.com/tc39/test262/blob/main/WEB_FEATURES.yml
+          # Temporal tests
+          mkdir -p wpt/third_party/test262/test/built-ins/Date/prototype/toTemporalInstant
+          mkdir -p wpt/third_party/test262/test/built-ins/Temporal
+          mkdir -p wpt/third_party/test262/test/intl402/Temporal
+          cp -r test262-spec/test/built-ins/Date/prototype/toTemporalInstant/* wpt/third_party/test262/test/built-ins/Date/prototype/toTemporalInstant
+          cp -r test262-spec/test/built-ins/Temporal/* wpt/third_party/test262/test/built-ins/Temporal
+          cp -r test262-spec/test/intl402/Temporal/* wpt/third_party/test262/test/intl402/Temporal
+
+          # Top-level-await tests
+          mkdir -p wpt/third_party/test262/test/language/module-code/top-level-await
+          cp -r test262-spec/test/language/module-code/top-level-await/* wpt/third_party/test262/test/language/module-code/top-level-await
+
+          # Always sync the harness files
+          rsync -a --delete test262-spec/harness/ wpt/third_party/test262/harness/
+          printf "[test262]\nsource = \"https://github.com/tc39/test262\"\nrev = \"${LATEST_SHA}\"\n" > wpt/third_party/test262/vendored.toml
+      - name: Commit changes
+        id: commit
+        continue-on-error: true
+        run: |
+          cd wpt
+          export BRANCH_NAME="$BRANCH_PREFIX-$(date +'%Y%m%d%H%M%S')"
+          echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
+          git config user.name "$GIT_AUTHOR_NAME"
+          git config user.email "$GIT_AUTHOR_EMAIL"
+          git checkout -B $BRANCH_NAME
+          git add third_party/test262/
+          git commit -m "$COMMIT_TITLE"


Optional nit: The scripts are somewhat long and may be subject to change. Would it be better to move some of this to a separate file?

Strong agree with this; I don't think we should have non-trivial scripts directly in GH actions files.

DanielRyanSmith

This suggestion came from a Gemini code review:

Focus: The "Selectively copies" step.
Critique:
The work is described as "selectively copying" files from Test262 to WPT.

The Risk: If a test file is deleted or renamed in the upstream tc39/test262 repository, a simple copy operation (like cp -r or checkout-index) will not remove the old file from your third_party/test262 directory. Over time, your vendored directory will accumulate "zombie tests"—files that no longer exist upstream but persist in WPT, potentially causing confusing failures or bloat.
Suggested Fix: You must use a sync method that handles deletions (e.g., rsync --delete or explicitly deleting the target directory before copying).
Constraint: Ensure you do not accidentally delete WPT-specific files (like README.wpt, DIR_METADATA, or OWNERS) that might live in that directory.

This commit introduces a GitHub Actions workflow to automate the vendoring of the Test262 test suite into `third_party/test262`. This automation is the final piece of the integration, ensuring that the tests can be kept up-to-date with the upstream `tc39/test262` repository. The workflow is configured to run weekly or can be triggered manually. It performs the following steps: 1. Checks out both the WPT and `tc39/test262` repositories. 2. Selectively copies a subset of the Test262 tests into WPT. 3. Copies the essential Test262 harness files. 4. Creates a `vendored.toml` file at the root of the vendored directory, recording the specific commit SHA from `tc39/test262` that the files were sourced from. This ensures traceability, as described in the RFC. 5. Commits the updated files and opens a pull request with the changes. Initially, only a subset of Test262 tests related to Interop 2026 features (`Temporal` and `top-level-await`) are being imported. This is a temporary measure to manage the volume of new test results and to avoid overloading wpt.fyi, which currently has limitations in processing the full Test262 result set (see wpt-fyi/issues/4681). The workflow can be expanded to include more tests in the future. This work directly supports the integration of Test262 into WPT as detailed in the RFC: web-platform-tests/rfcs#229 This commit is part of a series of smaller PRs split from the larger, original implementation in #55997.

jgraham · 2025-12-22T12:19:38Z

.github/workflows/update-test262-tests.yml

+        run: |
+          LATEST_SHA=$(git -C test262-spec rev-parse HEAD)
+          echo "Latest remote Test262 SHA: $LATEST_SHA"
+
+          rm -rf wpt/third_party/test262/test/
+          mkdir -p wpt/third_party/test262/test/
+          mkdir -p wpt/third_party/test262/harness/
+
+          # Selectively copy only Interop 2026 feature tests for now
+          # See: https://github.com/web-platform-tests/wpt.fyi/issues/4681
+          # Mapping of web feature to test directory can be found at: https://github.com/tc39/test262/blob/main/WEB_FEATURES.yml
+          # Temporal tests
+          mkdir -p wpt/third_party/test262/test/built-ins/Date/prototype/toTemporalInstant
+          mkdir -p wpt/third_party/test262/test/built-ins/Temporal
+          mkdir -p wpt/third_party/test262/test/intl402/Temporal
+          cp -r test262-spec/test/built-ins/Date/prototype/toTemporalInstant/* wpt/third_party/test262/test/built-ins/Date/prototype/toTemporalInstant
+          cp -r test262-spec/test/built-ins/Temporal/* wpt/third_party/test262/test/built-ins/Temporal
+          cp -r test262-spec/test/intl402/Temporal/* wpt/third_party/test262/test/intl402/Temporal
+
+          # Top-level-await tests
+          mkdir -p wpt/third_party/test262/test/language/module-code/top-level-await
+          cp -r test262-spec/test/language/module-code/top-level-await/* wpt/third_party/test262/test/language/module-code/top-level-await
+
+          # Always sync the harness files
+          rsync -a --delete test262-spec/harness/ wpt/third_party/test262/harness/
+          printf "[test262]\nsource = \"https://github.com/tc39/test262\"\nrev = \"${LATEST_SHA}\"\n" > wpt/third_party/test262/vendored.toml
+      - name: Commit changes
+        id: commit
+        continue-on-error: true
+        run: |
+          cd wpt
+          export BRANCH_NAME="$BRANCH_PREFIX-$(date +'%Y%m%d%H%M%S')"
+          echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
+          git config user.name "$GIT_AUTHOR_NAME"
+          git config user.email "$GIT_AUTHOR_EMAIL"
+          git checkout -B $BRANCH_NAME
+          git add third_party/test262/
+          git commit -m "$COMMIT_TITLE"


Strong agree with this; I don't think we should have non-trivial scripts directly in GH actions files.

.github/workflows/update-test262-tests.yml

jcscottiii · 2026-03-12T19:09:28Z

@DanielRyanSmith @jgraham this is ready for review again.

jgraham · 2026-03-16T10:16:40Z

.github/update-test262.sh

+# This script is intended to be run from the root of the WPT repository.
+# It expects a checkout of tc39/test262 in a directory named 'test262-spec'
+# at the same level as the WPT repository (or as configured below).


This should be in tools/ci. You can look at the ci_* scripts for the pattern we use to ensure that the pwd is set correctly.

jgraham · 2026-03-16T10:22:18Z

.github/update-test262.sh

+# -a: archive mode
+# --delete: delete extraneous files from dest dirs
+# --exclude: preserve WPT-specific metadata
+RSYNC_OPTS=(-a --delete --exclude 'README.wpt' --exclude 'DIR_METADATA' --exclude 'OWNERS' --exclude 'WEB_FEATURES.yml')


This exclude list feels very fragile, and like we're going to eventually be surprised that someone adds META.yml or similar, and then finds that it's deleted later. Ideally we'd have all the files that are synced in a directory that's only synced, but at the very least this kind of config should live inside the test262 directory itself, so that someone updating it might find it.

jcscottiii · 2026-03-16T17:24:12Z

lint.ignore

@@ -440,6 +440,7 @@ SET TIMEOUT: acid/acid3/test.html
 *: resources/webidl2/*
 *: tools/*
 *: */third_party/*


@jgraham FYI. The tests that will be imported into WPT_ROOT/third_party, do not trigger this rule. I had to add the rule below to silence the lint errors.

jgraham · 2026-03-17T14:46:15Z

.github/workflows/update-test262-tests.yml

          cd wpt
          git push --set-upstream origin $BRANCH_NAME
-          gh pr create --title "$COMMIT_TITLE" --body "$PR_BODY"
+          gh pr create --title "$COMMIT_TITLE" --body "$PR_BODY" --reviewer jcscottiii


This looks pretty sus :) If we need a reviewer set then it should probably be @wpt-pr-bot

Good call out! I was just following the wasm one: https://github.com/web-platform-tests/wpt/blob/master/.github/workflows/update-wasm-tests.yml

Happy to also create a pr to change that to wpt-pr-bot too.

Edit: I updated the wasm one to do the same thing #58554

.github/workflows/update-test262-tests.yml

wpt-pr-bot added the GitHub Actions label Dec 18, 2025

wpt-pr-bot assigned foolip Dec 18, 2025

wpt-pr-bot requested review from foolip, jgraham and past December 18, 2025 03:37

jcscottiii changed the title ~~feat(automation): Add workflow to vendor and update Test262 tests~~ feat(automation): Add workflow to vendor and update Test262 tests [pt 4/5] Dec 18, 2025

jcscottiii force-pushed the feat/test262-runner-integration branch from 35d9c44 to 3374ad9 Compare December 18, 2025 03:53

jcscottiii force-pushed the feat/test262-automation branch from 6406255 to 04e3166 Compare December 18, 2025 03:53

jcscottiii force-pushed the feat/test262-runner-integration branch from 3374ad9 to a2d6c63 Compare December 18, 2025 04:34

jcscottiii force-pushed the feat/test262-automation branch from 04e3166 to e8ec10a Compare December 18, 2025 04:35

jcscottiii force-pushed the feat/test262-runner-integration branch from a2d6c63 to 258c427 Compare December 18, 2025 14:06

jcscottiii force-pushed the feat/test262-automation branch from e8ec10a to b3adcf7 Compare December 18, 2025 14:08

This was referenced Dec 18, 2025

WIP: Test262 support #55997

Draft

Integrate Test262 into WPT web-platform-tests/rfcs#229

Merged

DanielRyanSmith approved these changes Dec 18, 2025

View reviewed changes

DanielRyanSmith reviewed Dec 18, 2025

View reviewed changes

jcscottiii force-pushed the feat/test262-runner-integration branch from 258c427 to 94dd567 Compare December 19, 2025 18:48

jcscottiii force-pushed the feat/test262-automation branch from b3adcf7 to a062397 Compare December 19, 2025 19:17

wpt-pr-bot added css-anchor-position css-color css-display css-flexbox css-gaps css-grid css-inline css-ruby css-scoping css-transitions css-typed-om labels Dec 19, 2025

wpt-pr-bot requested review from FremyCompany, frivoal, fsoder, guidou and hayatoito December 19, 2025 19:18

jcscottiii changed the base branch from feat/test262-runner-integration to master December 19, 2025 19:18

jcscottiii removed request for FremyCompany, annevk, boggydigital, caitp, dauwhe, dbaron, dirkschulze, ewilligers, frivoal, hayatoito and past December 19, 2025 19:19

jgraham reviewed Dec 22, 2025

View reviewed changes

jcscottiii mentioned this pull request Jan 14, 2026

feat(runner): Integrate Test262 execution with wptrunner and add smoke tests [pt 3/5] #56842

Open

address feedback

caac524

DanielRyanSmith approved these changes Mar 12, 2026

View reviewed changes

jgraham previously requested changes Mar 16, 2026

View reviewed changes

jcscottiii added 2 commits March 16, 2026 16:19

address feedback from @jgraham

0b5a74e

lint for imported files to ROOT/third_party

eb6fe17

jcscottiii commented Mar 16, 2026

View reviewed changes

jgraham reviewed Mar 17, 2026

View reviewed changes

jcscottiii commented Mar 17, 2026

View reviewed changes

.github/workflows/update-test262-tests.yml Outdated Show resolved Hide resolved

Apply suggestion from @jcscottiii

99c2ec5

jcscottiii mentioned this pull request Mar 17, 2026

Change PR reviewer to wpt-pr-bot for wasm workflow #58554

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(automation): Add workflow to vendor and update Test262 tests [pt 4/5]#56843

feat(automation): Add workflow to vendor and update Test262 tests [pt 4/5]#56843
jcscottiii wants to merge 5 commits intomasterfrom
feat/test262-automation

jcscottiii commented Dec 18, 2025

Uh oh!

DanielRyanSmith Dec 18, 2025 •

edited

Loading

Uh oh!

jgraham Dec 22, 2025

Uh oh!

DanielRyanSmith left a comment

Uh oh!

jgraham Dec 22, 2025

Uh oh!

Uh oh!

jcscottiii commented Mar 12, 2026

Uh oh!

jgraham Mar 16, 2026

Uh oh!

jgraham Mar 16, 2026

Uh oh!

jcscottiii Mar 16, 2026

Uh oh!

jgraham Mar 17, 2026

Uh oh!

jcscottiii Mar 17, 2026

Uh oh!

jcscottiii Mar 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jcscottiii commented Dec 18, 2025

Uh oh!

DanielRyanSmith Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DanielRyanSmith left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jcscottiii commented Mar 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DanielRyanSmith Dec 18, 2025 •

edited

Loading