Skip to content

feat(ci): Configure Taskcluster to run Test262 tests [pt 5/5]#56845

Open
jcscottiii wants to merge 3 commits intomasterfrom
feat/test262-taskcluster-ci
Open

feat(ci): Configure Taskcluster to run Test262 tests [pt 5/5]#56845
jcscottiii wants to merge 3 commits intomasterfrom
feat/test262-taskcluster-ci

Conversation

@jcscottiii
Copy link
Copy Markdown
Contributor

This commit integrates Test262 test execution into the WPT Taskcluster Continuous Integration system.

Key changes and their purpose:

  • Task Definition (tools/ci/tc/tasks/test.yml):
    • A new wpt-test262 component is defined, specifying the common properties and chunking strategy for Test262 tasks within Taskcluster.
    • The main Taskcluster task generation matrix is updated to include test262 as a recognized test suite. This ensures that Taskcluster will automatically create jobs for Test262 across configured browsers and channels (e.g., Firefox Nightly, Chrome Canary).
  • CI Validation (tools/ci/tc/tests/test_valid.py):
    • The integration tests for Taskcluster's decision logic (test_valid.py) are updated to expect the new wpt-*-test262-* tasks in the generated task graph. This provides automated validation that the CI configuration is correctly interpreted and will schedule the Test262 tasks as intended.

With these changes, Taskcluster will automatically detect and execute Test262 tests that are present in the third_party/test262 directory (as populated by the automated vendoring workflow).

This work is a fundamental part of the broader Test262 integration effort, as detailed in the RFC: web-platform-tests/rfcs#229

This commit is the final in a series of smaller PRs split from the larger, original implementation in #55997.

@jcscottiii jcscottiii changed the title feat(ci): Configure Taskcluster to run Test262 tests feat(ci): Configure Taskcluster to run Test262 tests [pt 5/5] Dec 18, 2025
@jcscottiii jcscottiii force-pushed the feat/test262-runner-integration branch from 35d9c44 to 3374ad9 Compare December 18, 2025 03:53
@jcscottiii jcscottiii force-pushed the feat/test262-taskcluster-ci branch from 395390f to f56ce84 Compare December 18, 2025 03:54
@jcscottiii jcscottiii force-pushed the feat/test262-runner-integration branch from 3374ad9 to a2d6c63 Compare December 18, 2025 04:34
@jcscottiii jcscottiii force-pushed the feat/test262-taskcluster-ci branch from f56ce84 to 10cdbb7 Compare December 18, 2025 04:35
@jcscottiii jcscottiii force-pushed the feat/test262-runner-integration branch from a2d6c63 to 258c427 Compare December 18, 2025 14:06
@jcscottiii jcscottiii force-pushed the feat/test262-taskcluster-ci branch from 10cdbb7 to dafdf3a Compare December 18, 2025 14:09
@jcscottiii
Copy link
Copy Markdown
Contributor Author

Since this impacts TaskCluster, I think @jgraham would be the better reviewer for this.

@jcscottiii jcscottiii assigned jgraham and unassigned jonathan-j-lee Dec 18, 2025
@jcscottiii jcscottiii force-pushed the feat/test262-runner-integration branch from 258c427 to 94dd567 Compare December 19, 2025 18:48
@jcscottiii jcscottiii changed the base branch from feat/test262-runner-integration to master December 19, 2025 19:20
@jcscottiii jcscottiii force-pushed the feat/test262-taskcluster-ci branch from dafdf3a to cf3c932 Compare December 19, 2025 19:21
test-type: crashtest

wpt-test262:
chunks: 16
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for this choice?

Copy link
Copy Markdown
Contributor Author

@jcscottiii jcscottiii Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taking a look at these runs:

Chunk 1 took ~18mins (I did see some below 18mins but one did take 34 mins).

  • Node Start to Docker image downloaded and task starting (log): 20:31:36 - 20:30:58 = ~30secs
  • Task start to Test starting (log): 20:33:39 - 20:30:58 = ~2min 30secs
  • Test finished = ~15mins

Assuming we have ~51k test262 tests. So that's about 3.2k tests per chunk. So for 3200 tests in 15 mins, that's about ~200tests per minute.

If we want to keep it under 60 mins per node, we could go to 8 chunks. Should be about ~33mins per chunk then.

@jcscottiii jcscottiii requested a review from jgraham March 12, 2026 20:35
This commit integrates Test262 test execution into the WPT Taskcluster
Continuous Integration system.

Key changes and their purpose:
- **Task Definition (`tools/ci/tc/tasks/test.yml`):**
  - A new `wpt-test262` component is defined, specifying the common
    properties and chunking strategy for Test262 tasks within Taskcluster.
  - The main Taskcluster task generation matrix is updated to include
    `test262` as a recognized test suite. This ensures that Taskcluster
    will automatically create jobs for Test262 across configured
    browsers and channels (e.g., Firefox Nightly, Chrome Canary).
- **CI Validation (`tools/ci/tc/tests/test_valid.py`):**
  - The integration tests for Taskcluster's decision logic (`test_valid.py`)
    are updated to expect the new `wpt-*-test262-*` tasks in the generated
    task graph. This provides automated validation that the CI configuration
    is correctly interpreted and will schedule the Test262 tasks as intended.

With these changes, Taskcluster will automatically detect and execute Test262
tests that are present in the `third_party/test262` directory (as populated by
the automated vendoring workflow).

This work is a fundamental part of the broader Test262 integration effort,
as detailed in the RFC: web-platform-tests/rfcs#229

This commit is the final in a series of smaller PRs split from the larger,
original implementation in #55997.
@jcscottiii jcscottiii force-pushed the feat/test262-taskcluster-ci branch from f474778 to daaf698 Compare March 12, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants