feat(runner): Integrate Test262 execution with wptrunner and add smoke tests [pt 3/5] by jcscottiii · Pull Request #56842 · web-platform-tests/wpt

jcscottiii · 2025-12-18T03:04:31Z

Summary

Integrates the TC39 Test262 test suite natively into the Web Platform Tests infrastructure. This enables automated, cross-browser ECMAScript conformance testing directly within wptrunner and standard WPT dashboards.

Architecture

Test262 tests are executed via a custom Window Wrapper implementation that maps the standard Test262 host API into WPT's testharness.js.

Wrapper Generation: tools/serve/serve.py intercepts .test262.html requests and dynamically generates a document that injects testharness.js, the $262 host provider, and Test262's standard evaluation fixtures.
Execution Boundary: Tests are isolated within an iframe child context to prevent global scope contamination and enforce strict mode expectations.
Communication Protocol: A postMessage bridge (test262-reporter.js -> test262-harness-bridge.js) synchronizes child-frame execution states. The adapter emits structured JSON payloads ({type: 'complete'}, {type: 'fail'}, {type: 'error'}) that trigger t.done(), assert_unreached(), and generic unhandled errors in the parent. This natively delegates all timeout and lifecycle management directly to testharness.js.

Supported Features

Negative Tests (negative): Safely traps expected parsing and runtime evaluation exceptions and translates them to successful Web Platform Tests assertions.
Asynchronous Tests (async): Configures $DONE and print() bindings to fulfill Test262 async constraints before halting the overarching WPT test pipeline.
Modules (module): Injects dynamic module loading wrappers, directly escalating syntax parsing and import boundary rejections into explicit WPT subtest failures.
Strict mode (onlyStrict): Supports execution isolation for strictly defined environment iterations.

DanielRyanSmith

This suggestion came from a Gemini code review:

Harness Adapter Robustness

Focus: resources/test262/harness-adapter.js
Critique: Bridging Test262's global-polluting API (e.g., $DONE, Test262Error) to the structured testharness.js API is error-prone.

Exception Handling: Ensure the adapter explicitly wraps the execution of the Test262 test code in a try/catch block (or uses window.onerror). Test262 tests often throw bare strings or non-Error objects. The adapter must normalize these into meaningful testharness failure messages, otherwise, the runner might just report a generic "Script error." or timeout.
$DONE Semantics: Verify that calling $DONE() (without arguments) signals a PASS, and $DONE(error) signals a FAIL. A common bug is treating any call to $DONE as a completion without checking the argument.

resources/test262/harness-adapter.js

jgraham · 2025-12-22T11:53:28Z

resources/test262/harness-adapter.js

+      var iframe = global.document.createElement('iframe');
+      iframe.style.cssText = 'display: none';
+      iframe.src = '';  // iframeSrc;
+      if (global.document.body === null) {


This feels potentially surprising, because it means that something like

<html> <script> let realm = $262.createRealm(); </script> <meta name="example" content="test"> <body> […]

is going to end up with the <meta> element in the . Do we actually need the capacity to handle both the cases where there is and isn't already a ` here?

We just need a place to insert the iframes. If you ensure all templates have a body that should be fine too.

jgraham · 2025-12-22T11:54:43Z

resources/test262/test262-provider.js

+      var script = global.document.createElement('script');
+      script.text = src;
+      window.__test262_evalScript_error_ = undefined;
+      global.document.head.appendChild(script);


Is there a specific reason it has to go in the head? It's not totally clear what invariants we're trying to maintain about the DOM.

I assume it could go anywhere in the document.

resources/test262/harness-adapter.js

third_party/test262/harness/assert.js

resources/test262/test262-provider.js

jgraham · 2025-12-22T14:17:58Z

resources/test262/testharness-client.js

@@ -0,0 +1,77 @@
+/*
+ * Minimalistic testharness-client tailored to run Test262


Since -client isn't an existing concept or convention, we shouldn't assume it's obvious what it actually is.

To help with this, I actually renamed this to be test262-reporter.js. And I renamed the harness-adapter.js to be test262-provider.js. Hopefully is a little bit clearer!

jgraham · 2025-12-22T14:30:51Z

resources/test262/testharness.js

+    };
+
+    function log(msg) {
+        document.getElementById('log').innerHTML += ""+msg+"<br>";


In normal testharness we turn this off in CI for performance reasons. I think we want to do the same here.

resources/test262/testharness.js

Comments addressed. This is assuming that #56841 will be merged as-is.

jcscottiii · 2026-03-12T21:19:57Z

@jgraham I renamed some of the javascript files to hopefully clearer names. But happy to revert back. Afterwards, I will adjust the PR description. #56841 has been adjusted with those names as well.

DanielRyanSmith

I'm going to approve this, but a typo was found that needs to be fixed:

Critical Typo in Safari Configuration

In tools/wptrunner/wptrunner/browsers/safari.py, the crashtest executor mapping was accidentally renamed to crashtag. This will break the execution of crash tests in Safari.

@@ -22,7 +22,8 @@
                  "executor": {"testharness": "WebDriverTestharnessExecutor",
                               "reftest": "WebDriverRefTestExecutor",
                               "wdspec": "WdspecExecutor",
-                              "crashtest": "WebDriverCrashtestExecutor"},
+                              "crashtag": "WebDriverCrashtestExecutor",
+                              "test262": "WebDriverTestharnessExecutor"},

Recommendation: Revert crashtag back to crashtest.

Other than this issue, the core integration of Test262, including the iframe realm isolation, $262 agent logic, and runner mappings, looks well-implemented and structurally sound.

tools/wptrunner/wptrunner/browsers/safari.py

jcscottiii · 2026-03-13T00:09:54Z

Thanks for catching that @DanielRyanSmith

jgraham · 2026-03-16T10:13:11Z

It looks like some of the comments from my earlier review remain unaddressed. In particular the way this reimplements a subset of testharness.js but also hard-codes part of the wpt-specific testharnessreport.js.

…e tests This commit completes the core integration of Test262 into the WPT runner (`wptrunner`), enabling the execution of Test262 JavaScript conformance tests within the browser environment. This work builds upon the manifest tooling (PR 1 #56840) and server-side handling (PR 2 #56841) to provide a full end-to-end testing solution for Test262. Key components introduced and integrated: - **Client-Side Harness:** New JavaScript files (`harness-adapter.js`, `testharness-client.js`, `testharness.js`) under `resources/test262/` provide the necessary environment for Test262 tests to execute in an `iframe` and report results back to the main WPT testharness. - **Test262 Assertion Libraries (Vendored):** For the purpose of enabling these initial smoke tests, the standard Test262 assertion libraries (`assert.js` and `sta.js`) are directly placed into `third_party/test262/harness/`. This provides the immediate dependencies required by the smoke tests. An autoamted vendoring solution, including version tracking via `vendored.toml`, will be implemented in a subsequent PR focused on automated updates, as described in the RFC. - **Wptrunner Integration:** Modifications to `tools/wptrunner/wpttest.py` and `tools/wptrunner/browsers/*.py` enable `wptrunner` to recognize the `test262` test type and execute it using existing testharness executors. - **Smoke Tests and Metadata:** A suite of basic Test262 smoke tests (`infrastructure/test262/`) and their corresponding expected results (`infrastructure/metadata/infrastructure/test262/`) are added to verify the correct functioning of the entire integration. - **Linting Exemption:** `lint.ignore` is updated to exclude the vendored Test262 harness files from linting, respecting their upstream style. This integration allows browser vendors to run Test262 directly with `wptrunner`, streamlining conformance testing efforts as specified in the RFC: web-platform-tests/rfcs#229 This commit is the third in a series of smaller PRs split from the larger, original implementation in #55997.

jcscottiii · 2026-03-18T21:40:52Z

resources/test262/README.md

@jgraham Let me know if you find this useful or if I should move this around.

jcscottiii · 2026-03-18T21:42:54Z

@jgraham I think I got all of your suggestions now. Please take a look. I also added some more infra tests because I saw that we should probably handle the async tests from test262 too.

lint.ignore

tools/serve/serve.py

This commit addresses reviewer feedback (jgraham) regarding the Test262 integration "reimplementing a subset of testharness.js". The architecture has been drastically simplified by eliminating the custom `postMessage` protocol between the Test262 iframe and the parent WPT test wrapper. Changes include: - Deleted `test262-harness-bridge.js` completely. - Updated the wrapper `<script>` generated by `tools/serve/serve.py` to expose a native `window.test262HarnessDone(status, error)` hook directly in the parent's `testharness.js` execution environment. - Refactored `test262-reporter.js` to natively dispatch test completions and harness errors to this synchronous hook instead of manually wrapping results in `{type: "complete"}` message payloads. - Ensured strict "Realm Isolation" is maintained for the `test262` tests, as required by `INTERPRETING.md`, by keeping all WPT globals cleanly isolated in the parent window. - Updated `tools/serve/test_serve.py` assertions to reflect the removed bridge dependency. - Leveraged `setTimeout` to safely elevate test harness crashes into the parent's macro-task queue, ensuring infrastructure `.ini` expectations (like timeouts and parsing ERRORs) correctly fail the WPT harness.

jcscottiii · 2026-03-27T15:05:35Z

@jgraham I tried to reduce the number of files in total and removed the bridge file in this commit. But let me know if you want me to revert that commit and leave it how I originally had it.

wpt-pr-bot added infra third_party wptrunner The automated test runner, commonly called through ./wpt run labels Dec 18, 2025

wpt-pr-bot assigned foolip Dec 18, 2025

wpt-pr-bot requested review from DanielRyanSmith, foolip, jgraham and jonathan-j-lee December 18, 2025 03:04

jcscottiii changed the title ~~feat(runner): Integrate Test262 execution with wptrunner and add smoke tests~~ feat(runner): Integrate Test262 execution with wptrunner and add smoke tests [pt 3/4] Dec 18, 2025

jcscottiii changed the title ~~feat(runner): Integrate Test262 execution with wptrunner and add smoke tests [pt 3/4]~~ feat(runner): Integrate Test262 execution with wptrunner and add smoke tests [pt 3/5] Dec 18, 2025

jcscottiii force-pushed the feat/test262-runner-integration branch from 35d9c44 to 3374ad9 Compare December 18, 2025 03:53

wpt-pr-bot added the manifest label Dec 18, 2025

jcscottiii force-pushed the feat/test262-runner-integration branch from 3374ad9 to a2d6c63 Compare December 18, 2025 04:34

jcscottiii force-pushed the feat/test262-serve-handler branch from 16aa645 to d5eb82e Compare December 18, 2025 14:05

jcscottiii force-pushed the feat/test262-runner-integration branch from a2d6c63 to 258c427 Compare December 18, 2025 14:06

This was referenced Dec 18, 2025

WIP: Test262 support #55997

Draft

Integrate Test262 into WPT web-platform-tests/rfcs#229

Merged

DanielRyanSmith reviewed Dec 18, 2025

View reviewed changes

DanielRyanSmith approved these changes Dec 18, 2025

View reviewed changes

jcscottiii force-pushed the feat/test262-serve-handler branch 2 times, most recently from dcdca82 to df146d5 Compare December 19, 2025 18:44

jcscottiii force-pushed the feat/test262-runner-integration branch from 258c427 to 94dd567 Compare December 19, 2025 18:48

jgraham previously requested changes Dec 22, 2025

View reviewed changes

wpt-pr-bot added the serve label Mar 11, 2026

jcscottiii requested a review from jgraham March 11, 2026 15:33

jcscottiii force-pushed the feat/test262-runner-integration branch 2 times, most recently from 36bd1d5 to 5a93949 Compare March 12, 2026 19:53

DanielRyanSmith approved these changes Mar 12, 2026

View reviewed changes

jcscottiii commented Mar 13, 2026

View reviewed changes

tools/wptrunner/wptrunner/browsers/safari.py Outdated Show resolved Hide resolved

jcscottiii force-pushed the feat/test262-serve-handler branch from 12f7b3e to 1f27a1e Compare March 17, 2026 16:27

Base automatically changed from feat/test262-serve-handler to master March 17, 2026 20:44

jcscottiii added 4 commits March 17, 2026 20:49

address feedback

a89c9f5

address more feedback

b847efa

address feedback

101ab76

jcscottiii force-pushed the feat/test262-runner-integration branch from 1280886 to 101ab76 Compare March 17, 2026 20:50

jcscottiii and others added 2 commits March 18, 2026 21:38

address feedback

84e9a59

Apply suggestion from @jcscottiii

2b013a0

jcscottiii commented Mar 18, 2026

View reviewed changes

resources/test262/README.md

Copy link
Copy Markdown

Contributor Author

jcscottiii Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jgraham Let me know if you find this useful or if I should move this around.

jcscottiii added 3 commits March 18, 2026 21:51

fix lint

fa7b9c9

move need third_party lint because other doesn't work

3bf82d5

fix lint

e049c6e

jcscottiii commented Mar 27, 2026

View reviewed changes

lint.ignore Show resolved Hide resolved

jcscottiii commented Mar 27, 2026

View reviewed changes

tools/serve/serve.py Outdated Show resolved Hide resolved

jcscottiii and others added 2 commits March 27, 2026 14:21

Update tools/serve/serve.py

998d445

		@@ -0,0 +1,77 @@
		/*
		* Minimalistic testharness-client tailored to run Test262

Conversation

jcscottiii commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Architecture

Supported Features

Uh oh!

DanielRyanSmith left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Harness Adapter Robustness

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jcscottiii commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DanielRyanSmith left a comment

Choose a reason for hiding this comment

I'm going to approve this, but a typo was found that needs to be fixed:

Uh oh!

Uh oh!

jcscottiii commented Mar 13, 2026

Uh oh!

jgraham commented Mar 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcscottiii commented Mar 18, 2026

Uh oh!

Uh oh!

Uh oh!

jcscottiii commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jcscottiii commented Dec 18, 2025 •

edited

Loading

DanielRyanSmith left a comment •

edited

Loading

jcscottiii commented Mar 12, 2026 •

edited

Loading

jcscottiii commented Mar 27, 2026 •

edited

Loading