Skip to content

Tentative testdriver methods#226

Merged
zcorpan merged 7 commits intoweb-platform-tests:mainfrom
jcsteh:tentative
Feb 3, 2026
Merged

Tentative testdriver methods#226
zcorpan merged 7 commits intoweb-platform-tests:mainfrom
jcsteh:tentative

Conversation

@jcsteh
Copy link
Copy Markdown
Contributor

@jcsteh jcsteh commented Jun 3, 2025

Copy link
Copy Markdown
Member

@gsnedders gsnedders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of my comments below are really "let's make this more explicit", rather than any real objection.

And mostly a question for the Core Team, but I do wonder what standard we thing we're holding things to based on #127: is an explainer enough for a testdriver API, or do we want a spec? Because if an explainer is enough, I'd question how often we'd actually add tentative API, because we probably do want something defining what the behaviour of the endpoint is, even if only at a high-level.

# RFC 226: Tentative testdriver methods

## Summary
Allow tentative methods to be added to testdriver.js that are not yet included in the WebDriver or WebDriver BiDi specifications.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be more explicit; we already have plenty of non-tentative things in testdriver which are not in either spec — #127 (final RFC) merely requires a WebDriver endpoint, not that it is in one of those specs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. I had assumed that the existence of a WebDriver endpoint implied it needed to be in the WebDriver spec. So how is WebDriver endpoint defined, given that Gecko uses Marionette for WPT and no other vendors implement this WebDriver endpoint yet?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, https://webbluetoothcg.github.io/web-bluetooth/#automated-testing defines a whole load of endpoints.

The policy is that it requires a spec for the endpoint, not that that spec be in any specific standard.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Now I better understand your original question regarding whether an explainer is sufficient. I agree that at least having an explainer makes sense, so if the consensus is that an explainer is acceptable, we probably don't need this RFC. RFC 127 just says "an existing WebDriver endpoint", which isn't explicit about a formal spec, though I guess that could be inferred. So perhaps we still need an RFC making it clear that an explainer is sufficient.

That said, one difference between a spec and an explainer is that an explainer is far more likely to change than a spec. So that might still be one reason to mark these methods as tentative: to indicate that the details are still being ironed out.

Copy link
Copy Markdown
Contributor

@Ms2ger Ms2ger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no fundamental concerns with this, assuming we'd use this for experimentation, not things that stick around indefinitely.

Copy link
Copy Markdown
Contributor

@jcscottiii jcscottiii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed this today during the RFC meeting. I'm still opposed to the wording in Detail 1. But the rest of the RFC looks fine. And in the absence of better alternatives for that one point, I will approve this.

@zcorpan
Copy link
Copy Markdown
Member

zcorpan commented Sep 2, 2025

@gsnedders Can you take another look at this?

@zcorpan
Copy link
Copy Markdown
Member

zcorpan commented Nov 4, 2025

An alternative to having tentative_ in the method names is to have an assertion for these methods that checks that the test itself is tentative (by checking the test's URL).

@zcorpan zcorpan requested a review from gsnedders December 2, 2025 17:35
Copy link
Copy Markdown
Contributor

@cookiecrook cookiecrook left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r+ with the addition of the new assertion expectation in tentative methods.

@cookiecrook
Copy link
Copy Markdown
Contributor

cookiecrook commented Dec 6, 2025

Most of my comments below are really "let's make this more explicit", rather than any real objection.

I think it's okay to dismiss @gsnedders review since it's 5 months old, was self-described as not a "real objection", and I believe has fully addressed in the latest iteration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants