feat(browser): introduce experimental pointer API - #10780

There's an issue with locators here which I'm not sure how to solve... We can't resolve them before-hand because interactions might insert them in the DOM, maybe a getter on target could work 🤔

Implemented target as a getter when it's a Locator.

Was thinking that this might get merged into keyboard, not sure tho as it would allow doing weird stuff like

userEvent.keyboard(target,  '[Shift>]')
userEvent.pointer([{ target, keys: '[MouseLeft][/Shift]' }])

Merged the unreleased part of pointer into the keyboard management.

todo: change version number here and in @since JSDocs

How does this relate to the client side scale position adjustment processPlaywrightPosition?

Yes, looks like we already scaled position but only on the client-side, I forgot about that 🙃

The issue here is more like we can't compose actions on the server side rather than it not working in general (it works when called from the client).

I wonder if we should then keep the position-scaling behavior on client-side, server-side, or maybe both? We could provide a "scaled" boolean internal option which disables scaling if it was already computed somewhere else.

I think the concern of switching fully on server-side is the extra roundtrip of evaluate, so keeping both seems fine to start with.

We could provide a "scaled" boolean internal option which disables scaling if it was already computed somewhere else.

I think it's currently doubling the ui scale for client click API since position is already compensated on client side?

position: options.position
      && await resolvePageCoordinates(context, options.position, true)

So it looks like it requires differentiating them somehow, but that means the client can pass the scale directly to server, so we don't need server-side scale evaluation?

I think it's currently doubling the ui scale for client click API since position is already compensated on client side?

Yes. In this branch, a call from the client, with a 0.5 iframe scale, resolves like this: [userEvent] 100px → [client] 50px → [server] 25px

but that means the client can pass the scale directly to server, so we don't need server-side scale evaluation?

Yes, something like this:

position: options._scaled
  ? options.position
  : options.position && resolvePageCoordinates(context, options.position, true)

This way we can compose actions on the server too, without having to scale manually each time we call them. If it's not something we want I can just call the Playwright method on the locator, works either way for me

but that means the client can pass the scale directly to server, so we don't need server-side scale evaluation?

Yes, something like this:

position: options._scaled
  ? options.position
  : options.position && resolvePageCoordinates(context, options.position, true)

Does this require server-side evaluate? If client can pass x/y/scale, then server pointer logic can use that to compensate before internal calling server click/hover? something like:

// client click
// -> server click (options.position already scaled on client)

// client pointer (obtain x/y/scale on client and pass as internal option)
// -> server pointer
//   -> for each server click, adjust position with x/y/scale passed from client
//     -> server click (options.position already scaled on caller side)

So, server click can assume position is always compensated

Unfortunately, I don't think we can get rid of the server-side evaluate 😕

With pointer being an asynchronous chain of actions, there's nothing stopping the user from resizing the pane containing the iframe while it runs, so we can't safely save the scaling and iframe coordinates when calling the server

For now I might just scale the values when calling the function, we can re-evaluate in the future if we have to call these functions in other places

With pointer being an asynchronous chain of actions, there's nothing stopping the user from resizing the pane containing the iframe while it runs, so we can't safely save the scaling and iframe coordinates when calling the server

Personally, I wouldn't worry about this case. This sounds like also already true for existing click scaling, which can take time if playwright retries internally, but compensated position is computed once on client side before playwright-level click.

I'm okay with current shape of having resolvePageCoordinates on server though.

I agree with Hiroshi here, the rescaling while the function runs looks more like intentionally breaking the intended implementation rather than a real use case (especially in headless mode!).

If we can avoid the round trip, I think we should. We already do that during "click" where you could technically also resize while it tries to find the element. The resizing argument is just too contrived in my opinion, we can come up with bad faith arguments all day - what if the user switches tabs? what if the user calls this in parallel? what if the user accesses top level context and intentionally changes the scaling!? If you are trying to break it, you will find a way to break it

Browser UI should be treated as a preview environment, not a test one. We should optimize for headless mode, in my opinion.

That makes sense, yes.

I moved the scaling and offset logic to the client and computed their values while serializing/normalizing the input.

I'm not entirely sure if that's the best place to do it tho since different providers could (do?) treat raw coordinates differently, but afaik we can't override userEvent methods from the provider.