SendMouseEvent

[jc4golf]

I'm trying to simulate a mouse click to a button on a website. I'm getting the coordinates of the button using HTML getBoundingClientRect() and then sending WVBrowser1.SendMouseEvent with mouse down and then mouse up using a TPoint with the x and y coordinates from getBoundingClientRect(). However, the mouse click is occurring on a different element on the web page. How can I make sure the coordinates I'm sending match the button I'm trying to click?

[salvadordf]

Hi.

See this example in the WindowlessBrowser demo to know how to simulate mouse events :
https://github.com/salvadordf/WebView4Delphi/blob/2c7dec03305b1fbb966d81911e3c4179d3e1294b/demos/Delphi_VCL/WindowlessBrowser/uWindowlessBrowser.pas#L117

[jc4golf]

Here is the code I'm using to send the mouse events:

Code: Select all

  P := Point(x, y);
  WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_DOWN,
    COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);
  WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_UP,
    COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);

Seems to me that the x and y values are relative to the webpage and need to be translated to the screen?

[salvadordf]

Read the code comments in the original code sample :
https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2compositioncontroller?view=webview2-1.0.2478.35#sendmouseinput

TMainForm.HandleMouseMessage is just a Delphi translation but the original has more code comments.

SendMouseInput expects client coordinates for the WebView and the mouse coordinates must be translated for some messages.

[jc4golf]

I've tried several things, all to no avail.

Here's one:

Code: Select all

function TMainForm.OffsetPointToWebView(aPoint: TPoint): TPoint;
begin
  Result := ScreenToclient(aPoint);
  Result.x := Result.x - FWVDirectCompositionHost.Left;
  Result.y := Result.y - FWVDirectCompositionHost.Top;
end;

Here's another (based on the code comments in the original code sample):

Code: Select all

  P := Point(x, y);
  P.x := P.x - WVBrowser1.Bounds.Left;
  P.y := P.y - WVBrowser1.Bounds.Top;

And adding either ScreenToClient(P) or ClientToScreen(P) doesn't help.

[salvadordf]

Are you using the WindowlessBrowser demo for the tests?

SendMouseInput only works when the browser uses a "Composition Controller" which is what the WindowlessBrowser demo uses.

I just tested your code in the WindowlessBrowser demo and the simulated click works without any coordinate system transformation.

Just send the getBoundingClientRect().x and getBoundingClientRect().y values to Delphi and populate the point with them.

Code: Select all

  P := Point(x, y);
  WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_DOWN,
    COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);
  WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_UP,
    COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);

Edit : Just to be sure, use the middle point of that element to simulate the click.

[salvadordf]

It should also be possible to use the Input.dispatchMouseEvent DevTools method to simulate mouse events :
https://chromedevtools.github.io/devtools-protocol/tot/Input/#method-dispatchMouseEvent

That method should work even without using a Composition Controller.

[jc4golf]

Okay, that worked. That's what I did originally, so I don't know why I thought it didn't work.

Thank you, Salvador.

[jc4golf]

My apologies, I thought it worked because without realizing it I had left a "click()" on the button in the javascript.

I'm not getting the correct x and y values from the getBoundingClientRect() function. Is there a way in Webview4Delphi similar to Cef4Delphi to read the DOM elements and get the bounds?

[salvadordf]

Please, provide a simplified demo that I can use to reproduce this issue in my computer.