Page 1 of 2
SendMouseEvent
Posted: Thu May 23, 2024 11:22 pm
by jc4golf
I'm trying to simulate a mouse click to a button on a website. I'm getting the coordinates of the button using HTML getBoundingClientRect() and then sending WVBrowser1.SendMouseEvent with mouse down and then mouse up using a TPoint with the x and y coordinates from getBoundingClientRect(). However, the mouse click is occurring on a different element on the web page. How can I make sure the coordinates I'm sending match the button I'm trying to click?
Re: SendMouseEvent
Posted: Fri May 24, 2024 11:11 am
by salvadordf
Hi.
See this example in the WindowlessBrowser demo to know how to simulate mouse events :
https://github.com/salvadordf/WebView4Delphi/blob/2c7dec03305b1fbb966d81911e3c4179d3e1294b/demos/Delphi_VCL/WindowlessBrowser/uWindowlessBrowser.pas#L117
Re: SendMouseEvent
Posted: Sat May 25, 2024 3:30 pm
by jc4golf
Here is the code I'm using to send the mouse events:
Code: Select all
P := Point(x, y);
WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_DOWN,
COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);
WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_UP,
COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);
Seems to me that the x and y values are relative to the webpage and need to be translated to the screen?
Re: SendMouseEvent
Posted: Sun May 26, 2024 9:42 am
by salvadordf
Read the code comments in the original code sample :
https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2compositioncontroller?view=webview2-1.0.2478.35#sendmouseinput
TMainForm.HandleMouseMessage is just a Delphi translation but the original has more code comments.
SendMouseInput expects client coordinates for the WebView and the mouse coordinates must be translated for some messages.
Re: SendMouseEvent
Posted: Sun May 26, 2024 10:05 pm
by jc4golf
I've tried several things, all to no avail.
Here's one:
Code: Select all
function TMainForm.OffsetPointToWebView(aPoint: TPoint): TPoint;
begin
Result := ScreenToclient(aPoint);
Result.x := Result.x - FWVDirectCompositionHost.Left;
Result.y := Result.y - FWVDirectCompositionHost.Top;
end;
Here's another (based on the code comments in the original code sample):
Code: Select all
P := Point(x, y);
P.x := P.x - WVBrowser1.Bounds.Left;
P.y := P.y - WVBrowser1.Bounds.Top;
And adding either ScreenToClient(P) or ClientToScreen(P) doesn't help.
Re: SendMouseEvent
Posted: Mon May 27, 2024 1:39 pm
by salvadordf
Are you using the WindowlessBrowser demo for the tests?
SendMouseInput only works when the browser uses a "
Composition Controller" which is what the WindowlessBrowser demo uses.
I just tested your code in the WindowlessBrowser demo and the simulated click works without any coordinate system transformation.
Just send the getBoundingClientRect().x and getBoundingClientRect().y values to Delphi and populate the point with them.
Code: Select all
P := Point(x, y);
WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_DOWN,
COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);
WVBrowser1.SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_UP,
COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE, 0, P);
Edit : Just to be sure, use the middle point of that element to simulate the click.
Re: SendMouseEvent
Posted: Mon May 27, 2024 2:33 pm
by salvadordf
It should also be possible to use the Input.dispatchMouseEvent DevTools method to simulate mouse events :
https://chromedevtools.github.io/devtools-protocol/tot/Input/#method-dispatchMouseEvent
That method should work even without using a Composition Controller.
Re: SendMouseEvent
Posted: Mon May 27, 2024 3:06 pm
by jc4golf
Okay, that worked. That's what I did originally, so I don't know why I thought it didn't work.
Thank you, Salvador.
Re: SendMouseEvent
Posted: Tue Jun 04, 2024 7:49 pm
by jc4golf
My apologies, I thought it worked because without realizing it I had left a "click()" on the button in the javascript.
I'm not getting the correct x and y values from the getBoundingClientRect() function. Is there a way in Webview4Delphi similar to Cef4Delphi to read the DOM elements and get the bounds?
Re: SendMouseEvent
Posted: Wed Jun 05, 2024 2:03 pm
by salvadordf
Please, provide a simplified demo that I can use to reproduce this issue in my computer.