Hit-testing fails on Win32 with viewport sidebar + docking + borderless window

[BobbyAnguelov]

Dear ImGui 1.89.3 WIP (18928)
--------------------------------
sizeof(size_t): 8, sizeof(ImDrawIdx): 2, sizeof(ImDrawVert): 20
define: __cplusplus=199711
define: _WIN32
define: _WIN64
define: _MSC_VER=1934
define: _MSVC_LANG=201402
define: IMGUI_HAS_VIEWPORT
define: IMGUI_HAS_DOCK
--------------------------------
io.BackendPlatformName: imgui_impl_win32
io.BackendRendererName: imgui_impl_dx11
io.ConfigFlags: 0x00000441
 NavEnableKeyboard
 DockingEnable
 ViewportsEnable
io.ConfigViewportsNoDecoration
io.ConfigInputTextCursorBlink
io.ConfigWindowsResizeFromEdges
io.ConfigMemoryCompactTimer = 60.0
io.BackendFlags: 0x00001C0E
 HasMouseCursors
 HasSetMousePos
 PlatformHasViewports
 HasMouseHoveredViewport
 RendererHasVtxOffset
 RendererHasViewports
--------------------------------
io.Fonts: 1 fonts, Flags: 0x00000000, TexSize: 512,64
io.DisplaySize: 1264.00,792.00
io.DisplayFramebufferScale: 1.00,1.00
--------------------------------
style.WindowPadding: 8.00,8.00
style.WindowBorderSize: 1.00
style.FramePadding: 4.00,3.00
style.FrameRounding: 0.00
style.FrameBorderSize: 0.00
style.ItemSpacing: 8.00,4.00
style.ItemInnerSpacing: 4.00,4.00

Version: 1.89.3
Branch: docking

Back-end/Renderer/Compiler/OS

Back-ends: imgui_impl_dx11cpp + imgui_impl_win32.cpp
Operating System: Win11 and Win10

My Issue/Question:

So this is a really strange issue that I've tried debugging but once I ended up deep in the imgui internals, I figured it would be better to post the question.

So I'm trying to create a custom title bar for my window. I'm using the ViewportSidebar feature to create my window titlebar and it's mostly working great except that occasionally the hit testing fails. This seems to be related to some combination of docking of windows and (maybe?) moving the main win32 window. It's a 100% reproduceable in my home engine but a little harder to repro in the example app but it still happens.

I think enabling the dockspace example exacerbates the issue. I've provided a repro case, so all you need to do is replace the main.cpp in the dx11 example and run it. you'll notice the new title bar and the three buttons that you can hover over and they detect that. Now example the dockspace example and dock undock the windows and you'll notice how the hit-testing starts to fail. In some cases it fixes but sometimes it doesn't (and in my home engine it never fixes itself). I've also attached a video showing the issue.

Anyone have any idea what's going on?

Screenshots/Video
https://user-images.githubusercontent.com/10767490/218336676-ffa7d46c-6c4e-4f11-90e8-203b63fe21cc.mp4

Standalone, minimal, complete and verifiable example:
Replace the DX11 example main.cpp with the version supplied, all changes are within // BUG REPRO comment blocks.
main.zip

[ocornut]

Thanks for taking the time to create that repro! (this is double useful as I was hoping to look toying with borderless stuff at some point so this is a nice sets of pointer).

I eventually manged to repro - but I don't yet understand what leads to it - and I noticed that when the bug happens, mouse position is not being updated while hovering that upper section:

The two values to watch are MouseViewport (here it is correct) and MousePos (can't see on the picture, but when the bug happens that values stays still even though I am moving my mouse cursor over that section, when the bug doesn't happen that value is updated while moving). You should be able to confirm the lack of events in Tools->Debug Log->IO.

Debugging suggestion: always print out mouse pos on screen and/or GetForegroundDrawList()->AddCircleFilled(ImGui::GetMousePos(), ...); to visualize it.

Dear ImGui is not receiving mouse inputs so it's probably not a Dear ImGui bug, but there is a possibility that imgui_impl_win32.cpp doesn't play well with your setup of hooking WM_NCCALCSIZE/WM_NCHITTEST. I myself haven't attempted this so I presently have no idea where the issue is.

Note that imgui_impl_win32.cpp calls io.AddMousePosEvent() in two locations. So if you are getting neither you may want to ad debug logs in the backend to try to understand what's preventing things from working. You can use IMGUI_DEBUG_LOG to add to the Tools->Debug Log window. You may want to print the result of ::GetCapture(), bd->MouseHwnd, bd->MouseTracked,

• FIrst location is WM_MOUSEMOVE handler: this normally covers the majority of situations.
• Second location is one specific code path in ImGui_ImplWin32_UpdateMouseData() which is designed to provides dear imgui with out-of-client-boundaries mouse positions when window is focused but not hovered and mouse hasn't been captured by the window (~haven't clicked and dragged from inside the window). That one is optional.

But it's simply possible that your logic using bool const isAnyInteractibleWidgetHovered = ImGui::IsAnyItemHovered(); and returning HT_CAPTION is faulty, you should also dump all of that stuff on the screen. Once you start returning HTCAPTION then you don't get mouse update fed to dear imgui so it won't update its knowledge of the mouse cursor position?

(
FYI Docking maneuvers will tend to do those things:

• create a window while mouse capture is active in another one (generally the parent).
• temporary set ImGuiViewportFlags_NoInputs on a viewport hosting a window being dragged, which alters our WM_NCHITTEST handler in ImGui_ImplWin32_WndProcHandler_PlatformWindow(). However this should not able to main viewport at all.

My intuition is that neither of those are related or the cause of your issue.
)

[BobbyAnguelov]

Interesting, my suspicion is similar that there is perhaps some bad interaction between the wndProc handlers but I know the viewportSidebar is also a newish feature so there may have been some bug. I'll try to investigate what is causing the loss of mouse updating.

I hate these sorts of bugs where everything looks to be working but isnt ;)

Will follow up with more info, and perhaps we can turn this into a borderless starter Gist or something.

[BobbyAnguelov]

Thanks for the lead about the mouse position. So what seems to be happening is that once I return the "HTCAPTION" result from the hit-test, we stop receiving the WM_MOUSEMOVE message and receive WM_NCMOUSEMOVE messages instead. This seems to be standard behavior and dearImgui doesnt handle those messages because they are generally irrelevant for it. Adding this code fixes the issue but what I don't understand is why it was working intermittently.

    case WM_NCMOUSEMOVE:
    {
        // We need to call TrackMouseEvent in order to receive WM_MOUSELEAVE events
        bd->MouseHwnd = hwnd;
        if (!bd->MouseTracked)
        {
            TRACKMOUSEEVENT tme = { sizeof(tme), TME_LEAVE, hwnd, 0 };
            ::TrackMouseEvent(&tme);
            bd->MouseTracked = true;
        }
        POINT mouse_pos = { (LONG)GET_X_LPARAM(lParam), (LONG)GET_Y_LPARAM(lParam) };
        io.AddMousePosEvent((float)mouse_pos.x, (float)mouse_pos.y);
        break;
    }

I'm not a win32 expert so would be good for someone else to validate this but for now it fixes the problem. I can easily add that to my client code but the default imgui impl could also be updated with it as it probably doesnt hurt.

[ocornut]

Ah! It turns out we have a thread discussing just this: #6045
I’ll resume testing this, and may ask you for confirmation that the new logic works with your setup.

[ocornut]

I'm not a win32 expert so would be good for someone else to validate this but for now it fixes the problem. I can easily add that to my client code but the default imgui impl could also be updated with it as it probably doesnt hurt.

May be wrong and lose mouse pos when app is unfocused, I'll post solution/fix in win32 backend once I find it (right now I don't have it).

[BobbyAnguelov]

Yeh, I don't know either, so far it seems to fix the issue, but I'll keep testing and see if I can trigger any further issues. At least now that I know what the core problem is, it's easy enough to debug and I will just patch the wndProc (and update this issues) if/as needed.