Last time, we saw that one way to intercept the ESC in the standard dialog message loop is to use your own dialog message loop. However, you might not be able to do this, say, because the dialog procedure uses End­Dialog(), and the dialog exit code is not retrievable from a custom message loop.

The Is­Dialog­Message includes an extensibility point that lets you hook into the message processing. You can register a message filter hook and listen for MSGF_DIALOG­BOX.

Before processing a message, the Is­Dialog­Message function does a Call­Msg­Filter with the message that it is about to process and the filter code MSGF_DIALOG­BOX. If the filter result is nonzero (indicating that one of the hooks wanted to block default processing), then the Is­Dialog­Message returns without doing anything. This lets us grab the ESC from Is­Dialog­Message before it turns into an IDCANCEL.

Here’s our first attempt. (There will be more than one.)

HWND hdlgHook;
#define DM_ESCPRESSED (WM_USER + 100)

LRESULT CALLBACK DialogEscHookProc(int nCode, WPARAM wParam, LPARAM lParam)
{
    if (code == MSGF_DIALOGBOX) {
        auto msg = (MSG*)lParam;
        if (IsDialogESC(hdlgHook, msg)) {
            return SendMessage(hdlg, DM_ESCPRESSED, 0, lParam);
        }
    }
    return CallNextHookEx(nullptr, nCode, wParam, lParam);
}

Our hook procedure first checks that it’s being called by Is­Dialog­Message. if so, and the message is a press of the ESC key destined for our dialog box (or a control on that dialog box), then send the dialog box a DM_ESC­PRESSED message to ask it what it thinks. The dialog procedure can return TRUE to block default processing or FALSE to allow default processing to continue.

Here is the handler in the dialog procedure itself:

INT_PTR CALLBACK DialogProc(HWND hdlg, UINT message, WPARAM wParam, LPARAM lParam)
{
    switch (message) {
    case WM_INITDIALOG:
        hdlgHook = hdlg;
        ⟦ other dialog initialization as before ⟧
        ⟦ ending with "return (whatever)" ⟧

    case DM_ESCPRESSED:
        if (⟦ we want to process the ESC key ourselves ⟧) {
            ⟦ do custom ESC key processing ⟧
            SetWindowLongPtr(hdlg, DWLP_MSGRESULT, TRUE);
            return TRUE;
        }
        break;
    ⟦ handle other messages ⟧
    }
    return FALSE;
}

When the dialog initializes, remember its handle as the dialog for which the Dialog­Esc­Hook­Proc is operating.

When the dialog is informed that the ESC key was pressed, we decide whether we want to process the ESC key ourselves. If so, then we do that custom processing and set up to return TRUE from the window procedure. For dialog procedures, this is done by setting the message result to the desired window procedure result and then returning TRUE to block default dialog box message processing and instead return the value we set (which is TRUE) from the window procedure.

Finally, we install the message hook before we create the dialog box and remove it when the dialog box dismisses.

auto hook = SetWindowsHookEx(WM_MSGFILTER, DialogEscHookProc,
                             nullptr, GetCurrentThreadId());
auto result = DialogBox(hinst, MAKEINTRESOURCE(IDD_WHATEVER),
                        hwndOwner, DialogProc);
UnhookWindowsHookEx(hook);

This is the basic idea, but we see that there are a few problems.

One is that we are communicating the dialog box handle through a global variable. This means that we can’t have multiple threads using this hook at the same time. Fortunately, that can be fixed by changing the variable to be thread_local, although this does drag in the cost of thread-local variables.

But even if we do that, we have a problem if two copies of this dialog box are shown by the same thread. For example, one of the controls in the dialog might launch another copy of this dialog, but with different parameters. For example, a “View certificate” dialog might have a button called “View parent certificate”.

We’ll take up these issues (and others) next time.