Skip to content

ServerApplication::hasConsole() returns false when stdout is closed (Windows) #5198

@aleks-f

Description

@aleks-f

Summary

On Windows, ServerApplication::hasConsole() only checks GetStdHandle(STD_OUTPUT_HANDLE). When a child process is launched with PROCESS_CLOSE_STDOUT (e.g., via ProcessRunner with NO_OUT option), stdout is NULL even though the process has a valid console window and stdin handle. This causes hasConsole() to return false, which triggers isService() to call StartServiceCtrlDispatcherW(), blocking indefinitely for a non-service process.

Reproduction

  1. Launch a ServerApplication-based process using ProcessRunner with the default NO_OUT option (PROCESS_CLOSE_STDOUT | PROCESS_CLOSE_STDERR)
  2. The child process has consoleWindow=YES, stdin=YES, stdout=NULL, stderr=NULL
  3. hasConsole() returns false because it only checks stdout
  4. isService() calls StartServiceCtrlDispatcherW() which blocks
  5. The process never reaches main()

Root cause

In Util/src/ServerApplication.cpp:

bool ServerApplication::hasConsole()
{
    HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
    return hStdOut != INVALID_HANDLE_VALUE && hStdOut != NULL;
}

This only checks stdout. When the parent launches the child with PROCESS_CLOSE_STDOUT, the child's stdout is NULL, but the process still has a valid console (inherited from the parent).

Fix

Check GetConsoleWindow() first (definitive proof of a console), then fall back to checking all three standard handles:

bool ServerApplication::hasConsole()
{
    if (GetConsoleWindow() != NULL) return true;
    HANDLE handles[] = {
        GetStdHandle(STD_OUTPUT_HANDLE),
        GetStdHandle(STD_INPUT_HANDLE),
        GetStdHandle(STD_ERROR_HANDLE)
    };
    for (HANDLE h : handles)
    {
        if (h != INVALID_HANDLE_VALUE && h != NULL)
            return true;
    }
    return false;
}

Affected files

  • platform/Util/src/ServerApplication.cpp - hasConsole() method

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions