Environment

Windows build number: Version 10.0.18362.719

Steps to reproduce

Compile and run the following C++ code.

#include <windows.h>
#include <conio.h>
#include <stdio.h>

void main()
{
  constexpr auto numColors = 24;
  HANDLE handle = GetStdHandle(STD_OUTPUT_HANDLE);

  // Enable VT mode.
  DWORD mode;
  GetConsoleMode(handle, &mode);
  mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
  SetConsoleMode(handle, mode);

  printf("\033[m\033[H\033[2J\033[3J"); // clear screen and scrollback
  for (auto i = 0; i < numColors; i++) {
    auto c = i*10 + 8;
    printf("\033[48;2;%d;%d;%dm ", c, c, c);
  }
  printf("\033[m <- Colors produced with VT sequences\n");

  CHAR_INFO info[numColors];
  SMALL_RECT region = { 0, 0, numColors, 1 };
  ReadConsoleOutput(handle, info, { numColors, 1 }, { 0, 0 }, &region);

  for (auto i = 0; i < numColors; i++) {
    SetConsoleTextAttribute(handle, info[i].Attributes);
    printf(" ");
  }
  printf("\033[m <- Colors seen by ReadConsoleOutput\n");

  WORD attr[numColors];
  DWORD read;
  ReadConsoleOutputAttribute(handle, attr, numColors, { 0, 0 }, &read);

  for (auto i = 0; i < numColors; i++) {
    SetConsoleTextAttribute(handle, attr[i]);
    printf(" ");
  }
  printf("\033[m <- Colors seen by ReadConsoleOutputAttribute\n");
}

This test case writes out a line of spaces in varying shades of gray using VT sequences to set the colors with RGB values. It then reads back those colors as legacy attributes using the ReadConsoleOutput and ReadConsoleOutputAttribute APIs, and writes out what they see.

Expected behavior

The two lines rendered with legacy attributes should at least approximate the RGB shades of gray.

Actual behavior

The ReadConsoleOutput API produces reasonable results (given the limitations of the 16 color palette), but the colors returned by the ReadConsoleOutputAttribute API are quite obviously wrong.

The problem is that ReadConsoleOutputAttribute is using the TextAttribute::GetLegacyAttributes method, which only works correctly for the first 16 indexed colors. ReadConsoleOutput uses the Settings::GenerateLegacyAttributes method, which produces decent results, although it can be quite slow.

I'd like to have a go at refactoring this code for a number of reasons:

1. We obviously should be using a consistent legacy color calculation, that produces reasonable results, for all the console APIs.
2. I think there are a few places in the code that are using TextAttribute::GetLegacyAttributes that don't actually need legacy attributes at all, and could avoid the color narrowing.
3. I think the GenerateLegacyAttributes implementation could be made faster with a simple table lookup, instead of the current search for the closest HSL match in the color table (I suspect this might be responsible for issues like Performance regression for Conhostv2 wrapped by ConEmu #783 and vim slower to open/close in conhost v2 compared to v1 #3950).