Describe the bug

When ConfusionMatrixDisplay is deciding on text colors, it uses colormap entry 256 as the text color for labeling cells below the mean value. This is line 125 of metrics/_plot/confusion_matrix.py:
cmap_min, cmap_max = self.im_.cmap(0), self.im_.cmap(256)

This causes incorrect coloring and hard-to-read graphs if using a ListedColormap with more than 256 entries. (Which was necessary for my application because I needed to renormalize the colormap)

Steps/Code to Reproduce

import numpy as np
import matplotlib
import matplotlib.cm
from sklearn.metrics import ConfusionMatrixDisplay

def getMapWith(colors):
    ans = matplotlib.cm.get_cmap('gray')(np.linspace(0, 1, colors)) ** 0.5 
    return matplotlib.colors.ListedColormap(ans)
x = np.random.rand(3, 3)
disp = ConfusionMatrixDisplay(x)

disp.plot(cmap=getMapWith(256)) # Works great
disp.plot(cmap=getMapWith(1024)) # Can't read

Versions

System:
    python: 3.8.5 (default, Jul 28 2020, 12:59:40)  [GCC 9.3.0]
executable: /home/mcrouch/test2/env/bin/python
   machine: Linux-5.4.0-47-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 21.0.1
   setuptools: 54.1.2
      sklearn: 0.24.1
        numpy: 1.19.5
        scipy: 1.6.1
       Cython: None
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True