I think we agreed that "labels" is a bad name for what it does in cross-validation. We could rename it to groups in the model-selection module, akin to what we did in #7169.
This is kind of a "now or never". It definitely has cost. I think it might be worth it (I just had a discussion where someone was looking for this exact feature but couldn't find it because of the name).
@jnothman and @GaelVaroquaux certainly have opinions ;)