Note for Reviewers: For queries such as WITH tmp AS (SELECT DISTINCT col1, col2 FROM tbl LIMIT 1000) SELECT ..., at present we were not trimming groups by default which would have meant that the group-by could have grown really huge.

For such queries, we can always leverage group-trimming, and this change enables that.

Query taken from AggregatePlans.json:

The plan with v2 optimizer is better in the multiple ways:

1. We automatically can detect that the group-by column, col2, is the partitioning column. So the leaf stages execute the aggregate directly instead of splitting them.
2. We are automatically able to skip the exchange required for the final aggregation above the union.

In other words, what the existing optimizer is not able to achieve even with hints, we can achieve without any hints.

Shows effectiveness of auto-enabling group-trim for group-by queries with no agg calls.

have a question. Is it intentionally to select the newFieldIndices.get(0)?

  public static RelCollation apply(RelCollation relCollation, PinotDistMapping mapping) {
    if (relCollation.getKeys().isEmpty()) {
      return relCollation;
    }
    List<RelFieldCollation> newFieldCollations = new ArrayList<>();
    for (RelFieldCollation fieldCollation : relCollation.getFieldCollations()) {
      List<Integer> newFieldIndices = mapping.getTargets(fieldCollation.getFieldIndex());
      if (CollectionUtils.isEmpty(newFieldIndices)) {
        break;
      }
      newFieldCollations.add(fieldCollation.withFieldIndex(newFieldIndices.get(0)));
    }
    return RelCollations.of(newFieldCollations);

Yeah that was by design because I wanted to keep it simple at the time. For context, this can occur in scenarios such as follows (which are rare or unlikely):

Project(col1=$0, col2=$0)
   Sort(collation=[order by $0 desc])
      TableScan(col1)

In this case, the project could be said to be ordered by both col1 and col2, but for now I am only preserving one of the indices to keep things simple. But I think it should be okay to add all field indexes too. I can take that up in a follow-up.