Commit 0179f67

`dict_encoding_sweep` compares `Dictionary<Int32, Utf8>` against `Utf8View` for

the same operator across a cardinality sweep. Most groups generate two

in-memory tables (one per encoding) that share values, SQL text, and seed. The

`parquet_scan` group instead writes the same rows with both `PLAIN` and

`RLE_DICTIONARY` physical encodings, then scans each file as both Arrow output

representations. File generation is outside the timed region and the harness

uses a neutral `Utf8` writer schema so neither requested output representation

is inherited from Parquet metadata. It verifies every data page has the

requested physical encoding before recording results.

Warmup and timed iterations are interleaved between Arrow output encodings so

cache and thermal state stay symmetric. Median / min / p25 / p75 runtimes

(nanoseconds) are appended to a CSV.

The intent is to make it easy for other engineers to reproduce where dictionary

encoding wins, where it loses, and at which NDV it crosses over, without

touching the harness itself.

Pass `--group` to pick one of:

| Group | Queries |

|-------|---------|

| `scan` | `scan_k`, `scan_all`, `scan_count_star`, `scan_count_k` |

| `parquet_scan` | `scan_k`, `scan_all`; runs the `PLAIN`/`RLE_DICTIONARY` × `Utf8View`/dictionary matrix |

| `filter` | `filter_eq`, `filter_ne`, `filter_in10`, `filter_like_prefix`, `filter_gt`, `filter_1pct_group` |

| `project` | `proj_length`, `proj_substr`, `proj_upper`, `proj_concat`, `proj_case_like` |

| `aggregation` | `agg_count_distinct_k`, `agg_group_count_k`, `agg_group_sum_v_k`, `agg_group_count_kw`, `agg_having_k` |

| `sort` | `sort_k`, `sort_k_desc`, `sort_k_limit_1000`, `sort_kw`, `sort_v_by_k` |

| `join` | `join_inner`, `join_inner_project`, `join_left`, `join_semi`, `join_anti_half` |

| `limit` | `limit_k_100`, `limit_k_1k`, `limit_k_10k`, `limit_all_1k`, `limit_offset_50k` |

| `repartition` | `repart_group_k`, `repart_hash_join`, `repart_project_k` |

| `coalesce_batches` | `cb_filter_eq`, `cb_filter_project`, `cb_filter_group` |

| `coalesce_partitions` | `cp_scan_k`, `cp_count_star`, `cp_sum_v` |

| `spm` | `spm_sort_k`, `spm_sort_k_limit`, `spm_sort_v_by_k` |

Narrow to one query with `--query <id>`.

cargo build --release --bin dict_encoding_sweep -p datafusion-benchmarks

./target/release/dict_encoding_sweep \

--group filter --rows 1000000 \

--ndvs 10,100,500,1000,4000,8192 \

--iters 15 --warmups 3 \

--output /tmp/dict_sweep/filter_1m.csv

For scan-level Parquet decoding, one invocation produces four rows per query

and NDV: `PLAIN→Utf8View`, `PLAIN→Dictionary`,

`RLE_DICTIONARY→Utf8View`, and `RLE_DICTIONARY→Dictionary`.

./target/release/dict_encoding_sweep \

--group parquet_scan --query scan_k --rows 1000000 \

--ndvs 10,100,500,1000,2000,4000,8192 \

--iters 15 --warmups 3 \

--output /tmp/dict_sweep/parquet_scan_1m.csv

The generated Parquet files are uncompressed so the measurements emphasize

page decoding and Arrow array construction. `k` is the only column whose

physical encoding changes. An explicit target schema requests both Arrow

representations from both files, making source encoding and output

representation independent axes. Focused coverage also verifies the four

physical plans decode directly in `DataSourceExec`, without a projection cast

between `Utf8View` and dictionary arrays.

For a Criterion target suitable for the benchmark bot, the following command

compares native `Utf8View` and dictionary output from an `RLE_DICTIONARY`

Parquet source across NDVs 1 through 500,000:

The target fixes the input at 1,000,000 rows of 16-byte strings in two

500,000-row groups. It verifies the physical page encoding and rejects plans

that insert a representation cast above `DataSourceExec`.

Useful flags:

| Flag | Default | Notes |

|------|---------|-------|

| `--group` | required | Operator group (see table above). |

| `--query` | all | Restrict to a single query id within the group. |

| `--rows` | required | Rows in the generated table. |

| `--ndvs` | required | Comma-separated distinct-value counts to sweep. |

| `--width` | `16` | Fixed byte width per value. |

| `--nulls` | `0.0` | Null fraction in column `k`. |

| `--batch-size` | `8192` | Session + record-batch size. |

| `--target-partitions` | `1` | Set `>1` to force `RepartitionExec` nodes. |

| `--dict-chunk` | `500000` | Rows per shared dictionary values-array chunk; also the Parquet row-group size for `parquet_scan`. |

| `--iters` | `15` | Timed iterations per cell per encoding. |

| `--warmups` | `3` | Untimed warmup iterations per cell per encoding. |

The output CSV uses one row per `(encoding, cell)`. Columns:

`source_encoding` is `memory`, `parquet_plain`, or

`parquet_rle_dictionary`. Parquet rows include the generated file size in

`note`.

Pair the two rows for the same cell by `(group, query_id, rows, ndv,

width_bytes, null_rate, batch_size, dict_chunk_rows, source_encoding)` and divide

`dict.median_ns / view.median_ns` to get the per-cell speedup.