Repository navigation
Commit b987501
bench: add a ClickBench extended query for a grouped COUNT(DISTINCT) over a string (apache#25026)
## Which issue does this PR close?
No existing issue. This is benchmark coverage carved out of apache#24859 so
that it can land first: a benchmark query added in the same PR that
needs it cannot appear in an A/B run, because the bot compares against
the merge base and the merge base does not have the query.
## Rationale for this change
`COUNT(DISTINCT)` has a specialized `GroupsAccumulator` for the integer
types and for no other type. Every other type falls back to
`GroupsAccumulatorAdapter`, which holds one boxed `Accumulator`, and
therefore one hash table, for each group. The cost of that fallback is
per group, so the group cardinality is what decides how much it costs.
Nothing in either suite measures that:
- Extended Q2 is the only query that puts a `COUNT(DISTINCT)` on a
non-integer column at all. It groups by `BrowserCountry`, so it
exercises the adapter at a low group cardinality, which is where the
adapter is cheapest.
- Standard Q8, Q10, Q11 and Q13 hold a distinct aggregate that stands
alone, so `SingleDistinctToGroupBy` already rewrites them and they never
reach the adapter.
- Standard Q9 carries an `AVG`, which that rule has never accepted.
- Standard Q22 is the one query that pairs a lone distinct aggregate
with a non-distinct count, and it counts distinct `UserID`, an `Int64`,
which has a specialized accumulator.
So a lone `COUNT(DISTINCT <string>)` grouped by a high cardinality key,
next to a non-distinct `COUNT(*)`, is uncovered. That is an ordinary
analytics shape, it is the shape apache#24857 changed the memory profile of,
and it is the shape apache#24859 proposes to rewrite. Neither of those could
show its effect on any benchmark in this repository.
## What changes are included in this PR?
The query:
```sql
SELECT "SearchPhrase", COUNT(*) AS c, COUNT(DISTINCT "MobilePhoneModel") AS models
FROM hits
WHERE "SearchPhrase" <> ''
GROUP BY "SearchPhrase"
ORDER BY c DESC
LIMIT 10;
```
The extended ClickBench queries live in three places, and a query added
to only one of them is invisible to the others. This PR adds it to all
three:
- `benchmarks/queries/clickbench/extended/q14.sql`, which feeds `dfbench
clickbench --queries-path`. Queries are discovered from the directory,
so the runner needs no change.
-
`benchmarks/sql_benchmarks/clickbench_extended/benchmarks/q14.benchmark`,
which feeds `benchmark_runner clickbench_extended`. Structurally
identical to `q13.benchmark` apart from the query. Suite files are also
discovered from the directory; `benchmark_runner clickbench_extended
--list` now reports 15 queries.
- `datafusion/sqllogictest/test_files/clickbench_extended.slt`, which
runs the extended queries against the committed ten row
`clickbench_hits_10.parquet` fixture.
That last file had only ever mirrored q0 through q6. q7 through q13 were
added to the queries directory without a matching sqllogictest entry,
and q14 would have widened that gap, so this backfills q7 through q14
together. That is why the diff is larger than one query.
Having to add one query in three places is itself the problem, and the
copies have already drifted: extended q6 carries a cast in its
`sql_benchmarks` copy that the other two do not have. I filed apache#25031 for
that; it is out of scope here.
There is one pre-existing staleness this PR does not fix.
`datafusion/core/benches/sql_planner.rs` builds its ClickBench planning
set from a hardcoded `(0..=7)` over the extended directory, so extended
Q8 through Q13 were already outside it before this PR and Q14 joins
them. That is a separate cleanup.
## What is the testing strategy for this PR?
The sqllogictest entries are the test. `clickbench_extended.slt` runs
every extended query against the committed ten row fixture and asserts
its output, so q7 through q14 are now executed on every CI run rather
than only by whoever runs the benchmark suite by hand. `cargo test -p
datafusion-sqllogictest --test sqllogictests -- clickbench` passes, 2 of
2 files.
`cargo test -p datafusion-benchmarks` passes, 192 tests, and
`benchmark_runner clickbench_extended --list` reports the suite at 15
queries, confirming the new `.benchmark` file parses and is discovered.
I do not have a `hits.parquet` to hand, so I have not run the query
against the full dataset. The plan shape, which is the whole value of
the query, was checked separately: on an equivalent local table with the
same filter, grouping, `COUNT(*)` and string `COUNT(DISTINCT)`, the
aggregate keeps `aggr=[[count(Int64(1)), count(DISTINCT ...)]]`, which
is the `GroupsAccumulatorAdapter` path this query exists to measure, and
it is not rewritten away.
What I have not established is the effect size on the real dataset,
since that depends on how many distinct `SearchPhrase` values survive
the filter. If a reviewer with the data runs it, that number is worth
having on this PR.
`ci/scripts/doc_prettier_check.sh` passes on the README change.
## Are there any user-facing changes?
No. This adds a benchmark query and documentation only.
---------
Co-authored-by: Claude Opus 5 <[email protected]>1 parent 6ab4ce6 commit b987501
4 files changed
Lines changed: 139 additions & 0 deletions
File tree
- benchmarks
- queries/clickbench
- extended
- sql_benchmarks/clickbench_extended/benchmarks
- datafusion/sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
261 | 261 | | |
262 | 262 | | |
263 | 263 | | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
264 | 302 | | |
265 | 303 | | |
266 | 304 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
64 | 134 | | |
65 | 135 | | |
66 | 136 | | |
0 commit comments