Skip to content

Commit 9c12160

Browse files
string-bench: benchmark onpair-16 and add the OnPair paper datasets (#10920)
Signed-off-by: Joe Isaacs <[email protected]>
1 parent ec1012f commit 9c12160

10 files changed

Lines changed: 325 additions & 140 deletions

File tree

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎benchmarks/string-bench/Cargo.toml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,15 +20,17 @@ clap = { workspace = true, features = ["derive"] }
2020
futures = { workspace = true }
2121
indicatif = { workspace = true }
2222
mimalloc = { workspace = true }
23-
parquet = { workspace = true }
23+
parquet = { workspace = true, features = ["async", "object_store", "zstd"] }
2424
regex = { workspace = true }
2525
tabled = { workspace = true, features = ["std"] }
2626
tokio = { workspace = true, features = ["full"] }
2727
tracing = { workspace = true }
28+
url = { workspace = true }
2829
vortex = { workspace = true }
2930
vortex-arrow = { workspace = true }
3031
vortex-bench = { workspace = true }
3132
vortex-btrblocks = { workspace = true }
33+
vortex-cloud = { workspace = true, features = ["hf"] }
3234
vortex-fsst = { workspace = true }
3335
vortex-onpair = { workspace = true }
3436

‎benchmarks/string-bench/README.md‎

Lines changed: 28 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,11 @@ Compares Vortex string encoders on configured real-world Utf8 columns.
77

88
## What it measures
99

10-
Three metrics per (column, encoder), from the default `vortex` suite:
10+
Three metrics per (column, encoder), from the default `vortex` suite. OnPair
11+
runs twice, with 12-bit and 16-bit dictionary budgets (`onpair-12`,
12+
`onpair-16`); FSST runs once (`fsst`). Both suites force the configured scheme
13+
rather than letting btrblocks choose, so `onpair-16` measures a budget the
14+
default compressor does not use.
1115

1216
| Metric | Meaning | Table unit | `gh-json` unit |
1317
| --- | --- | --- | --- |
@@ -65,16 +69,30 @@ writes in chunks and compresses those children.
6569

6670
The current input catalog is:
6771

68-
| Output name | Source |
69-
| --- | --- |
70-
| `clickbench/URL/shard-N` | ClickBench `hits_N.parquet`, `URL` column |
71-
| `tpch/l_comment` | TPC-H SF1 `lineitem`, `l_comment` column |
72+
| Output name | Source | Rows |
73+
| --- | --- | --- |
74+
| `clickbench/URL/shard-N` | ClickBench `hits_N.parquet`, `URL` column | ~1M |
75+
| `tpch/l_comment` | TPC-H SF1 `lineitem`, `l_comment` column | 6M |
76+
| `amazon/book_reviews` | Amazon Reviews 2023, Books reviews, `text` | first 200k |
77+
| `amazon/book_titles` | Amazon Reviews 2023, Books metadata shard 0, `title` | 159k |
78+
| `news/headlines` | "A Million News Headlines" (ABC News), `headline_text` | 1.23M |
79+
| `twitter/sentiment140` | Sentiment140 tweets, train split, `text` | 1.36M |
80+
| `fineweb/text` | FineWeb `sample/10BT/001_00000.parquet`, `text` | first 10k |
81+
82+
The Amazon, news, and tweet inputs are four of the five datasets the
83+
[OnPair paper](https://arxiv.org/abs/2508.02280) evaluates on. The fifth,
84+
OpenWebText URLs, has no public parquet mirror; ClickBench `URL` covers that
85+
shape. FineWeb adds long web documents.
7286

7387
On first use, the selected ClickBench shard is downloaded through
7488
`vortex-bench`'s shared idempotent downloader. TPC-H SF1 is generated
75-
deterministically through the shared TPC-H dataset helper. Both are stored
76-
under `vortex-bench/data/`, which is gitignored, and reused on later runs.
77-
Input preparation is outside benchmark timing.
89+
deterministically through the shared TPC-H dataset helper. The Hugging Face
90+
inputs are read from parquet files pinned to a commit or tag, with range
91+
requests, so a row prefix of a large file (FineWeb's shard is 2 GB) fetches
92+
only the row groups it needs; the selected column is cached as zstd parquet
93+
under `vortex-bench/data/string-bench/`. All of these are stored under
94+
`vortex-bench/data/`, which is gitignored, and reused on later runs. Input
95+
preparation is outside benchmark timing.
7896

7997
## Running locally
8098

@@ -107,6 +125,7 @@ The develop benchmark workflow runs the default suite after each merge to
107125
`develop` and publishes the results to the shared benchmark history.
108126

109127
Metric names are `<metric>/<input>/<encoder>`, for example
110-
`read/clickbench/URL/shard-0/onpair-12` and `size/tpch/l_comment/fsst`. The unit
128+
`read/clickbench/URL/shard-0/onpair-12`, `read/clickbench/URL/shard-0/onpair-16`,
129+
and `size/tpch/l_comment/fsst`. The unit
111130
is reported separately in the JSON output and the CI table, which groups rows by
112131
unit — so metric-first names keep the rows you compare adjacent.

‎benchmarks/string-bench/src/codec.rs‎

Lines changed: 6 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -22,63 +22,18 @@ use vortex_bench::Format;
2222
use vortex_bench::measurements::CustomUnitMeasurement;
2323
use vortex_fsst::fsst_compress;
2424
use vortex_fsst::fsst_train_compressor;
25-
use vortex_onpair::Config;
26-
use vortex_onpair::DEFAULT_CONFIG;
27-
use vortex_onpair::MaxDictBits;
2825
use vortex_onpair::onpair_compress;
2926

27+
use crate::Candidate;
3028
use crate::SESSION;
3129
use crate::StringColumn;
32-
use crate::StringEncoder;
3330
use crate::duration_ms;
3431
use crate::median;
35-
use crate::onpair_label;
3632
use crate::prepare_column;
3733
use crate::throughput;
3834
use crate::verify_canonicalized;
3935

40-
/// A fully configured direct array-compression candidate: an encoder family
41-
/// plus the fixed configuration that path needs.
42-
pub enum DirectCandidate {
43-
/// OnPair with a deterministic config, whose dictionary budget is the only
44-
/// thing the benchmark varies.
45-
OnPair(Config),
46-
/// FSST (no configuration).
47-
Fsst,
48-
}
49-
50-
impl DirectCandidate {
51-
/// OnPair with a dictionary budget of up to `2^max_dict_bits` tokens,
52-
/// reusing every other default. `max_dict_bits` must be in `9..=16`; OnPair
53-
/// stores codes as `u16`, so all of those fit.
54-
pub fn on_pair(max_dict_bits: u8) -> Result<Self> {
55-
let max_dict_bits = MaxDictBits::new(max_dict_bits).map_err(|e| {
56-
anyhow::anyhow!(
57-
"invalid maximum dictionary bit width {max_dict_bits} (want 9..=16): {e}"
58-
)
59-
})?;
60-
Ok(Self::OnPair(Config {
61-
max_dict_bits,
62-
..DEFAULT_CONFIG
63-
}))
64-
}
65-
66-
/// The encoder family, for the post-compression type check.
67-
fn family(&self) -> StringEncoder {
68-
match self {
69-
Self::OnPair(_) => StringEncoder::OnPair,
70-
Self::Fsst => StringEncoder::Fsst,
71-
}
72-
}
73-
74-
/// Stable label used in benchmark output, e.g. `onpair-12` or `fsst`.
75-
fn label(&self) -> String {
76-
match self {
77-
Self::OnPair(config) => onpair_label(config),
78-
Self::Fsst => StringEncoder::Fsst.label().to_string(),
79-
}
80-
}
81-
36+
impl Candidate {
8237
/// Run the encoder's direct array-level train + compress path.
8338
pub(crate) fn compress(&self, array: &ArrayRef, ctx: &mut ExecutionCtx) -> Result<ArrayRef> {
8439
match self {
@@ -168,7 +123,7 @@ pub fn bench_column(
168123
column: &StringColumn,
169124
iterations: usize,
170125
warmup: usize,
171-
candidate: &DirectCandidate,
126+
candidate: &Candidate,
172127
verify: bool,
173128
) -> Result<ColumnResult> {
174129
crate::validate_iterations(iterations)?;
@@ -240,8 +195,9 @@ mod tests {
240195
let (column, expected_uncompressed_bytes) = crate::repeated_fixture();
241196

242197
for (candidate, expected_label) in [
243-
(DirectCandidate::on_pair(12)?, "onpair-12"),
244-
(DirectCandidate::Fsst, "fsst"),
198+
(Candidate::on_pair(12)?, "onpair-12"),
199+
(Candidate::on_pair(16)?, "onpair-16"),
200+
(Candidate::Fsst, "fsst"),
245201
] {
246202
let result = bench_column(&column, 1, 0, &candidate, true)?;
247203

‎benchmarks/string-bench/src/lib.rs‎

Lines changed: 172 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
//! ("Codec diagnostic").
2222
2323
use std::path::PathBuf;
24+
use std::sync::Arc;
2425
use std::sync::LazyLock;
2526
use std::time::Duration;
2627

@@ -29,9 +30,15 @@ use anyhow::bail;
2930
use clap::ValueEnum;
3031
use futures::StreamExt;
3132
use futures::TryStreamExt;
33+
use parquet::arrow::ArrowWriter;
3234
use parquet::arrow::ParquetRecordBatchStreamBuilder;
3335
use parquet::arrow::ProjectionMask;
36+
use parquet::arrow::async_reader;
37+
use parquet::basic::Compression;
38+
use parquet::basic::ZstdLevel;
39+
use parquet::file::properties::WriterProperties;
3440
use tokio::fs::File;
41+
use url::Url;
3542
use vortex::VortexSessionDefault;
3643
use vortex::aggregate_fn::fns::uncompressed_size_in_bytes::UncompressedSizeOpts;
3744
use vortex::aggregate_fn::fns::uncompressed_size_in_bytes::uncompressed_size_in_bytes;
@@ -50,13 +57,16 @@ use vortex_bench::IdempotentPath;
5057
use vortex_bench::datasets::Dataset;
5158
use vortex_bench::datasets::data_downloads::download_data;
5259
use vortex_bench::datasets::tpch_l_comment::TPCHLCommentCanonical;
60+
use vortex_bench::idempotent_async;
61+
use vortex_cloud::hf::make_hf_store_from_url;
5362
use vortex_fsst::FSST;
5463
use vortex_onpair::Config;
64+
use vortex_onpair::DEFAULT_CONFIG;
65+
use vortex_onpair::MaxDictBits;
5566
use vortex_onpair::OnPair;
5667

5768
mod codec;
5869
pub use codec::ColumnResult;
59-
pub use codec::DirectCandidate;
6070
pub use codec::bench_column;
6171

6272
const CLICKBENCH_SHARD_COUNT: u32 = 100;
@@ -118,6 +128,52 @@ impl std::fmt::Display for StringEncoder {
118128
}
119129
}
120130

131+
/// A fully configured encoder candidate, run by both suites: an encoder family
132+
/// plus the fixed configuration that path needs.
133+
pub enum Candidate {
134+
/// OnPair with a deterministic config, whose dictionary budget is the only
135+
/// thing the benchmark varies.
136+
OnPair(Config),
137+
/// FSST (no configuration).
138+
Fsst,
139+
}
140+
141+
impl Candidate {
142+
/// OnPair with a dictionary budget of up to `2^max_dict_bits` tokens,
143+
/// reusing every other default. `max_dict_bits` must be in `9..=16`; OnPair
144+
/// stores codes as `u16`, so all of those fit.
145+
pub fn on_pair(max_dict_bits: u8) -> Result<Self> {
146+
let max_dict_bits = MaxDictBits::new(max_dict_bits).map_err(|e| {
147+
anyhow::anyhow!(
148+
"invalid maximum dictionary bit width {max_dict_bits} (want 9..=16): {e}"
149+
)
150+
})?;
151+
Ok(Self::OnPair(Config {
152+
max_dict_bits,
153+
..DEFAULT_CONFIG
154+
}))
155+
}
156+
157+
/// The encoder family, for the post-compression type check.
158+
pub fn family(&self) -> StringEncoder {
159+
match self {
160+
Self::OnPair(_) => StringEncoder::OnPair,
161+
Self::Fsst => StringEncoder::Fsst,
162+
}
163+
}
164+
165+
/// Stable label used in benchmark output, e.g. `onpair-12` or `fsst`.
166+
/// Changing a candidate's config renames its metrics and so restarts their
167+
/// benchmark history, which is intended: neither size nor speed is
168+
/// comparable across dictionary budgets.
169+
pub fn label(&self) -> String {
170+
match self {
171+
Self::OnPair(config) => onpair_label(config),
172+
Self::Fsst => StringEncoder::Fsst.label().to_string(),
173+
}
174+
}
175+
}
176+
121177
/// Label for an OnPair configuration, e.g. `onpair-12`. The dictionary budget is
122178
/// read from the config that was actually used, so a label can never claim a
123179
/// budget the measured data was not encoded with.
@@ -291,6 +347,121 @@ pub async fn load_tpch_l_comment(ctx: &mut ExecutionCtx) -> Result<StringColumn>
291347
})
292348
}
293349

350+
/// One Utf8 column of a public parquet file on the Hugging Face Hub.
351+
pub struct HfColumn {
352+
/// Output name used in measurement names and matched by `--columns`.
353+
pub name: &'static str,
354+
/// `hf://` URL of the parquet file, pinned to a commit or tag so the input
355+
/// cannot change under the benchmark history.
356+
pub url: &'static str,
357+
/// Column to read.
358+
pub column: &'static str,
359+
/// Keep only the first `max_rows` rows. The file is read with range
360+
/// requests, so a prefix of a large file fetches only the row groups it
361+
/// needs.
362+
pub max_rows: Option<usize>,
363+
}
364+
365+
/// The datasets the OnPair paper (Gargiulo & Venturini, "OnPair: Short Strings
366+
/// Compression for Fast Random Access", arXiv:2508.02280) evaluates on, from
367+
/// public parquet mirrors, plus a slice of FineWeb as long-document web text.
368+
///
369+
/// The paper's fifth dataset, OpenWebText URLs, has no parquet mirror; the
370+
/// ClickBench `URL` column covers that shape.
371+
pub const HF_COLUMNS: &[HfColumn] = &[
372+
// Amazon Reviews 2023 (Hou et al. 2024), Books. The paper takes the first
373+
// 500 MiB of reviews; 200k reviews is about 80 MiB.
374+
HfColumn {
375+
name: "amazon/book_reviews",
376+
url: "hf://datasets/cogsci13/Amazon-Reviews-2023-Books-Review@df2faf70120558bddedb83c4d798385e5036b821/data/full-00000-of-00033.parquet",
377+
column: "text",
378+
max_rows: Some(200_000),
379+
},
380+
// Amazon Reviews 2023, Books item titles: the first of 28 metadata shards.
381+
HfColumn {
382+
name: "amazon/book_titles",
383+
url: "hf://datasets/cogsci13/Amazon-Reviews-2023-Books-Meta@35db8e29f8681b8694da79558a5b86a381e6e260/data/full-00000-of-00028.parquet",
384+
column: "title",
385+
max_rows: None,
386+
},
387+
// "A Million News Headlines" (ABC News), the whole dataset.
388+
HfColumn {
389+
name: "news/headlines",
390+
url: "hf://datasets/rajistics/million-headlines@refs%2Fconvert%2Fparquet/default/train/0000.parquet",
391+
column: "headline_text",
392+
max_rows: None,
393+
},
394+
// Sentiment140 tweets: the 1.36M-row train split of the 1.6M tweets.
395+
HfColumn {
396+
name: "twitter/sentiment140",
397+
url: "hf://datasets/adilbekovich/Sentiment140Twitter@refs%2Fconvert%2Fparquet/default/train/0000.parquet",
398+
column: "text",
399+
max_rows: None,
400+
},
401+
// FineWeb document text: the first 10k documents (about 30 MiB) of the
402+
// shard the FineWeb query benchmark uses.
403+
HfColumn {
404+
name: "fineweb/text",
405+
url: "hf://datasets/HuggingFaceFW/[email protected]/sample/10BT/001_00000.parquet",
406+
column: "text",
407+
max_rows: Some(10_000),
408+
},
409+
];
410+
411+
/// Reader for a remote parquet file, used only to fill the local input cache.
412+
#[expect(
413+
deprecated,
414+
reason = "arrow-rs deprecated this in favour of a hand-rolled AsyncFileReader; a one-time, \
415+
untimed input download does not need one (arrow-rs#10308)"
416+
)]
417+
type ObjectReader = async_reader::ParquetObjectReader;
418+
419+
/// Load one Hugging Face column, fetching and caching it locally first if it
420+
/// is not already present.
421+
pub async fn load_hf_column(source: &HfColumn, ctx: &mut ExecutionCtx) -> Result<StringColumn> {
422+
let path = cache_hf_column(source).await?;
423+
let struct_array = read_parquet_projected(path, source.column).await?;
424+
Ok(StringColumn {
425+
name: source.name.to_string(),
426+
array: to_utf8_column(struct_array, source.column, ctx)?,
427+
})
428+
}
429+
430+
/// Read `source`'s rows over HTTP and cache just that column as a local
431+
/// zstd-compressed parquet file under `vortex-bench/data/string-bench/`.
432+
///
433+
/// The cache file is named after the column, so changing a source's URL or row
434+
/// cap needs the stale file deleted.
435+
async fn cache_hf_column(source: &HfColumn) -> Result<PathBuf> {
436+
let cache_path = format!("string-bench/{}.parquet", source.name.replace('/', "_"));
437+
idempotent_async(cache_path, |tmp| async move {
438+
let (store, path) = make_hf_store_from_url(&Url::parse(source.url)?)?;
439+
let builder = ParquetRecordBatchStreamBuilder::new(ObjectReader::new(store, path)).await?;
440+
let col_idx = builder.schema().index_of(source.column)?;
441+
let mask = ProjectionMask::roots(builder.parquet_schema(), [col_idx]);
442+
let mut builder = builder.with_projection(mask);
443+
if let Some(max_rows) = source.max_rows {
444+
builder = builder.with_limit(max_rows);
445+
}
446+
let mut reader = builder.build()?;
447+
448+
let props = WriterProperties::builder()
449+
.set_compression(Compression::ZSTD(ZstdLevel::try_new(3)?))
450+
.build();
451+
let mut writer = ArrowWriter::try_new(
452+
std::fs::File::create(&tmp)?,
453+
Arc::clone(reader.schema()),
454+
Some(props),
455+
)?;
456+
while let Some(batch) = reader.try_next().await? {
457+
writer.write(&batch)?;
458+
}
459+
writer.close()?;
460+
Ok(())
461+
})
462+
.await
463+
}
464+
294465
/// Read a single column from a parquet file (projected at the parquet level to
295466
/// avoid decoding the other columns) into a chunked struct array.
296467
async fn read_parquet_projected(path: PathBuf, column: &str) -> Result<ArrayRef> {

0 commit comments

Comments
 (0)