You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0a128b6
Browse filesBrowse the repository at this point in the historyBrowse files
Add `arrow-ipc` as an on-disk format in the compression-size,
compression-throughput, and random-access benchmark suites.
Compression benchmark runs emit Arrow IPC file sizes, serialization and
deserialization timings, and decoded in-memory Arrow bytes as
`uncompressed_bytes`. Random-access runs read row-addressable Arrow IPC
files.
The decoded Arrow byte count supplies logical throughput and
compression-ratio estimates. The Arrow IPC rows remain file-format
results. The develop workflow emits these records across the existing
dataset matrix.
---------
Signed-off-by: Will Manning <[email protected]>
Copy file name to clipboardExpand all lines: benchmarks/compress-bench/README.md
+10-2Lines changed: 10 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,15 @@
1
1
# Compression benchmark
2
2
3
-
Measures compression and decompression throughput, plus resulting file sizes, for Vortex
4
-
versus Parquet (and optionally Lance) across a range of datasets: NYC taxi data, several
3
+
Measures compression and decompression throughput, plus resulting file sizes, for Vortex,
4
+
Parquet, uncompressed Arrow IPC, and optionally Lance.
5
+
6
+
[Arrow IPC](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) is Apache
7
+
Arrow's built-in file format, formerly called Feather V2. This suite writes it without
8
+
optional buffer compression. Its timings therefore isolate serialization and deserialization
9
+
without codec cost. Its file size provides the approximately 1x baseline for compression
10
+
ratios. Parquet provides the established reference for a compressed columnar representation.
Copy file name to clipboardExpand all lines: benchmarks/random-access-bench/README.md
+8-2Lines changed: 8 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,6 +3,12 @@
3
3
Measures point-lookup latency: fetching individual rows by index from a file, rather than
4
4
scanning it. This is the workload behind the `Random Access` PR comment.
5
5
6
+
[Arrow IPC](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) is Apache
7
+
Arrow's built-in file format, formerly called Feather V2. This suite writes it without
8
+
optional buffer compression, so it provides the established constant-time access reference.
9
+
Parquet provides the established reference for a compressed columnar representation. Together,
10
+
they let the suite compare Vortex and Lance against both ends of the storage trade-off.
11
+
6
12
Two access patterns are generated with a fixed seed (see [`src/main.rs`](./src/main.rs)):
7
13
8
14
-**correlated**: several clusters of consecutive indices scattered across the dataset,
@@ -11,8 +17,8 @@ Two access patterns are generated with a fixed seed (see [`src/main.rs`](./src/m
11
17
simulating lookups with no locality.
12
18
13
19
Each pattern runs over four datasets (`taxi`, `feature-vectors`, `nested-lists`,
14
-
`nested-structs`) in Parquet, Lance, and Vortex, both with a cached open file handle and
15
-
reopening the file perlookup. CI drives the full matrix via
20
+
`nested-structs`) in Arrow IPC, Parquet, Lance, and Vortex. Each format uses a cached open file
21
+
handle and a per-lookup reopen mode. CI drives the full matrix via
0 commit comments