Repository navigation
Conversation
The ParquetPushDecoder may request adjacent byte ranges (e.g. 4-534 and 534-1064) that can be merged into a single range (4-1064), reducing the number of object store I/O operations. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
run benchmarks |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing coalesce-push-decoder-ranges (9f06337) to c17c87c (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Which issue does this PR close?
N/A
Rationale for this change
The
ParquetPushDecodermay request adjacent byte ranges (e.g.4-534and534-1064) that can be merged into a single range (4-1064), reducing the number of object store I/O operations.What changes are included in this PR?
coalesce_ranges()function that merges adjacent/overlapping byte ranges in-placeNeedsDatabranch of thePushDecoderStreamStatetransition loop inopener.rsAre these changes tested?
Yes, existing tests in
object_store_access.rsverify the coalesced ranges.Are there any user-facing changes?
No API changes. Users may see improved I/O performance when reading Parquet files due to fewer object store requests.
🤖 Generated with Claude Code