Repository navigation
Conversation
Protect host Arrow owners during exception unwinding without adding synchronization to successful ownership transfers. Cover allocation failure after an earlier asynchronous transfer, stream completion, and recovery. Signed-off-by: Alexander Droste <[email protected]>
Move Parquet reading, writing, and source generation out of shared utilities. Update query includes, writer calls, and benchmark build sources while preserving the existing Parquet execution paths and managed-memory generation. Signed-off-by: Alexander Droste <[email protected]>
Extract reader-injected query execution while retaining the legacy Parquet wrappers and pushdown behavior. Add named-table generation for shared-format fixtures, with a scoped managed pool for callback-based generation and an explicit resource for returned lineitem tables. Signed-off-by: Alexander Droste <[email protected]>
Add vortex_writer_options and write_vortex(options, stream, mr), an optional pinned Vortex dependency, and writer tests independent of benchmark builds. Reuse bounded Arrow host staging with CPU compression and synchronous file finalization. Document local-file, flat-type, device-0, shared-library and experimental-layout limitations, allocation semantics, and the disabled-feature error path. Keep embedded Vortex symbols private. Signed-off-by: Alexander Droste <[email protected]>
Add projected local GPU reads, Arrow Device import, owning cuDF materialization, and stream/owner cleanup. Delegate fixture writes to the public writer. Cover slices, nulls, bitmap boundaries, projections, failure recovery and concurrent reads. Build the adapter with cuDF warning flags and register its build-tree-only test with RAPIDS GPU resource accounting. Document runtime-library restrictions. Signed-off-by: Alexander Droste <[email protected]>
Add paired fixtures and independent CPU references for Q1, Q5, Q6, Q9 and Q10. Validate each selected format, workload and I/O path before timing, preserve Parquet pushdown coverage, and overlap independent Vortex table reads. Separate buffered/direct I/O from page-cache state, retain only one fixture scale per query, and measure wall time and RMM peaks after setup cleanup. Add cache and memory-counter regression tests and GPU-accounted query smoke tests. Keep Vortex-enabled query binaries build-tree-only and document comparison asymmetries and validation limitations. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Move the writer and its enabled-path tests out of libcudf, and configure Vortex only for enabled benchmark builds. Reader and writer now share one FFI archive per executable rather than embedding another copy in libcudf. Preserve the public API proposal on ad/vortex-public-writer and the independent Arrow cleanup on ad/arrow-host-export-cleanup. Retain the Arrow safety prerequisite in this branch. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Use aggregate settings instead of a public-style builder and accessors. Generate default names during metadata construction and compact string views in place. Keep writer behavior tests while dropping assertions about plain field assignment, and let async futures join on test error paths. Signed-off-by: Alexander Droste <[email protected]>
Share Q5/Q10 discounted revenue, inline Q6's single binary operation, and generate Q1/Q6 fixtures through the existing managed-memory callback. Preserve native query types and boundary checks, and drain generated-table use before the callback returns. Signed-off-by: Alexander Droste <[email protected]>
Keep query-specific read order and validation explicit while sharing the resource logger, I/O context, warmup, timed owner scopes, and common summaries. Preserve consumer synchronization before owner destruction and device synchronization before counter reset and timer stop. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Keep table_with_names::to_parquet and its original implementation, located in the explicit Parquet source directory. Avoid renaming native benchmark write calls merely to add Vortex support. Signed-off-by: Alexander Droste <[email protected]>
Undo unrelated deduplication of the native revenue helpers and fixture table lists. Retain shared execution needed by Vortex comparisons and simplifications confined to the new fixture and local benchmark paths. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Keep both export entry points and a copy above the CUDA 13 policy threshold, but use smaller inputs and remove unrelated successful-export checks. State the timing-based sentinel and nested-drain limits explicitly instead of claiming independent release-order coverage. Signed-off-by: Alexander Droste <[email protected]>
Pair each input, CPU input where needed, and independent expectation directly rather than selecting by row count or parallel-vector index. Preserve all 19 cases and 44 reader-policy/engine executions, and include the scenario name in validation errors. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Retain prefix, offset and empty slices, logical-content checks, actual character-buffer sizes and consumer-stream completion. Avoid building and comparing unrelated numeric columns in this focused test. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Replace validator self-tests and broad predicate/oracle boundary scenarios with a four-row known-result check through both Parquet and Vortex files. Keep generated-data projection/reference validation, selected-path validation and empty-SUM handling outside timing. Leave the native query and license header unchanged. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
Move adapter, writer, and utility tests into the existing cpp/tests directories and centralize their CTest registrations. Build unit tests independently of BUILD_BENCHMARKS; only query smoke tests require benchmark executables. Share private Vortex support between tests and benchmarks and update build instructions. Signed-off-by: Alexander Droste <[email protected]>
Restore core Arrow host export and its tests to upstream. Document the unresolved export failure-path safety prerequisite separately from the Vortex integration. Signed-off-by: Alexander Droste <[email protected]>
Link to the Vortex README for detailed comparison settings and limitations. Describe current behavior without PR history or before-and-after commentary. Signed-off-by: Alexander Droste <[email protected]>
Share query execution between benchmarks and ordinary GTests, keeping synthetic boundary and join cases outside benchmark setup. Configure shared data-generation support for tests independently of benchmarks and simplify local validation arguments. Signed-off-by: Alexander Droste <[email protected]>
Replace the custom adapter test runner and Arrow comparator with GTest and cuDF column checks. Preserve exact values while allowing redundant all-valid masks, retain stream-completion checks, and use existing CUDA error and benchmark NVTX helpers. Signed-off-by: Alexander Droste <[email protected]>
Use ConfigureTest for memory statistics and the cuDF GTest wrapper for CPU-only fixture tests. Reset cached test state before counting operations so repeated or shuffled fixture tests remain independent. Signed-off-by: Alexander Droste <[email protected]>
Keep build and run commands, test entry points, comparison semantics, and essential limitations. Remove repeated implementation and build-wiring detail. Signed-off-by: Alexander Droste <[email protected]>
Add concise file-level descriptions of fixture caching, query execution and references, private Vortex I/O, and regression-test coverage. Preserve code and existing license headers. Signed-off-by: Alexander Droste <[email protected]>
Condense reader and writer contracts and remove repeated helper descriptions while preserving file-level purpose, ownership, synchronization, I/O, and format constraints. Documentation only; no code or license changes. Signed-off-by: Alexander Droste <[email protected]>
Keep projection checks at their callers while preserving sequential/direct comparisons and temporary owner lifetimes. Signed-off-by: Alexander Droste <[email protected]>
Compose load_data and compute_profit directly in fixture validation and regression tests without changing native benchmark execution. Signed-off-by: Alexander Droste <[email protected]>
Remove Q5 support for silently ignoring unrelated tables and use checked map access for Q10 customers already guaranteed by order filtering. Signed-off-by: Alexander Droste <[email protected]>
Use cuDF representation sizes for decimal fixtures and combine per-column metadata and exact-value checks without removing assertions or stream-completion coverage. Signed-off-by: Alexander Droste <[email protected]>
Use plain GoogleTest and drop the unused GoogleMock and cuDF test-header dependencies while retaining all fixture lifetime and retry checks. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
github-actions
Bot
added
libcudf
Parquet vs. Vortex: hot and cold cache, SF1Measured on 2026-09-24 at cuDF commit Hardware and storage
MethodMean CPU wall time in milliseconds over 30 timed samples per state. Both formats use buffered I/O ( Hot means a read-only warmup of the OS page cache. Cold means input-file pages are evicted and zero resident pages verified before each timed iteration. Hardware and host-storage caches are not flushed. Timing includes reads, optional query execution, owning cuDF materialization, cleanup, and synchronization. Fixture generation, writes, validation, and eviction are excluded. Hot cache
Cold cache
|
|
cc: @GregoryKimball |
|
Thanks @0ax1 for sharing this POC. I've been studying it to generate some Nsight Systems profiles. The biggest signal I see is that the Vortex reader manages a higher HtoD throughput than the cuDF Parquet reader. The file sizes between Parquet and Vortex are similar, but the HtoD time is about cut in half. Here is what I'm seeing at SF10 for Q5 with sequential reads (DGX B200 single GPU).
I'm not sure the root cause, but it could be less synchronization in the HtoD pipelining from Rust compared to KvikIO with default settings. ( @kingcrimsontianyu , @vyasr perhaps there is something we can learn from https://github.com/vortex-data/vortex/blob/develop/vortex-cuda/src/pooled_read_at/file.rs and https://github.com/vortex-data/vortex/blob/develop/vortex-cuda/src/pinned.rs )
🤷 |
The idea is to avoid the stream sync immediately after each I/O, so that I/O from task k+1 on a worker overlaps H2D from task k. Before the remote multi-poll backend and the KvikIO still uses default streams for the H2D copy (https://github.com/rapidsai/kvikio/blob/main/cpp/src/detail/stream.cpp#L37) to maintain certain contract with the cuFile API and I advocate removal of this contract to be able to use non-blocking streams. Not sure if H2D on default streams would throttle the bandwidth. Perhaps we may tune up the |
|
@GregoryKimball & @kingcrimsontianyu Much appreciate both of you having a look at the POC! |
|
@kingcrimsontianyu thank you for the discussion of these IO signals today. I did a follow-up study with 32 threads and 16 MiB tasks. This brings execution for parquet down from 119 to 73 ms. 
Seems like this closes the IO gap. So maybe the decode-decompress difference is more like 73ms for parquet and 65ms for vortex. |
|
@GregoryKimball Thanks for the test. This might be another reason for us to add an io_uring backend for local I/O so as to maintain decent I/O performance at low thread count rapidsai/kvikio#828, in a similar way multi-poll does to remote I/O. |
|
Holding IO aside, the decompress/decode phase for Vortex is indeed faster for the projected lineitem file. About 10ms faster at SF10 on DGX B200. parquet (empty space is the decompression engine) vortex |
|
@0ax1 thanks for this PR, and apologies for the slow response here! These numbers certainly look promising. Regarding #23877, my read of this PR is that we can achieve almost all of the benefits that we want by using the Vortex C++ or Python libraries directly and just feeding the data into cuDF via the Arrow data interface (which seems to be what you're already doing in this PR). I think the main thing we have to be careful of is ensuring that we actually take zero-copy views of the Vortex data. Aside from that, I don't see any major challenges. Should we turn this pull request into an official cuDF example of using Vortex? We don't need to introduce Vortex as a dependency or introduce a Vortex reader in cuDF, we just need an example to point to showing how you can use the Vortex library itself to read Vortex files and then pass them zero-copy to cuDF. |
|
@vyasr @GregoryKimball Thanks again for having a look! We would love to keep the possibility open to upstream Vortex into cuDF but starting off as an official cuDF example sounds great actually. Shall I just open a new PR for that or how do you picture the flow here? |
|
Closing, as we decided to add Vortex as an example for now. |
Summary
Adds benchmark-private
write_vortex/read_vortexhelpers and Parquet/Vortex comparisons for Q1, Q5, Q6, Q9, and Q10, enabled byCUDF_WITH_VORTEX=ON(default OFF). No installed libcudf reader/writer API is introduced.I/O and HtoD approach
This POC uses Vortex’s own I/O, not
cudf::io::datasource. Local data reads use pooled pinned host buffers and asynchronous HtoD transfers before GPU decoding; metadata uses buffered host reads.The adapter currently accepts local paths only. It does not expose file-like objects, host/device-buffer sources, or remote objects through cuDF. Vortex has Rust-side host-buffer and object-store readers, but connecting those—or cuDF datasources—to this adapter is follow-up work. Direct I/O bypasses the OS data-page cache; it is not GPUDirect Storage.
Comparison and validation
Measures read-only and read-plus-query CPU wall time, including materialization, cleanup, and synchronization. Both formats use matched data and projections. Q5/Q9/Q10 overlap Vortex table reads; Parquet reads remain sequential.
Validated relevant builds/tests with Vortex OFF and ON, including tests without benchmarks. All five queries passed SF0.01 warm/cold buffered comparisons and separate warm direct-I/O Vortex runs; warm smoke tests cover every Q9 engine. Builds used the existing cache with the documented NVCC workaround.
SF100 Nsight Systems profiles are still outstanding; these runs establish correctness, not performance conclusions.
Current limitations include Linux/device 0, flat columns, an experimental file layout, build-tree runtime dependencies, and the documented host Arrow export failure-path lifetime issue.