gpu-scan-cli scan --gpu-file panics with "as_slice must be called on host buffer"

What happened?

Summary

gpu-scan-cli scan --gpu-file panics on every Vortex file I produce via gpu-scan-cli convert.

Immediate panic on every tokio worker:

2026-05-07T01:11:35Z INFO vortex_cuda: Registering CUDA kernels

thread 'tokio-rt-worker' panicked at vortex-error/src/lib.rs:361:13:
Assertion failed error: as_slice must be called on host buffer
0: PrimitiveData::as_slice
1: Primitive::scalar_at
2: ArrayRef::execute_scalar
3: Patches::chunk_offset_at
4: Patches::search_index
5: Patches::slice
6: BitPacked::slice
7: BitPacked::reduce_parent
8: ArrayRef::slice
9: FoR::slice
10: FoR::reduce_parent
11: ArrayRef::slice
12: CudaFlatReader::projection_evaluation
13: ChunkedReader::projection_evaluation
14: StructReader::projection_evaluation
15: scan::tasks::split_exec

(Many threads panic in parallel; backtrace is the same on each.)

Steps to reproduce

# 1. Build the tools                                      
cargo build --release -p vortex-tui -p gpu-scan-cli
                                                                                                                                                                                
# 2. Ingest a Parquet file (TPC-H SF100 `lineitem`)
./target/release/vx convert lineitem.parquet                                                                                                                                    
# → lineitem.vortex                                                                                                                                                   
                                                                                                                                                                                
# 3. Re-encode for CUDA                                                                                                                                                         
./target/release/gpu-scan-cli convert \                                                                                                                                         
  --input  lineitem.vortex \                                                                                                                                                    
  --output lineitem.cuda.vortex                                                                                                                                                 
# → succeeds                                                                                                                                                                    
                                                                                                                                                                                
# 4. Scan                                                 
RUST_LOG=info RUST_BACKTRACE=1 \
  ./target/release/gpu-scan-cli scan lineitem.cuda.vortex --gpu-file                                                                                                            
# → panics                                                    

### Environment

- Vortex commit a9dc3a2eb (develop)
- CUDA 13.0
- Linux aarch64 (NVIDIA GB10)                                                                                                                                                   
- Tested on TPC-H SF100 lineitem; reproduces consistently

### Additional context

I tried excluding ALPScheme and ALPRDScheme from only_cuda_compatible(), thinking the patches were ALP-related, but the same panic still occurs — looks like integer columns also trigger it.

Sorry about this we are in the process of migrating these. Ill have a fix up soon

fixed here: #7839

@joseph-isaacs

Thanks @joseph-isaacs for the quick turnaround on #7839! Pulled the latest develop (ff120401a, which includes the fix) and re-tested.

Integer columns now scan cleanly.
The original backtrace through BitPacked::reduce_parent is gone. Confirmed by re-running the same gpu-scan-cli scan --gpu-file against my regenerated .cuda.vortex file

However, the same panic still triggers if ALP-encoded float columns are present.

New backtrace:

thread 'main' panicked at vortex-error/src/lib.rs:361:13:
Assertion failed error: as_slice must be called on host buffer
0: PrimitiveData::as_slice
1: Primitive::scalar_at
2: ArrayRef::execute_scalar
3: Patches::chunk_offset_at
4: Patches::search_index
5: Patches::slice
6: vortex_cuda::dynamic_dispatch::plan_builder::FusedPlan::walk_slice
7: vortex_cuda::dynamic_dispatch::plan_builder::FusedPlan::walk
8: vortex_cuda::dynamic_dispatch::plan_builder::DispatchPlan::new
9: ArrayRef::execute_cuda
10: gpu_scan_cli::cmd_scan

So the integer fix works, but ALP appears to take a different code path that's still affected.

@manish-sethi

@manish-sethi I'll have a look!

fix[gpu]: CUDA sliced patch views with device patches (#7911)

@0ax1

Thanks @0ax1 for the fix in #7911. — The original as_slice on host buffer panic (#7838) no longer reproduces for ALP.

However, there seems to be a race condition that causes non-deterministic behavior. At times, it passes the scan, but more often it hangs, and rarely it panics.

Following is the summary of hang via gdb

  - 20 tokio-rt-worker threads all parked in `futex_wait`. None in vortex code.                                                                                                                                  
  - 1 CUDA worker thread (`cuda00001400006`) waiting in `__poll` inside libcuda.
  - 1 internal CUDA thread waiting on `__new_sem_wait_slow64` inside libcuda.                                                                                                                                    
  - Main thread blocked in `tokio::runtime::park` → `block_on`.

Following is the trace when it panics - Multiple workers panic simultaneously with the same message.

 thread 'tokio-rt-worker' panicked at
    vortex-cuda/src/pinned.rs:300:33:
  Other error: failed to access pinned host buffer:                                                                                                                                                              
  DriverError(CUDA_ERROR_ILLEGAL_ADDRESS,                                                                                                                                                                        
              "an illegal memory access was encountered")

Just to clarify - this is observed only for the columns that include patches.

@manish-sethi

Thanks for reporting @manish-sethi! Will also have a look at this one.

fix[gpu]: retain device buffers for dyn dispatch kernel (#7980)

@joseph-isaacs

Thanks @joseph-isaacs and @0ax1 for the fixes.

I have two quick questions - if you can help answer

I see that the branch name is develop. Is it an indication that this is still under development? If that's the case, when is it expected to be released (or more stable)
Somewhere in the docs, I noticed that cuDF integration is in plan. Are there any dates/plans that I can look at?

@manish-sethi

@manish-sethi 👋 We use develop as our main branch from which we cut releases frequently, like today. You're in luck with the cuDF integration, I just started working on that today, and planned to finish it this week.

@0ax1

Thanks @0ax1 for your reply!

On the fix: I tested your earlier fix. It worked on GB10 GPU.
However, when I tried it on L4 GPU, I get a new panic.

thread 'main' panicked at /data/dev/vortex/vortex-array/src/buffer.rs:234:33:                                                                                                                                  
  unwrap_host called for Device allocation                                                                                                                                                                       
                                                                                                                                                                                                                 
    16: unwrap_host                                                                                                                                                                                              
          at vortex-array/src/buffer.rs:234:33                                                                                                                                                                   
    17: try_new<u8>                                                                                                                                                                                              
          at encodings/fastlanes/src/bitpacking/array/unpack_iter.rs:100:36
    18: unpacked_chunks<u8>                                                                                                                                                                                      
          at encodings/fastlanes/src/bitpacking/array/mod.rs:232:9
    19: unpacked_chunks<vortex_array::array::view::ArrayView<vortex_fastlanes::bitpacking::vtable::BitPacked>, u8>                                                                                               
          at encodings/fastlanes/src/bitpacking/array/mod.rs:316:9                                                                                                                                               
    20: unpack_into_primitive_builder<u8>                                                                                                                                                                        
          at encodings/fastlanes/src/bitpacking/array/bitpack_decompress.rs:67:37                                                                                                                                
    21: unpack_primitive_array<u8>                                                                                                                                                                               
          at encodings/fastlanes/src/bitpacking/array/bitpack_decompress.rs:40:5                                                                                                                                 
    22: unpack_array                                                                                                                                                                                             
          at encodings/fastlanes/src/bitpacking/array/bitpack_decompress.rs:31:9
    23: execute                                                                                                                                                                                                  
          at encodings/fastlanes/src/bitpacking/vtable/mod.rs:267:13
    24: execute_unchecked<vortex_fastlanes::bitpacking::vtable::BitPacked>                                                                                                                                       
          at vortex-array/src/array/mod.rs:474:9                                                                                                                                                                 
    25: execute_encoding_unchecked                                                                                                                                                                               
          at vortex-array/src/array/erased.rs:624:33                                                                                                                                                             
    26: execute_until<vortex_array::canonical::AnyCanonical>
          at vortex-array/src/executor.rs:231:40                                                                                                                                                                 
    27: execute                                                                                                                                                                                                  
          at vortex-array/src/canonical.rs:545:28
    28: execute<vortex_array::canonical::Canonical>                                                                                                                                                              
          at vortex-array/src/executor.rs:82:9                                                                                                                                                                   
    29: {async_block#0}
          at vortex-cuda/src/executor.rs:433:14                                                                                                                                                                  
    30: poll<…Future<Output=Result<Canonical, VortexError>>…>                                                                                                                                                    
          at core/future/future.rs:133:9
    31: {async_fn#0}                                                                                                                                                                                             
          at vortex-cuda/src/kernel/arrays/dict.rs:73:51 
    32: {async_block#0}                                                                                                                                                                                          
          at vortex-cuda/src/kernel/arrays/dict.rs:59:72 
    33: {async_block#0}                                                                                                                                                                                          
          at vortex-cuda/src/kernel/arrays/dict.rs:45:5                                                                                                                                                          
    34: poll<…>
          at core/future/future.rs:133:9                                                                                                                                                                         
    35: {async_fn#0}                                                                                                                                                                                             
          at vortex-cuda/src/hybrid_dispatch/mod.rs:147:18
    36: {async_block#0}                                                                                                                                                                                          
          at vortex-cuda/src/executor.rs:421:61          
    37: poll<…>                                                                                                                                                                                                  
          at core/future/future.rs:133:9
    38: {async_block#0}                                                                                                                                                                                          
          at vortex-playground/src/bin/vortex_column_bench.rs:159:80
    ── tokio runtime + main ──                                                                                                                                                                                   
    49: main
          at vortex-playground/src/bin/vortex_column_bench.rs:206:7

We don't support mixed execution between CPU/GPU yet in which we resume decompressing device buffers on the host. My assumption here is that for the L4 we blow past the shared memory capacity which is why it tries to fall back to CPU execution. I've made the error more explicit as part of: #8016.

@0ax1

@0ax1 - Thanks for looking into this. Just to update you on this, there was a problem at my end. It was not a shared memory issue. Mistakenly, ptx files for the wrong architecture carried over :(

Also, looking forward to your work on cuDF integration. Is there an issue that I can track for this?
Similarly, is integration with Velox also in the plan for the near future?

BTW, what is right communication channel to ask these kind of questions?

There's some high-level tracking for GPU work going on right now: #7712

Mistakenly, ptx files for the wrong architecture carried over

Thanks for reporting this! Yeah, we need to make the build setup more robust. I'll open a PR for this today.

BTW, what is right communication channel to ask these kind of questions?

GH issues are perfectly fine. The most direct way of communication is joining our public Slack channel: https://vortex.dev/slack. The whole team has an eye on that channel from our end.

#8047 PR to re-trig a PTX build once the GPU config changes.

@0ax1

Thanks - @0ax1.

GH issues are perfectly fine. The most direct way of communication is joining our public Slack channel: https://vortex.dev/slack. The whole team has an eye on that channel from our end.

Sure, for continuity, I will continue here on this topic for now. For context: I'm trying to wire Vortex into Velox's cuDF path so I can run TPC-H queries against .vortex files. Based on your answer, I'd decide my next course of action.

About the cuDF integration that you mentioned you are working on, could you please describe the scope? Is there an active repo or PR that I can track or test against when you are done with this work?
Similarly, is there any active work going on for integrating with Velox?

There's no ongoing or planned work on Velox integration as of now but it is something we talked about and would want to support in the future.

The cuDF support is almost there. We already support arrow device array export which bridges into cuDF. I'm currently working through any blockers NVIDIA/cudf#22620 and will after that add e2e support for pyVortex cuDF. In terms of timeline, picture this to be available within 2 weeks. All changes for cuDF support will land in the Vortex repo. For future ongoing conversation, I'd suggest joining the Slack channel.

I added a rough outline for the work to do on cuDF here: #8100

@0ax1

Thanks @0ax1 for the information and for pointing me to the related issue! Specifically, for my prototype, I can plan to use item#5 in #8100 for Velox integration.

Sure, I'll ask further questions on the Slack channel. Thanks again!

fix(cuda): copy encoded device-resident children to the host in into_…

Navigation Menu

Repository navigation

gpu-scan-cli scan --gpu-file panics with "as_slice must be called on host buffer" #7838

What happened?

Summary

Steps to reproduce

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

gpu-scan-cli scan --gpu-file panics with "as_slice must be called on host buffer" #7838

Description

What happened?

Summary

Steps to reproduce

Activity

joseph-isaacs commented on May 8, 2026

joseph-isaacs commented on May 8, 2026

manish-sethi commented on May 8, 2026

0ax1 commented on May 11, 2026

manish-sethi commented on May 14, 2026

0ax1 commented on May 18, 2026

manish-sethi commented on May 18, 2026

0ax1 commented on May 18, 2026

manish-sethi commented on May 19, 2026

0ax1 commented on May 19, 2026

manish-sethi commented on May 20, 2026

0ax1 commented on May 21, 2026

0ax1 commented on May 21, 2026

manish-sethi commented on May 21, 2026

0ax1 commented on May 26, 2026

0ax1 commented on May 26, 2026

manish-sethi commented on May 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions