Skip to content

Tracking Issue: cuDF support #8100

New issue
New issue
Closed
Closed
Tracking Issue: cuDF support#8100
Assignees
0ax1
Labels
tracking-issueShared implementation context for work likely to span multiple PRs.

Description

@0ax1
0ax1
opened on May 26, 2026

1. cuDF compatibility baseline

  • Track cuDF version containing rapidsai/cudf#22620.
  • Regression coverage for producer-owned ArrowArray.private_data.

2. Harden existing Arrow Device export

  • Audit device_type, device_id, sync_event, reserved, release callbacks, child ownership, and private_data.
  • Add tests for release idempotency, nested children, sliced arrays, and producer-owned private data.

3. Add null-mask support

  • Export Vortex validity bitmaps as CUDA buffers.
  • Populate Arrow null bitmap slots and null_count.
  • Support nulls for primitives, bools, decimals, temporals, strings, structs, and nested types.

4. Complete cuDF-compatible dtype coverage

  • Verify all primitive widths.
  • Finish decimal support.
  • Expand Utf8View / BinaryView tests:
    • inline/out-of-line values
    • multiple variadic buffers
    • non-ASCII UTF-8
    • sliced arrays
  • Add list/fixed-size-list export where cuDF supports the Arrow layout.
  • Return clear errors for unsupported dtypes.

5. Implement ArrowDeviceArrayStream

  • Add bindings for ArrowDeviceArrayStream.
  • Implement get_schema, get_next, get_last_error, and release.
  • Expose scan-to-device-stream from vortex-cuda.
  • Ensure stream batches stay on one CUDA device.

6. Expand cuDF e2e tests

  • Cover nullable primitives/strings.
  • Cover bools.
  • Cover decimals.
  • Cover temporals.
  • Cover structs/nested structs.
  • Cover lists/fixed-size lists.
  • Cover sliced arrays.

7. PyVortex cuDF integration

  • Expose Arrow Device export from PyVortex.

    • Add array.__arrow_c_device_array__(requested_schema=None, **kwargs).
    • Return Arrow schema, device-array.
    • Ensure PyCapsule destructors correctly release Arrow-owned resources.

  • Add user-facing cuDF APIs.

    • Add array.to_cudf().
    • Add file.to_cudf() / reader to_cudf().
    • Add scan/prepared-scan to_cudf().
    • Lazy-import cuDF when to_cudf() is called.
    • Do not silently fall back to host Arrow unless explicitly requested.

  • Validate Python → cuDF behavior.

    • Test real Vortex → Arrow Device → cuDF imports.
    • Cover nullable columns, strings/binary, nested columns, and scans.
    • Document required cuDF version, supported dtypes, limitations, and examples.

Activity

  1. 0ax1
    self-assigned this
    on May 26, 2026
  2. 0ax1
    added
    tracking-issueShared implementation context for work likely to span multiple PRs.
    on May 26, 2026

  3. manish-sethi commented on Jun 11, 2026

    @manish-sethi
    manish-sethi
    on Jun 11, 2026

    Can you please clarify the scope of the following two aspects

    • It seems for strings, cuDF works with utf8 format only and not yet with utf8 view format. Will the conversion be left to cuDFs import function or is there a plan to provide the decode kernels that directly produce utf8 format to avoid the conversion
    • It seems cuDF works mostly with RMM memory allocator; is it in the scope to take a memory allocator via integration APIs to avoid a copy of the final buffer while importing as cudf::column

  4. 0ax1 commented on Jun 17, 2026

    @0ax1
    0ax1
    on Jun 17, 2026
    ContributorAuthor

    @manish-sethi Support for Arrow Device array string view landed as part of: NVIDIA/cudf#22620.

    is it in the scope to take a memory allocator via integration APIs

    Definitely! But this is part of a bigger task. I'll post here when things are in flight and I have an idea on the time-line.

  5. manish-sethi commented on Jun 18, 2026

    @manish-sethi
    manish-sethi
    on Jun 18, 2026

    @manish-sethi Support for Arrow Device array string view landed as part of: rapidsai/cudf#22620.

    @0ax1 It seems this is a bug fix in cuDF. My question was mainly regarding the format of export utf8 v/s utf8 view

  6. 0ax1 commented on Jun 18, 2026

    @0ax1
    0ax1
    on Jun 18, 2026
    ContributorAuthor

    @manish-sethi Support for Arrow Device array string view landed as part of: rapidsai/cudf#22620.

    @0ax1 It seems this is a bug fix in cuDF. My question was mainly regarding the format of export utf8 v/s utf8 view

    cuDF has cudf::string_view support and the PR fixed the import from Arrow Device array STRING_VIEW into cuDF.
    But yeah there's no string_view column type and the string still gets materialized for a string column in cuDF So as of now that conversion is left up to cuDF. There's no direct path to decompress into the final format. I can look into supporting decoding varbin to utf8 directly on the gpu but ideally cuDF would add support for string view columns.

  7. manish-sethi commented on Jun 22, 2026

    @manish-sethi
    manish-sethi
    on Jun 22, 2026

    @manish-sethi Support for Arrow Device array string view landed as part of: rapidsai/cudf#22620.

    @0ax1 It seems this is a bug fix in cuDF. My question was mainly regarding the format of export utf8 v/s utf8 view

    cuDF has cudf::string_view support and the PR fixed the import from Arrow Device array STRING_VIEW into cuDF. But yeah there's no string_view column type and the string still gets materialized for a string column in cuDF So as of now that conversion is left up to cuDF. There's no direct path to decompress into the final format. I can look into supporting decoding varbin to utf8 directly on the gpu but ideally cuDF would add support for string view columns.

    Yes, my concern was mainly regarding cuDF causing materialization into STRING, as its operators do not yet directly work on STRING_VIEW. So, if it is easy to provide support for decoding varbin to utf8 directly, that would take this inefficiency out of this path - Thanks!

  8. 0ax1 commented on Jun 24, 2026

    @0ax1
    0ax1
    on Jun 24, 2026
    ContributorAuthor

    @manish-sethi Support for Arrow Device array string view landed as part of: rapidsai/cudf#22620.

    @0ax1 It seems this is a bug fix in cuDF. My question was mainly regarding the format of export utf8 v/s utf8 view

    cuDF has cudf::string_view support and the PR fixed the import from Arrow Device array STRING_VIEW into cuDF. But yeah there's no string_view column type and the string still gets materialized for a string column in cuDF So as of now that conversion is left up to cuDF. There's no direct path to decompress into the final format. I can look into supporting decoding varbin to utf8 directly on the gpu but ideally cuDF would add support for string view columns.

    Yes, my concern was mainly regarding cuDF causing materialization into STRING, as its operators do not yet directly work on STRING_VIEW. So, if it is easy to provide support for decoding varbin to utf8 directly, that would take this inefficiency out of this path - Thanks!

    Yeah this was good to point out! Expect the e2e perf become more polished over time.

  9. 0ax1
    closed this as completedon Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

tracking-issueShared implementation context for work likely to span multiple PRs.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions