Repository navigation
[Python] Implement a repr for Array and RecordBatch/Table for non-CPU data #41664
Description
Currently, if you have a pyarrow Array or RecordBatch/Table object that is backed by non-CPU data, just displaying the object (__repr__) crashes, because our PrettyPrint functionality assumes it deals with data on the CPU.
At a minimum, we should make the repr not crash, for example by first checking whether we have CPU data, and if not only printing generic information (the array type or the schema) and not a preview of the data.
But, I think we could also do better by actually ensuring the repr works and is informative for non-CPU data as well. For the pretty printing part of the repr, we only need a small subset of the data (by default first and last 5 elements), and copying such portion to the CPU just for printing should generally be fine.
If we implement this on the Python side, this depends on exposing the generic CopyTo functionality (#41126) to copy to CPU device. However, we could maybe also implement this on the C++ side in PrettyPrint itself? (taking a quick look at the current implementation, I think that would require quite some refactoring, though)
Activity
jorisvandenbossche commentedon May 15, 2024
Two additional thoughts:
to_string()/PrettyPrint) this is tackled by having an additionalcontainer_windowto determine how many elements to print within a nested array. But I can imagine that honoring that also for deciding which data to copy will get complicated (an initial implemention just copying a slice of the actual array sounds good enough to me).Merge remote-tracking branch 'upstream/main' into apachegh-41664-pret…
Merge remote-tracking branch 'upstream/main' into apachegh-41664-pret…
Merge remote-tracking branch 'upstream/main' into apachegh-41664-pret…
Merge remote-tracking branch 'upstream/main' into apachegh-41664-pret…
GH-41664: [C++][Python] PrettyPrint non-cpu data by copying to defaul…
jorisvandenbossche commentedon Jun 26, 2024
Issue resolved by pull request 42010
#42010