Skip to content

Commit 38c0fd3

Browse files
apacheGH-43352: [Docs][Python] Add all tensor classes documentation
1 parent e3c549e commit 38c0fd3

2 files changed

Lines changed: 81 additions & 15 deletions

File tree

‎docs/source/python/data.rst‎

Lines changed: 74 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -537,13 +537,16 @@ schema without having to get any of the batches.::
537537

538538
It can also be sent between languages using the :ref:`C stream interface <c-stream-interface>`.
539539

540+
Sparse Tensor Classes
541+
=====================
542+
540543
SparseCOOTensor
541-
===============
544+
---------------
542545

543546
The ``SparseCOOTensor`` represents a sparse tensor in Coordinate (COO) format, where non-zero elements are stored as tuples of row and column indices.
544547

545548
Example Usage:
546-
--------------
549+
^^^^^^^^^^^^^^
547550

548551
.. code-block:: python
549552
@@ -561,12 +564,12 @@ Example Usage:
561564
562565
563566
SparseCSRMatrix
564-
================
567+
---------------
565568

566-
`SparseCSRMatrix` represents a sparse tensor in Coordinate (COO) format, where non-zero elements are stored as tuples of row and column indices.
569+
`SparseCSRMatrix` represents a sparse matrix in Compressed Sparse Row (CSR) format, where non-zero elements are stored in a compressed manner using arrays for data, indices, and indptr.
567570

568571
Example Usage:
569-
--------------
572+
^^^^^^^^^^^^^^
570573

571574
.. code-block:: python
572575
@@ -581,12 +584,12 @@ Example Usage:
581584
582585
583586
SparseCSCMatrix
584-
===============
587+
---------------
585588

586589
`SparseCSCMatrix` represents a sparse matrix in Compressed Sparse Column (CSC) format, where non-zero elements are stored in a compressed manner using arrays for data, indices, and indptr.
587590

588591
Example Usage:
589-
--------------
592+
^^^^^^^^^^^^^^
590593

591594
.. code-block:: python
592595
@@ -600,13 +603,14 @@ Example Usage:
600603
>>> print(sparse_matrix)
601604
<pyarrow.SparseCSCMatrix object at 0x7fabcde12345>
602605
606+
603607
SparseCSFTensor
604-
===============
608+
---------------
605609

606610
`SparseCSFTensor` represents a sparse tensor in Compressed Sparse Fiber (CSF) format, optimized for multi-dimensional sparse data storage.
607611

608612
Example Usage:
609-
--------------
613+
^^^^^^^^^^^^^^
610614

611615
.. code-block:: python
612616
@@ -620,4 +624,64 @@ Example Usage:
620624
621625
>>> sparse_tensor = pa.SparseCSFTensor.from_numpy(data, indices, shape)
622626
>>> print(sparse_tensor)
623-
<pyarrow.SparseCSFTensor object at 0x7fabcde54321>
627+
<pyarrow.SparseCSFTensor object at 0x7fabcde54321>
628+
629+
630+
Conversion of RecordBatch to Tensor
631+
-----------------------------------
632+
633+
Each array of the ``RecordBatch`` has its own contiguous memory that is not necessarily
634+
adjacent to other arrays. A different memory structure that is used in machine learning
635+
libraries is a two-dimensional array (also called a 2-dim tensor or a matrix) which takes
636+
only one contiguous block of memory.
637+
638+
For this reason, there is a function ``pyarrow.RecordBatch.to_tensor()`` available
639+
to efficiently convert tabular columnar data into a tensor.
640+
641+
Data types supported in this conversion are unsigned, signed integer, and float
642+
types. Currently, only column-major conversion is supported.
643+
644+
Example Usage:
645+
^^^^^^^^^^^^^^
646+
647+
.. code-block:: python
648+
649+
>>> import pyarrow as pa
650+
>>> arr1 = [1, 2, 3, 4, 5]
651+
>>> arr2 = [10, 20, 30, 40, 50]
652+
>>> batch = pa.RecordBatch.from_arrays(
653+
... [
654+
... pa.array(arr1, type=pa.uint16()),
655+
... pa.array(arr2, type=pa.int16()),
656+
... ], ["a", "b"]
657+
... )
658+
>>> batch.to_tensor()
659+
<pyarrow.Tensor>
660+
type: int32
661+
shape: (9, 2)
662+
strides: (4, 36)
663+
>>> batch.to_tensor().to_numpy()
664+
array([[ 1, 10],
665+
[ 2, 20],
666+
[ 3, 30],
667+
[ 4, 40],
668+
[ 5, 50]], dtype=int32)
669+
670+
With ``null_to_nan`` set to ``True`` one can also convert data with
671+
nulls. They will be converted to ``NaN``:
672+
673+
.. code-block:: python
674+
675+
>>> import pyarrow as pa
676+
>>> batch = pa.record_batch(
677+
... [
678+
... pa.array([1, 2, 3, 4, None], type=pa.int32()),
679+
... pa.array([10, 20, 30, 40, None], type=pa.float32()),
680+
... ], names=["a", "b"]
681+
... )
682+
>>> batch.to_tensor(null_to_nan=True).to_numpy()
683+
array([[ 1., 10.],
684+
[ 2., 20.],
685+
[ 3., 30.],
686+
[ 4., 40.],
687+
[nan, nan]])

‎python/benchmarks/parquet.py‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,17 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18-
from pandas.testing import rands
19-
import numpy as np
2018

19+
import numpy as np
2120
import pyarrow as pa
21+
2222
try:
2323
import pyarrow.parquet as pq
2424
except ImportError:
2525
pq = None
2626

27+
from pyarrow.tests.util import rands
28+
2729

2830
class ParquetWriteBinary(object):
2931

@@ -34,7 +36,7 @@ def setup(self):
3436
num_cols = 10
3537

3638
unique_values = np.array([rands(value_size) for
37-
i in range(nuniques)], dtype='O')
39+
_ in range(nuniques)], dtype='O')
3840
values = unique_values[np.random.randint(0, nuniques, size=length)]
3941
self.table = pa.table([pa.array(values) for i in range(num_cols)],
4042
names=['f{}'.format(i) for i in range(num_cols)])
@@ -58,7 +60,7 @@ def time_convert_pandas_and_write_binary_table(self):
5860

5961

6062
def generate_dict_strings(string_size, nunique, length, random_order=True):
61-
uniques = np.array([rands(string_size) for i in range(nunique)], dtype='O')
63+
uniques = np.array([rands(string_size) for _ in range(nunique)], dtype='O')
6264
if random_order:
6365
indices = np.random.randint(0, nunique, size=length).astype('i4')
6466
else:
@@ -71,7 +73,7 @@ def generate_dict_table(num_cols, string_size, nunique, length,
7173
data = generate_dict_strings(string_size, nunique, length,
7274
random_order=random_order)
7375
return pa.table([
74-
data for i in range(num_cols)
76+
data for _ in range(num_cols)
7577
], names=['f{}'.format(i) for i in range(num_cols)])
7678

7779

0 commit comments

Comments
 (0)