Skip to content

Commit 5b805d7

Browse files
amol-jorisvandenbossche
authored andcommitted
ARROW-13832: [Doc] Improve compute documentation
Document a bit better the compute functions and add a section about grouped aggregations. Also list the available aggregation functions automatically. ![sshot2](https://user-images.githubusercontent.com/601423/144261429-90ca370c-8431-424b-9f6d-44a17846c055.png) Closes #11830 from amol-/ARROW-13832 Authored-by: Alessandro Molina <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
1 parent 3f6773a commit 5b805d7

3 files changed

Lines changed: 199 additions & 28 deletions

File tree

‎docs/source/conf.py‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
import sys
3737
import warnings
3838
from unittest import mock
39+
from docutils.parsers.rst import Directive, directives
3940

4041
import pyarrow
4142

@@ -468,3 +469,49 @@ def setup(app):
468469
# This will also rebuild appropriately when the value changes.
469470
app.add_config_value('cuda_enabled', cuda_enabled, 'env')
470471
app.add_config_value('flight_enabled', flight_enabled, 'env')
472+
app.add_directive('arrow-computefuncs', ComputeFunctionsTableDirective)
473+
474+
475+
class ComputeFunctionsTableDirective(Directive):
476+
"""Generate a table of Arrow compute functions.
477+
478+
.. arrow-computefuncs::
479+
:kind: hash_aggregate
480+
481+
The generated table will include function name,
482+
description and option class reference.
483+
484+
The functions listed in the table can be restricted
485+
with the :kind: option.
486+
"""
487+
has_content = True
488+
option_spec = {
489+
"kind": directives.unchanged
490+
}
491+
492+
def run(self):
493+
from docutils.statemachine import ViewList
494+
from docutils import nodes
495+
import pyarrow.compute as pc
496+
497+
result = ViewList()
498+
function_kind = self.options.get('kind', None)
499+
500+
result.append(".. csv-table::", "<computefuncs>")
501+
result.append(" :widths: 20, 60, 20", "<computefuncs>")
502+
result.append(" ", "<computefuncs>")
503+
for fname in pc.list_functions():
504+
func = pc.get_function(fname)
505+
option_class = ""
506+
if func._doc.options_class:
507+
option_class = f":class:`{func._doc.options_class}`"
508+
if not function_kind or func.kind == function_kind:
509+
result.append(
510+
f' "{fname}", "{func._doc.summary}", "{option_class}"',
511+
"<computefuncs>"
512+
)
513+
514+
node = nodes.section()
515+
node.document = self.state.document
516+
self.state.nested_parse(result, 0, node)
517+
return node.children

‎docs/source/python/api/compute.rst‎

Lines changed: 47 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -45,28 +45,6 @@ Aggregations
4545
tdigest
4646
variance
4747

48-
Grouped Aggregations
49-
--------------------
50-
51-
.. autosummary::
52-
:toctree: ../generated/
53-
54-
hash_all
55-
hash_any
56-
hash_approximate_median
57-
hash_count
58-
hash_count_distinct
59-
hash_distinct
60-
hash_max
61-
hash_mean
62-
hash_min
63-
hash_min_max
64-
hash_product
65-
hash_stddev
66-
hash_sum
67-
hash_tdigest
68-
hash_variance
69-
7048
Arithmetic Functions
7149
--------------------
7250

@@ -498,3 +476,50 @@ Structural Transforms
498476
make_struct
499477
replace_with_mask
500478
struct_field
479+
480+
Compute Options
481+
---------------
482+
483+
.. autosummary::
484+
:toctree: ../generated/
485+
486+
ArraySortOptions
487+
AssumeTimezoneOptions
488+
CastOptions
489+
CountOptions
490+
CountOptions
491+
DayOfWeekOptions
492+
DictionaryEncodeOptions
493+
ElementWiseAggregateOptions
494+
ExtractRegexOptions
495+
FilterOptions
496+
IndexOptions
497+
JoinOptions
498+
MakeStructOptions
499+
MatchSubstringOptions
500+
ModeOptions
501+
NullOptions
502+
PadOptions
503+
PartitionNthOptions
504+
QuantileOptions
505+
ReplaceSliceOptions
506+
ReplaceSubstringOptions
507+
RoundOptions
508+
RoundToMultipleOptions
509+
ScalarAggregateOptions
510+
ScalarAggregateOptions
511+
SelectKOptions
512+
SetLookupOptions
513+
SliceOptions
514+
SortOptions
515+
SplitOptions
516+
SplitPatternOptions
517+
StrftimeOptions
518+
StrptimeOptions
519+
StructFieldOptions
520+
TakeOptions
521+
TDigestOptions
522+
TDigestOptions
523+
TrimOptions
524+
VarianceOptions
525+
WeekOptions

‎docs/source/python/compute.rst‎

Lines changed: 105 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,17 +23,33 @@ Compute Functions
2323
=================
2424

2525
Arrow supports logical compute operations over inputs of possibly
26-
varying types. Many compute functions support both array (chunked or not)
27-
and scalar inputs, but some will mandate either. For example,
28-
``sort_indices`` requires its first and only input to be an array.
26+
varying types.
2927

30-
Below are a few simple examples:
28+
The standard compute operations are provided by the :mod:`pyarrow.compute`
29+
module and can be used directly::
3130

3231
>>> import pyarrow as pa
3332
>>> import pyarrow.compute as pc
3433
>>> a = pa.array([1, 1, 2, 3])
3534
>>> pc.sum(a)
3635
<pyarrow.Int64Scalar: 7>
36+
37+
The grouped aggregation functions raise an exception instead
38+
and need to be used through the :meth:`pyarrow.Table.group_by` capabilities.
39+
See :ref:`py-grouped-aggrs` for more details.
40+
41+
Standard Compute Functions
42+
==========================
43+
44+
Many compute functions support both array (chunked or not)
45+
and scalar inputs, but some will mandate either. For example,
46+
``sort_indices`` requires its first and only input to be an array.
47+
48+
Below are a few simple examples::
49+
50+
>>> import pyarrow as pa
51+
>>> import pyarrow.compute as pc
52+
>>> a = pa.array([1, 1, 2, 3])
3753
>>> b = pa.array([4, 1, 2, 8])
3854
>>> pc.equal(a, b)
3955
<pyarrow.lib.BooleanArray object at 0x7f686e4eef30>
@@ -48,7 +64,7 @@ Below are a few simple examples:
4864
<pyarrow.DoubleScalar: 72.54>
4965

5066
These functions can do more than just element-by-element operations.
51-
Here is an example of sorting a table:
67+
Here is an example of sorting a table::
5268

5369
>>> import pyarrow as pa
5470
>>> import pyarrow.compute as pc
@@ -62,8 +78,91 @@ Here is an example of sorting a table:
6278
0
6379
]
6480

65-
81+
For a complete list of the compute functions that PyArrow provides
82+
you can refer to :ref:`api.compute` reference.
6683

6784
.. seealso::
6885

6986
:ref:`Available compute functions (C++ documentation) <compute-function-list>`.
87+
88+
.. _py-grouped-aggrs:
89+
90+
Grouped Aggregations
91+
====================
92+
93+
PyArrow supports grouped aggregations over :class:`pyarrow.Table` through the
94+
:meth:`pyarrow.Table.group_by` method.
95+
The method will return a grouping declaration
96+
to which the hash aggregation functions can be applied::
97+
98+
>>> import pyarrow as pa
99+
>>> t = pa.table([
100+
... pa.array(["a", "a", "b", "b", "c"]),
101+
... pa.array([1, 2, 3, 4, 5]),
102+
... ], names=["keys", "values"])
103+
>>> t.group_by("keys").aggregate([("values", "sum")])
104+
pyarrow.Table
105+
values_sum: int64
106+
keys: string
107+
----
108+
values_sum: [[3,7,5]]
109+
keys: [["a","b","c"]]
110+
111+
The ``"sum"`` aggregation passed to the ``aggregate`` method in the previous
112+
example is the ``hash_sum`` compute function.
113+
114+
Multiple aggregations can be performed at the same time by providing them
115+
to the ``aggregate`` method::
116+
117+
>>> import pyarrow as pa
118+
>>> t = pa.table([
119+
... pa.array(["a", "a", "b", "b", "c"]),
120+
... pa.array([1, 2, 3, 4, 5]),
121+
... ], names=["keys", "values"])
122+
>>> t.group_by("keys").aggregate([
123+
... ("values", "sum"),
124+
... ("keys", "count")
125+
... ])
126+
pyarrow.Table
127+
values_sum: int64
128+
keys_count: int64
129+
keys: string
130+
----
131+
values_sum: [[3,7,5]]
132+
keys_count: [[2,2,1]]
133+
keys: [["a","b","c"]]
134+
135+
Aggregation options can also be provided for each aggregation function,
136+
for example we can use :class:`CountOptions` to change how we count
137+
null values::
138+
139+
>>> import pyarrow as pa
140+
>>> import pyarrow.compute as pc
141+
>>> table_with_nulls = pa.table([
142+
... pa.array(["a", "a", "a"]),
143+
... pa.array([1, None, None])
144+
... ], names=["keys", "values"])
145+
>>> table_with_nulls.group_by(["keys"]).aggregate([
146+
... ("values", "count", pc.CountOptions(mode="all"))
147+
... ])
148+
pyarrow.Table
149+
values_count: int64
150+
keys: string
151+
----
152+
values_count: [[3]]
153+
keys: [["a"]]
154+
>>> table_with_nulls.group_by(["keys"]).aggregate([
155+
... ("values", "count", pc.CountOptions(mode="only_valid"))
156+
... ])
157+
pyarrow.Table
158+
values_count: int64
159+
keys: string
160+
----
161+
values_count: [[1]]
162+
keys: [["a"]]
163+
164+
Following is a list of all supported grouped aggregation functions.
165+
You can use them with or without the ``"hash_"`` prefix.
166+
167+
.. arrow-computefuncs::
168+
:kind: hash_aggregate

0 commit comments

Comments
 (0)