Skip to content

[C++] utf8_slice_codeunits returns wrong results with extreme start or step #55509

Open

Description

@Dricmoy

Describe the bug, including details regarding any error messages, version, and platform.

utf8_slice_codeunits returns wrong data or errors for valid SliceOptions with very large start or step. With pyarrow 26.0.1 on macOS arm64:

>>> import sys, pyarrow as pa, pyarrow.compute as pc
>>> arr = pa.array(["abc"])
>>> pc.utf8_slice_codeunits(arr, 1, sys.maxsize, sys.maxsize)[0]   # expected "b"
<pyarrow.StringScalar: '\x00'>
>>> pc.utf8_slice_codeunits(arr, sys.maxsize, 0, -1)[0]            # expected "cb"
<pyarrow.StringScalar: '\x00\x00'>
>>> pc.utf8_slice_codeunits(arr, -1, -4, -sys.maxsize - 1)[0]      # expected "c"
<pyarrow.StringScalar: '\x00'>
>>> pc.utf8_slice_codeunits(arr, -sys.maxsize - 1, -1, 1)          # expected ["ab"]
pyarrow.lib.ArrowInvalid: Negative buffer resize: -4

The cause is the output size estimate in SliceCodeunitsTransform::MaxCodeunits (scalar_string_utf8.cc), which is unchanged on main:

int64_t max_slice_codepoints = (stop - opt.start + opt.step - 1) / opt.step;

start isn't clamped, and adding step overflows when step is near INT64_MIN/INT64_MAX. In the first example, 3 - 1 + INT64_MAX - 1 wraps, the estimate comes out as 0, and the 1-byte result is written to a 0-byte buffer. A Debug + UBSan build of main reports signed integer overflow on that line for all four cases.

#36575 fixed the same overflow for the default stop, and binary_slice is being fixed in GH-51222 / #52451. The only documented constraint on SliceOptions is that step is non-zero, and sys.maxsize is the suggested way to slice to the end (#28940, #34917).

AI disclosure: I used an AI assistant to help find and reproduce this, and I checked the results myself.

Component(s)

C++

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions