Skip to content

BUG: Make bool(void_scalar) and void_scalar.astype(bool) consistent - #9856

Merged
charris merged 1 commit into
numpy:masterfrom
eric-wieser:fix-void-bool-2
Oct 14, 2017
Merged

charris merged 1 commit into
numpy:masterfrom
eric-wieser:fix-void-bool-2

Conversation

@eric-wieser

@eric-wieser eric-wieser commented Oct 12, 2017 •

Copy link
Copy Markdown
Member

Previously the behaviour of void.astype(bool) was:

  • Python 2:
    • itemsize == 0 - False
    • itemsize > 0 - True
  • Python 3:
    • itemsize == 0 - False, warn about casting empty arrays to bool
    • itemsize == 1 - True iff the sole element is 0
    • itemsize > 1- ValueError: setting an array element with a sequence.

Compared to the behaviour of bool(void), which is:

  • True iff all bytes are zero

The problem is that y = x.astype(bool) is implemented as y[i] = bool(x[i].item())

This patch changes it to use y[i] = bool(x[i]).

void.item() returns different results on python 2 and 3, which causes the crash.

IMO, item() is not something we should ever be using internally.

Works towards #9847


This is a less objectionable subset of #9848 with no compatibility ramifications

@charris

charris commented Oct 14, 2017 •

Copy link
Copy Markdown
Member

I get slightly different results with Python 2.7 and 1.14-dev

In [13]: void(3).astype(bool)
Out[13]: True

In [14]: void(2).astype(bool)
Out[14]: True

In [15]: void(1).astype(bool)
Out[15]: True

In [16]: void(0).astype(bool)
Out[16]: False

...

In [23]: void('123').astype(bool)
Out[23]: True

In [24]: void('12').astype(bool)
Out[24]: True

In [25]: void('1').astype(bool)
Out[25]: True

In [26]: void('').astype(bool)
Out[26]: False

So always True unless empty, no warnings.

EDIT: Same with void array scalars.

@charris

charris commented Oct 14, 2017

Copy link
Copy Markdown
Member

Also differs for Python 3.

Comment thread numpy/core/tests/test_multiarray.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to document that this only tests scalars and scalar arrays.

@charris

charris commented Oct 14, 2017

Copy link
Copy Markdown
Member

So the rule in all these cases will be that 0D arrays and scalars containing empty byte strings, empty unicode strings, and V scalars converted to boolean yield False, and all other cases yield True. I assume this extends consistently to higher dimensional arrays.

@eric-wieser

Copy link
Copy Markdown
Member Author

Turns out I mispoke about python 2, and hadn't tested void(0).

Also differs for Python 3.

Can you elaborate on this?

@eric-wieser

eric-wieser commented Oct 14, 2017 •

Copy link
Copy Markdown
Member Author

So the rule in all these cases will be that 0D arrays and scalars containing empty byte strings, empty unicode strings, and V scalars converted to boolean yield False,

Yes. This was already true when converting with bool(...). The rule for void is "all bytes are zero" → False.

There's nothing special about 0d here, other than the fact that the LHS of the test bool(x) == x.astype(bool) is meaningful

@eric-wieser

eric-wieser commented Oct 14, 2017 •

Copy link
Copy Markdown
Member Author

Actually, the above is what the previous patch did. This patch only affects the visible behaviour of void scalars, and leaves strings alone

@charris

charris commented Oct 14, 2017

Copy link
Copy Markdown
Member

This seems correct to me, but will need a release note.

@charris

charris commented Oct 14, 2017 •

Copy link
Copy Markdown
Member

Can you elaborate on this?

I get ValueError: setting an array element with a sequence., but may be doing something different.

This patch only affects the visible behaviour of void scalars

Yep, but there are tests waiting to be enabled for the others.

@eric-wieser

Copy link
Copy Markdown
Member Author

I get ValueError: setting an array element with a sequence.

You're right, I had a different execution path in my head. I've updated the top comment with correct information.

Caused by void scalars decaying to 1d uint8 arrays before casting - `getitem` is dangerous for intermediate results

Works towards numpy#9847
@eric-wieser

Copy link
Copy Markdown
Member Author

Release note added

@charris
charris merged commit d4afa3e into numpy:master Oct 14, 2017
@charris

charris commented Oct 14, 2017

Copy link
Copy Markdown
Member

Thanks Eric.

eendebakpt added a commit to eendebakpt/numpy that referenced this pull request Sep 8, 2026
The legacy `STRING/UNICODE/VOID_to_<T>` cast loops build a full array
scalar for every element and hand it to the destination's setitem.  For
S and U sources that scalar is needlessly expensive: `PyArray_Scalar` for
NPY_UNICODE allocates a scratch buffer, builds a `str`, wraps it in an
argument tuple and then runs `str.__new__` on the `np.str_` subtype,
which copies the data a second time.  The plain bytes/str returned by
getitem behaves identically in every setitem and is much cheaper.

Use getitem for S and U sources, keeping `PyArray_Scalar` for VOID
sources (where `void.item()` and `bool(void)` disagree, numpygh-9847, the
reason the scalar was introduced in numpygh-9856) and for VOID destinations
(`VOID_setitem` reads the source through the buffer protocol, which
`np.str_` supports but `str` does not).

Measured on 2e5 elements: `U -> int64` 3.1x faster, `U -> float64` 2.5x,
`U -> bool` 6.5x, `U -> S` 3.8x, `S -> int64` 1.4x, `S -> float64` 1.35x.
The reproducer in numpygh-11014 is resolved: `np.array(string_array, dtype=int)`
goes from 2.2x slower than `np.array(string_array.tolist(), dtype=int)` to
slightly faster.

Closes numpygh-11014

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants