Skip to content

Stop mixing up integer and floating point overflow #2898

Closed

Description

@njsmith

The integer arithmetic code in scalarmathmodule.c.src has overflow-detection logic in it. That's great. When it detects an overflow, the way it signals that is by calling npy_set_floatstatus_overflow(). That's not so great.

Two problems:

  • Integer rollover is really not the same thing as floating point overflow. (I'm much more worried about integer overflow for one thing, since it doesn't nicely go to inf, it just scrambles your data in some impossible to notice way.)
  • Setting and checking the float status bits is REALLY EXPENSIVE. Right now we set and check the float status bits on every scalar integer operation, and this makes those operations slower by a factor of two. So yeah, we shouldn't do that.

Proposed solution:

  • Add a new arithmetic error condition, let's call it "rollover", which is specifically for integer overflow, and make the current "overflow" just be used for floats. (So this is a new argument to np.seterr, etc.)
  • Test and signal this condition using a new thread-local storage key, rather than the floating point status bits. (This should be much cheaper, though you know, double check this.)

Python exports an undocumented C API for thread-local storage:

PyAPI_FUNC(int) PyThread_create_key(void);
PyAPI_FUNC(void) PyThread_delete_key(int);
PyAPI_FUNC(int) PyThread_set_key_value(int, void *);
PyAPI_FUNC(void *) PyThread_get_key_value(int);
PyAPI_FUNC(void) PyThread_delete_key_value(int key);

For our purposes, we'd do something like:

/* Initialize a thread-local storage spot for our flag during module setup */
static int integer_error_flag_key =  PyThread_create_key()
/* Clear the flag before running integer code */
/* NB this sets its value to NULL, and set_key_value is a no-op if this is not called first */
PyThread_delete_key_value(integer_error_flag_key);
/* Raise the flag when necessary */
PyThread_set_key_value(integer_error_flag_key, (void *) 1);
/* Check the flag */
if (PyThread_get_key_value(integer_error_flag_key) != NULL) {
    /* aieee */
}

Activity

  1. mattip commented on Sep 14, 2019

    @mattip
    Member

    Closing, since the code in question no longer has integer overflow detection at all. The idea to set a thread-local-storage flag can be revived when we revisit the overflow problem with a new dtype. Please reopen if I misunderstood.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions