ENH: timedelta64 mod and floordiv

timedelta64 mod and divmod do not support numeric other. floordiv behaves as expected:

td = pd.Timedelta(3456789)
td64 = td.to_timedelta64()

>>> divmod(td64, 2)
numpy.core._exceptions.UFuncTypeError: ufunc 'divmod' cannot use operands with types dtype('<m8[ns]') and dtype('int64')

>>> td64 % 2
numpy.core._exceptions.UFuncTypeError: ufunc 'remainder' cannot use operands with types dtype('<m8[ns]') and dtype('int64')

>>> td64 // 2
numpy.timedelta64(1728394,'ns')

>>> divmod(td, 2)
(Timedelta('0 days 00:00:00.001728394'), Timedelta('0 days 00:00:00.000000001'))

In pandas we implement mod and divmod in terms of floordiv (https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/timedeltas.pyx#L1440). I'd be happy to port this (and the accompanying tests) upstream if numpy is interested.

@jbrockmendel

@jbrockmendel sounds good! We already have np.floordiv(td64, td64) defined. I think that implementation probably pretty much works here as well? It just needs to use a signature of mq->mm and similarly for the remainder ufunc. The real work here is that in umath/type_resolution.c you will have to copy the output dtypes correctly. It should not be difficult though (but may be a bit of digging). Then add the loops in generate_umath_api.py...

The only thing to make sure is what happens with NaT values, I guess.

Great, I'll give this a try and ping if i run into trouble

I think I found the right spot, but can't build locally (well, failure at import-time)

>>> import numpy as np
Python(20678,0x10a88c5c0) malloc: can't allocate region
*** mach_vm_map(size=18446744072119918592) failed (error code=3)
Python(20678,0x10a88c5c0) malloc: *** set a breakpoint in malloc_error_break to debug
init_dgelsd failed init
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bmendel/Desktop/np/numpy/numpy/__init__.py", line 286, in <module>
    raise RuntimeError(msg)
RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
RankWarning: Polyfit may be poorly conditioned

The docs eventually led me here which says it is not maintained. Are there OSX build docs somewhere else?

Since you do not want a build to actually work with, maybe just use the environment variables: NPY_BLAS_ORDER= NPY_LAPACK_ORDER= to not link against any BLAS at all...

NPY_BLAS_ORDER= NPY_LAPACK_ORDER= to not link against any BLAS at all

Maybe that should be in the error message...

@mattip

@mattip sounds good, I am not a mac expert though, and that no BLAS seems like bad general advice to be honest.

NPY_BLAS_ORDER= NPY_LAPACK_ORDER= to not link against any BLAS at all...

Tried this with

export NPY_BLAS_ORDER=
export NPY_LAPACK_ORDER=
python3 setup.py build_ext --inplace

and got the same error at import-time. did you have something else in mind for how to set those environment variables?

I had to assign empty strings (if that's what bash calls them) when I ran into this:

export NPY_BLAS_ORDER=""
export NPY_LAPACK_ORDER=""

I had to assign empty strings (if that's what bash calls them) when I ran into this:

no dice

I had to assign empty strings

Sorry for misleading you. The empty strings aren't necessary--I ended up using them at one point when I was simplifying a different suggestion, and I guess I never simplified further and removed the quotes completely. I just tried again without the quotes, and it worked, meaning my numpy build ended up with no references to Accelerate.

@jbrockmendel

How can it still pick up cblas if you set those at compile time, we really could use better docs about mac-os setup maybe...? @jbrockmendel did you do a git clean -xdf and recompile everything with those env variables set?

did you do a git clean -xdf and recompile everything with those env variables set?

That fixed it, thanks.

that no BLAS seems like bad general advice to be honest.

I will try to expand the error message. My use case was a drive-by NumPy contributor who wants to fix an issue and tries to locally compile.:

If you have built locally, you should either install OpenBLAS or build NumPy
with a dummy, slow BLAS backend via
export NPY_BLAS_ORDER=""; 
export NPY_LAPACK_ORDER="";
rm -rf build; 
then rebuild numpy"

@seberg

First attempt was to go into core/src/umath/ufunc_typ_resolution.c and add in PyUFunc_DivmodTypeResolver after L2249:

        else if (PyTypeNum_ISINTEGER(type_num2)){
            // GH#16195 divmod(timedelta64, integer)
            out_dtypes[0] = PyArray_PromoteTypes(PyArray_DESCR(operands[0]),
                                                PyArray_DESCR(operands[1]));
            out_dtypes[1] = out_dtypes[0];
            Py_INCREF(out_dtypes[1]);
            out_dtypes[2] = PyArray_DescrFromType(NPY_LONGLONG);
            out_dtypes[3] = out_dtypes[0];
            Py_INCREF(out_dtypes[3]);
        }

Still getting the same exception. Any other suggestions for where to tinker @seberg?

The type resolution is necessary to get the datetimes right probably, but you first need to add the new types in: numpy/core/code_generators/generate_umath.py

but you first need to add the new types in: numpy/core/code_generators/generate_umath.py

OK, some best-guessing has led me to focus on

'divmod':
    Ufunc(2, 2, None,
          docstrings.get('numpy.core.umath.divmod'),
          'PyUFunc_DivmodTypeResolver',
          TD(intflt),
          [TypeDescription('m', FullTypeDescr, 'mm', 'qm')],
          # TD(O, f='PyNumber_Divmod'),  # gh-9730
          ),

Focusing on the [TypeDescription('m', FullTypeDescr, 'mm', 'qm')]. AFAICT the 3rd arg 'mm' is specifying that np.divmod supports divmod(td64, td64) and the last arg 'qm' is specifying that the output is a tuple with (numeric, td64) dtypes.

Based on this, I tried adding an entry to this list TypeDescription('m', FullTypeDescr, 'mq', 'mm').

The build log then contains an error message

numpy/core/include/numpy/__umath_generated.c:81:281: error: use of undeclared identifier 'TIMEDELTA_mq_mm_divmod'; did you mean 'TIMEDELTA_mm_qm_divmod'?

Sorry yes, the next step... That is actually correct, now you need to add the C symbol, if you grep for TIMEDELTA_mm_qm_divmod you can add it after that function (or probably templating it with our funny c.src templating language to do the same).

now you need to add the C symbol, if you grep for TIMEDELTA_mm_qm_divmod you can add it after that function (or probably templating it with our funny c.src templating language to do the same).

OK, grepping for TIMEDELTA_mm_qm_divmod i get results in __umath_generated.c, umath/loops.c, umath/loops.c.src, umath/loops.h, umath/loops.h.src.

TIMEDELTA_mq_mm_divmod is already present in __umath_generated.c, added a line in loops.h and loops.h.src, didnt add anything to loops.c or loops.c.src, since it looks like that is the whole function implementation and my impression from the thread so far is that is generated from... somewhere.

No more build-time error, but an ImportError with Symbol not found: _TIMEDELTA_mq_mm_divmod

Sorry, I was playing and should have been clearer (and we don't add ufuncs too often)...

the __umath_generated.c is what you edited when you edited the python. But that file needs to find a symbol, and that is defined in loops.h.src, they are not generated, but rather templated C-code. If you want to use the same code as for the qm case, you can simply add a #define ... there to define it to the same function. If you need special code, you will have to modify the actual c.src as well.

So likely all you need is add one:

#define TIMEDELTA_mm_mq_divmod TIMEDELTA_mm_qm_divmod

and you are done. (aside from checking the NaT edge cases maybe)

@seberg

@seberg im not getting anywhere with this locally; can i trade you tasks and implement something in pandas you're looking forward to?

I will try to remember to look into it, it shouldn't be hard, but NumPy can take a bit getting the hang off... May ping you for review or so.

@jbrockmendel

@jbrockmendel I started here: https://github.com/numpy/numpy/compare/master...seberg:divmod-timedelta?expand=1 mainly needs test, and I did not look into remainder yet, but if you are curious you can see what I did... It should really not be far from what you had, except maybe that I modified the template to address NaT results correctly.

thanks, ill take a look. if you want to build off of pandas tests, the relevant ones would be in pandas/tests/arithmetic/test_timedelta64.py and pandas/tests/scalar/timedelta/test_arithmetic.py

Navigation Menu

Repository navigation

ENH: timedelta64 mod and floordiv #16195

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ENH: timedelta64 mod and floordiv #16195

Description

Activity

seberg commented on May 11, 2020

jbrockmendel commented on May 11, 2020

jbrockmendel commented on May 11, 2020

seberg commented on May 11, 2020

mattip commented on May 11, 2020

seberg commented on May 11, 2020

jbrockmendel commented on May 11, 2020

WarrenWeckesser commented on May 11, 2020

jbrockmendel commented on May 11, 2020

WarrenWeckesser commented on May 11, 2020

seberg commented on May 11, 2020

jbrockmendel commented on May 11, 2020

mattip commented on May 12, 2020

jbrockmendel commented on May 14, 2020

seberg commented on May 14, 2020

jbrockmendel commented on May 16, 2020

seberg commented on May 16, 2020

jbrockmendel commented on May 16, 2020

seberg commented on May 16, 2020

jbrockmendel commented on May 21, 2020

seberg commented on May 22, 2020

seberg commented on May 25, 2020

jbrockmendel commented on May 25, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions