Skip to content

array interface for scipy.sparse matrices? #4239

Open

Description

@shoyer

I noticed the other day that scipy.sparse matrices do not implement numpy's ndarray interface. This means that calling np.asarray(sparse_matrix) yields a 0d array consisting of a sparse matrix, which I found rather strange.

Is there a principled reason for this?

If not, I would suggest adding a __array__ method (essentially as an alias for .todense()).

Activity

  1. argriffing commented on Dec 7, 2014

    @argriffing
    Contributor

    Is there a principled reason for this?

    No there is not a principled reason, and it is an endless source of bugs. Trying to treat np.matrix, np.ndarray, scipy.sparse, or masked matrices through a common interface is a lost cause in my opinion, and the people with the talent and time to fix this are working on numpy replacements rather than working on numpy or scipy.

  2. shoyer commented on Dec 7, 2014

    @shoyer
    ContributorAuthor

    One implication of adding __array__ would be that numpy ufuncs (e.g., np.cos) would all work, returning ndarray objects, instead of raising. That sounds good, but that would break operations like np.sin, which currently works by returning another sparse array. So it looks like this won't be feasible until (at least) the arrival of __numpy_ufunc__.

  3. argriffing commented on Dec 7, 2014

    @argriffing
    Contributor

    numpy ufuncs (e.g., np.cos) would all work, returning ndarray objects, instead of raising. That sounds good, but that would break operations like np.sin

    Heh, I didn't know np.sin worked like this, but now I see that it does. Is this somehow hard-coded because sin(0) = 0? On one hand it's kind of cute, but on the other hand it's kind of horrifying.

    Edit: yes that's what's happening

    # Add the numpy unary ufuncs for which func(0) = 0 to _data_matrix.
    for npfunc in [np.sin, np.tan, np.arcsin, np.arctan, np.sinh, np.tanh,
                   np.arcsinh, np.arctanh, np.rint, np.sign, np.expm1, np.log1p,
                   np.deg2rad, np.rad2deg, np.floor, np.ceil, np.trunc, np.sqrt]:
        name = npfunc.__name__
    
        def _create_method(op):
            def method(self):
                result = op(self.data)
                x = self._with_data(result, copy=True)
                return x
    
            method.__doc__ = ("Element-wise %s.\n\n"
                              "See numpy.%s for more information." % (name, name))
            method.__name__ = name
    
            return method
    
        setattr(_data_matrix, name, _create_method(npfunc))

    Edit 2: also implemented separately for sparse matrices that are not 'data' matrices

            elif func in (np.sin, np.tan, np.arcsin, np.arctan, np.sinh, np.tanh,
                          np.arcsinh, np.arctanh, np.rint, np.sign, np.expm1, np.log1p,
                          np.deg2rad, np.rad2deg, np.floor, np.ceil, np.trunc, np.sqrt):
                func_name = func.__name__
                if hasattr(self, func_name):
                    result = getattr(self, func_name)()
                else:
                    result = getattr(self.tocsr(), func_name)()
  4. argriffing commented on Dec 7, 2014

    @argriffing
    Contributor

    calling np.asarray(sparse_matrix) yields a 0d array consisting of a sparse matrix, which I found rather strange.

    If "which I found rather strange" is euphemistic for "which caused me to curse my computer for ten minutes while tracking down the bug" then could this be counted as a vote for making a change in numpy to not create dtype=object numpy ndarrays without explicitly requesting dtype=object?

  5. shoyer commented on Dec 7, 2014

    @shoyer
    ContributorAuthor

    could this be counted as a vote for making a change in numpy to not create dtype=object numpy ndarrays without explicitly requesting dtype=object?

    This does seem like a good idea. I will add my +1 to the numpy issue.

  6. larsmans commented on Dec 10, 2014

    @larsmans
    Contributor

    that numpy ufuncs (e.g., np.cos) would all work

    ... but they'd be horribly expensive for anything but toy matrices. I see the flaws in scipy.sparse, but this is not one of them. Densifying a sparse matrix should, IMO, always be an explicit operation, because doing it on too big a matrix tends to send even the fastest machines thrashing.

    Is this somehow hard-coded because sin(0) = 0?

    Yes, only sparsity-preserving operations are implemented. I've never found an actual need for sin, but I have applied tanh, log1p and sqrt to sparse matrices.

  7. anntzer commented on Sep 17, 2015

    @anntzer
    Contributor

    As a side note, scipy.sparse.linalg.LinearOperator (which is completely separate class) could benefit likewise.

  8. larsmans commented on Sep 17, 2015

    @larsmans
    Contributor

    LinearOperator is intended for matrix-free methods.

  9. anntzer commented on Sep 17, 2015

    @anntzer
    Contributor

    But they can trivially be densified by matrix multiplication with the identity matrix (which was the implementation I was going to suggest for __array__).

  10. anntzer commented on Nov 2, 2015

    @anntzer
    Contributor

    Another confusing example:

    In [12]: np.diagonal(sparse.eye(3))
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-12-0c3821015a4f> in <module>()
    ----> 1 np.diagonal(sparse.eye(3))
    
    /usr/lib/python3.5/site-packages/numpy/core/fromnumeric.py in diagonal(a, offset, axis1, axis2)
       1309         return asarray(a).diagonal(offset, axis1, axis2)
       1310     else:
    -> 1311         return asanyarray(a).diagonal(offset, axis1, axis2)
       1312 
       1313 
    
    ValueError: diag requires an array of at least two dimensions
    

    Of course asarray decided to transform the sparse matrix into a 0d array.

  11. rgommers commented on Nov 2, 2015

    @rgommers
    Member

    @anntzer arguably that's an inconsistency (/ bug) in numpy. Here's a fix: https://github.com/rgommers/numpy/commits/diag-sparse

    I agree with @larsmans that densifying needs to be explicit.

  12. stefanv commented on Oct 9, 2021

    @stefanv
    Member

    Please see #14822; I implemented this there, but the change is controversial and had to be reverted.

  13. stefanv commented on Nov 17, 2021

    @stefanv
    Member

    It looks like there is some agreement that allowing np.asarray(sparse_matrix) to return a dense array is too risky. We also now have an array interface (see #14822), so the expectation may not be for np.asarray to densify.

    What is the best behavior to implement to close this? Error on np.asarray with an explanation that A.todense() should be used instead? Admittedly that's a bit annoying when you may be receiving a sparse or dense array.

  14. rgommers commented on Nov 17, 2021

    @rgommers
    Member

    +1 for erroring with a clear message, I'd say TypeError is appropriate here.

    Admittedly that's a bit annoying when you may be receiving a sparse or dense array.

    If you want to write code that densifies any incoming sparse array, you can spare the 2-3 lines of code to check for it and call .todense() on it I'd say.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions