You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I noticed the other day that scipy.sparse matrices do not implement numpy's ndarray interface. This means that calling np.asarray(sparse_matrix) yields a 0d array consisting of a sparse matrix, which I found rather strange.
Is there a principled reason for this?
If not, I would suggest adding a __array__ method (essentially as an alias for .todense()).
No there is not a principled reason, and it is an endless source of bugs. Trying to treat np.matrix, np.ndarray, scipy.sparse, or masked matrices through a common interface is a lost cause in my opinion, and the people with the talent and time to fix this are working on numpy replacements rather than working on numpy or scipy.
One implication of adding __array__ would be that numpy ufuncs (e.g., np.cos) would all work, returning ndarray objects, instead of raising. That sounds good, but that would break operations like np.sin, which currently works by returning another sparse array. So it looks like this won't be feasible until (at least) the arrival of __numpy_ufunc__.
numpy ufuncs (e.g., np.cos) would all work, returning ndarray objects, instead of raising. That sounds good, but that would break operations like np.sin
Heh, I didn't know np.sin worked like this, but now I see that it does. Is this somehow hard-coded because sin(0) = 0? On one hand it's kind of cute, but on the other hand it's kind of horrifying.
Edit: yes that's what's happening
# Add the numpy unary ufuncs for which func(0) = 0 to _data_matrix.fornpfuncin [np.sin, np.tan, np.arcsin, np.arctan, np.sinh, np.tanh,
np.arcsinh, np.arctanh, np.rint, np.sign, np.expm1, np.log1p,
np.deg2rad, np.rad2deg, np.floor, np.ceil, np.trunc, np.sqrt]:
name=npfunc.__name__def_create_method(op):
defmethod(self):
result=op(self.data)
x=self._with_data(result, copy=True)
returnxmethod.__doc__= ("Element-wise %s.\n\n""See numpy.%s for more information."% (name, name))
method.__name__=namereturnmethodsetattr(_data_matrix, name, _create_method(npfunc))
Edit 2: also implemented separately for sparse matrices that are not 'data' matrices
calling np.asarray(sparse_matrix) yields a 0d array consisting of a sparse matrix, which I found rather strange.
If "which I found rather strange" is euphemistic for "which caused me to curse my computer for ten minutes while tracking down the bug" then could this be counted as a vote for making a change in numpy to not create dtype=object numpy ndarrays without explicitly requesting dtype=object?
... but they'd be horribly expensive for anything but toy matrices. I see the flaws in scipy.sparse, but this is not one of them. Densifying a sparse matrix should, IMO, always be an explicit operation, because doing it on too big a matrix tends to send even the fastest machines thrashing.
Is this somehow hard-coded because sin(0) = 0?
Yes, only sparsity-preserving operations are implemented. I've never found an actual need for sin, but I have applied tanh, log1p and sqrt to sparse matrices.
But they can trivially be densified by matrix multiplication with the identity matrix (which was the implementation I was going to suggest for __array__).
It looks like there is some agreement that allowing np.asarray(sparse_matrix) to return a dense array is too risky. We also now have an array interface (see #14822), so the expectation may not be for np.asarray to densify.
What is the best behavior to implement to close this? Error on np.asarray with an explanation that A.todense() should be used instead? Admittedly that's a bit annoying when you may be receiving a sparse or dense array.
+1 for erroring with a clear message, I'd say TypeError is appropriate here.
Admittedly that's a bit annoying when you may be receiving a sparse or dense array.
If you want to write code that densifies any incoming sparse array, you can spare the 2-3 lines of code to check for it and call .todense() on it I'd say.
Activity
argriffing commentedon Dec 7, 2014
No there is not a principled reason, and it is an endless source of bugs. Trying to treat np.matrix, np.ndarray, scipy.sparse, or masked matrices through a common interface is a lost cause in my opinion, and the people with the talent and time to fix this are working on numpy replacements rather than working on numpy or scipy.
shoyer commentedon Dec 7, 2014
One implication of adding
__array__would be that numpy ufuncs (e.g.,np.cos) would all work, returning ndarray objects, instead of raising. That sounds good, but that would break operations likenp.sin, which currently works by returning another sparse array. So it looks like this won't be feasible until (at least) the arrival of__numpy_ufunc__.argriffing commentedon Dec 7, 2014
Heh, I didn't know
np.sinworked like this, but now I see that it does. Is this somehow hard-coded becausesin(0) = 0? On one hand it's kind of cute, but on the other hand it's kind of horrifying.Edit: yes that's what's happening
Edit 2: also implemented separately for sparse matrices that are not 'data' matrices
argriffing commentedon Dec 7, 2014
If "which I found rather strange" is euphemistic for "which caused me to curse my computer for ten minutes while tracking down the bug" then could this be counted as a vote for making a change in numpy to not create
dtype=objectnumpy ndarrays without explicitly requestingdtype=object?shoyer commentedon Dec 7, 2014
This does seem like a good idea. I will add my +1 to the numpy issue.
larsmans commentedon Dec 10, 2014
... but they'd be horribly expensive for anything but toy matrices. I see the flaws in
scipy.sparse, but this is not one of them. Densifying a sparse matrix should, IMO, always be an explicit operation, because doing it on too big a matrix tends to send even the fastest machines thrashing.Yes, only sparsity-preserving operations are implemented. I've never found an actual need for
sin, but I have appliedtanh,log1pandsqrtto sparse matrices.anntzer commentedon Sep 17, 2015
As a side note,
scipy.sparse.linalg.LinearOperator(which is completely separate class) could benefit likewise.larsmans commentedon Sep 17, 2015
LinearOperatoris intended for matrix-free methods.anntzer commentedon Sep 17, 2015
But they can trivially be densified by matrix multiplication with the identity matrix (which was the implementation I was going to suggest for
__array__).anntzer commentedon Nov 2, 2015
Another confusing example:
Of course asarray decided to transform the sparse matrix into a 0d array.
rgommers commentedon Nov 2, 2015
@anntzer arguably that's an inconsistency (/ bug) in numpy. Here's a fix: https://github.com/rgommers/numpy/commits/diag-sparse
I agree with @larsmans that densifying needs to be explicit.
stefanv commentedon Oct 9, 2021
Please see #14822; I implemented this there, but the change is controversial and had to be reverted.
stefanv commentedon Nov 17, 2021
It looks like there is some agreement that allowing
np.asarray(sparse_matrix)to return a dense array is too risky. We also now have an array interface (see #14822), so the expectation may not be fornp.asarrayto densify.What is the best behavior to implement to close this? Error on
np.asarraywith an explanation thatA.todense()should be used instead? Admittedly that's a bit annoying when you may be receiving a sparse or dense array.rgommers commentedon Nov 17, 2021
+1 for erroring with a clear message, I'd say
TypeErroris appropriate here.If you want to write code that densifies any incoming sparse array, you can spare the 2-3 lines of code to check for it and call
.todense()on it I'd say.