histogram doesn't always work with masked arrays

numpy.histogram doesn't always work with masked arrays where the masked data is np.nans.

import numpy as np
x=np.random.randn(100)
x[50]=np.nan
xm=np.ma.masked_array(x, mask=np.isnan(x))

result=np.histogram(xm,bins=10)
print(result[0])

result=np.histogram(xm,bins=range(-3,3))
print(result[0])

The last case doesn't work for me
edit: oops, i mixed the result for matplotlib hist and histogram. None of the examples above work with np.histogram

The problem seems to be that asarray on line 647 in master returns an unmasked array with nans and that a.min() and a.max() returns nan on line 662.

changing asarray to asanyarray works for me but I don't know if it has any other implications.

I also read that histogram might not be designed to work with masked arrays

I'm guessing you found this StackOverflow question, which suggests that asanyarray change, as well as giving a workaround?

No, I haven't seen that discussion. Thanks for that answer.

I just implicitely believed that all of matplotlib and numpy should work with masked arrays (just ignoring the masked data). I also found out about the problem when plt.hist(xm, bins=10) worked (without giving any exception at least) but plt(xm, bins=range(10)) didn't work and the error message was kind of strange.

xm.compressed() is a god solution in this case though.

What about writing in the documentation that the result might be wrong for masked arrays?

What about writing in the documentation that the result might be wrong for masked arrays?

That's true of a large number of functions not in the np.ma module. Perhaps the ma docs should emphasize this, if they do not already.

I'd like to note this problem also affects masked data which does not have NaN in a really unhelpful way - it uses all of the data from the array and not only the unmasked points giving unexpected results and gives no warning.

import numpy as np
n=100
mymask = np.zeros(n)
# mask data in the middle of the array
mymask[12:15] = 1
y = np.ma.array(range(n), mask=mymask) + 0.5
hist, bin_edges = np.histogram(y)
print(hist, bin_edges)

returns:
[10 10 10 10 10 10 10 10 10 10] [ 0. 10. 20. 30. 40. 50. 60. 70. 80. 90. 100.]

it produces no warning or error. I would expect the output to be
[10 7 10 10 10 10 10 10 10 10] [ 0. 10. 20. 30. 40. 50. 60. 70. 80. 90. 100.]

which is what is returned by

hist, bin_edges = np.histogram(y.compressed(), range=(0, 100))
print(hist, bin_edges)

For reference, a bit of digging reveals that this discussion goes back to at least 2010 with a discussion on the NumPy-Discussion mailing list. https://www.mail-archive.com/[email protected]/msg27657.html

On that thread a number of important questions were asked about the correct approach to implement this by Bruce Southey.

Typical questions that come to mind are:

Should a user be warned that the input is a masked array?

Should histogram count the number of masked values?

What is the expected output when normed=True?

What type of array should be the weights and bin arguments?

What is the dimensions of the weight and bin arguments since it only needs to have the number of bins?

If the input array is masked should the weight and bins arguments also be masked arrays when applicable? If so, what happens if the masks are in different locations between arrays?

Given the complexity of this, I'd therefore propose that masked arrays should ideally raise a ValueError here, and at least warn the user if it is called with a masked array

I think we might be open to changing the implementation to use asanyarray, with a set of tests to demonstrate the different error modes around matching sizes, mask mismatch, and what happens to Nan. Hopefully the change would give some thought to point (6).

Some suggested approaches to the questions above:

Most users of masked arrays would expect this to just work, so no warning is needed
by default the count will be the number of unmasked variables. A later extension would be to allow optionally for the count to be the total size of the array using an additional argument
I think the output with normed=True (now called density) should be consistent with the count, so see (2)
Weights can be an unmasked array or a masked one, Bins should be an int or sequence but should not be masked (error).
as now weights need to be same shape as a. bins - same shape as currently
if shape is masked and any points are masked the mask should match the input array. If unmasked, print a warning that applying the mask from the input data to the weights.

as for NaN. If masked, and the mask is respected (i.e. ignored) then should not be an issue. If an unmasked array has NaN in it, or a masked array has points which are NaN that are NOT masked, then this should remain an error as at present (although a more helpful error perhaps can be returned)

Hello, I just encountered this issue recently with a masked array passed to np.histogram (called from matplotlib.pyplot.hist) that produced a big artificial spike in the resulting histogram by lumping all the masked points into a given bin rather than ignoring them (printing the values show '--' so I'm not sure how they got assigned a specific number somewhere near the middle). It seems that there was a PR above that was supposed to close this issue. But it's still open so I was wondering whether this was fixed in the end, and if so, starting from which numpy version? Thank you!

Hello, I just encountered this issue recently with a masked array passed to np.histogram (called from matplotlib.pyplot.hist) that produced a big artificial spike in the resulting histogram by lumping all the masked points into a given bin rather than ignoring them (printing the values show '--' so I'm not sure how they got assigned a specific number somewhere near the middle). It seems that there was a PR above that was supposed to close this issue. But it's still open so I was wondering whether this was fixed in the end, and if so, starting from which numpy version? Thank you!

as this has been an outstanding question since 2010, I wouldn't hold your breath. No one has offered to review the PR above. If this affects you it's probably quicker to work around this issue by pre-processing the data to remove the NaN before passing to the histogram function. See above for approaches to this,

Another possibility not mentioned here is to avoid the problematic numpy masked arrays and use marray instead.

Navigation Menu

Repository navigation

histogram doesn't always work with masked arrays #10019

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

histogram doesn't always work with masked arrays #10019

Description

Activity

eric-wieser commented on Nov 14, 2017

fredrik-1 commented on Nov 14, 2017

eric-wieser commented on Nov 14, 2017

nhsavage commented on Dec 20, 2025

nhsavage commented on Dec 20, 2025

mattip commented on Dec 20, 2025

nhsavage commented on Dec 22, 2025

nhsavage commented on Dec 22, 2025

stephjuneau commented on Jul 22, 2026

nhsavage commented on Jul 23, 2026

mattip commented on Jul 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions