Repository navigation
Description
Description
Using np.strings.find or np.char.find on a table MaskedColumn that has a string dtype raises an error. It used to work in astropy 6. It seems to be broken in astropy 7 and 8.
Expected behavior
If there is a column of strings, I expect to be able to find the string in the column. This is what worked in past versions of astropy, the latest being version 6.1.7:
In [2]: from astropy.table.column import MaskedColumn
...: import numpy as np
...:
...:
...: data = ["foo", "bar", "baz"]
...: col = MaskedColumn(data=data, mask=False, fill_value="N/A", dtype=np.dtype("<U8"))
...: np.char.find(col, "foo")
Out[2]: array([ 0, -1, -1])And this was with:
In [3]: astropy.__version__
Out[3]: '6.1.7'
In [4]: np.__version__
Out[4]: '1.26.4'This may seem like an obscure workflow, but when using astroquery to pull JWST data from MAST, the dataURI column in the returned astropy table is a MaskedColumn, and if one doesn't want to download every file that the API returns, one often wants to filter out some files, i.e. not download the *.jpg files or the *_rateints.fits files.
How to Reproduce
from astropy.table.column import MaskedColumn
import numpy as np
data = ["foo", "bar", "baz"]
col = MaskedColumn(data=data, mask=False, fill_value="N/A", dtype=np.dtype("<U8"))
np.strings.find(col, "foo")And the traceback is:
In [33]: np.strings.find(col, "foo")
Out[33]: ---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:510, in _check_fill_value(fill_value, ndtype)
509 try:
--> 510 fill_value = np.asarray(fill_value, dtype=ndtype)
511 except (OverflowError, ValueError) as e:
512 # Raise TypeError instead of OverflowError or ValueError.
513 # OverflowError is seldom used, and the real problem here is
514 # that the passed fill_value is not compatible with the ndtype.
ValueError: invalid literal for int() with base 10: np.str_('N/A')
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1338, in Column.__repr__(self)
1337 def __repr__(self):
-> 1338 return self._base_repr_(html=False)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1326, in Column._base_repr_(self, html)
1322 from astropy.utils.xml.writer import xml_escape
1324 descr = xml_escape(descr)
-> 1326 data_lines, outs = self._formatter._pformat_col(
1327 self, show_name=False, show_unit=False, show_length=False, html=html
1328 )
1330 out = descr + "\n".join(data_lines)
1332 return out
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:295, in TableFormatter._pformat_col(self, col, max_lines, show_name, show_unit, show_dtype, show_length, html, align)
283 col_strs_iter = self._pformat_col_iter(
284 col,
285 max_lines,
(...) 290 outs=outs,
291 )
293 # Replace tab and newline with text representations so they display nicely.
294 # Newline in particular is a problem in a multicolumn table.
--> 295 col_strs = [
296 val.replace("\t", "\\t").replace("\n", "\\n") for val in col_strs_iter
297 ]
298 if len(col_strs) > 0:
299 col_width = max(len(x) for x in col_strs)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:562, in TableFormatter._pformat_col_iter(self, col, max_lines, show_name, show_unit, outs, show_dtype, show_length)
560 else:
561 try:
--> 562 yield format_col_str(idx)
563 except ValueError:
564 raise ValueError(
565 f'Unable to parse format string "{col_format}" for '
566 f'entry "{col[idx]}" in column "{col.info.name}" '
(...) 569 "for possible format specifications."
570 )
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:522, in TableFormatter._pformat_col_iter.<locals>.format_col_str(idx)
520 def format_col_str(idx):
521 if not multidims:
--> 522 return format_func(col_format, col if is_scalar else col[idx])
524 # Prevents columns like Column(data=[[(1,)],[(2,)]], name='a')
525 # with shape (n,1,...,1) from being printed as if there was
526 # more than one element in a row
527 if multidims_all_ones:
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:89, in astropy.table._column_mixins._MaskedColumnGetitemShim.__getitem__()
87 cdef class _MaskedColumnGetitemShim(_ColumnGetitemShim):
88 def __getitem__(self, item):
---> 89 return base_getitem(self, item, masked_column_getitem)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:59, in astropy.table._column_mixins.base_getitem()
57 return self.data[item]
58
---> 59 value = getitem(self, item)
60
61 try:
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:83, in astropy.table._column_mixins.masked_column_getitem()
81
82 cdef inline object masked_column_getitem(object self, object item):
---> 83 value = MaskedArray.__getitem__(self, item)
84 return self._copy_attrs_slice(value)
85
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3288, in MaskedArray.__getitem__(self, indx)
3278 """
3279 x.__getitem__(y) <==> x[y]
3280
3281 Return the item described by i, as a masked array.
3282
3283 """
3284 # We could directly use ndarray.__getitem__ on self.
3285 # But then we would have to modify __array_finalize__ to prevent the
3286 # mask of being reshaped if it hasn't been set up properly yet
3287 # So it's easier to stick to the current version
-> 3288 dout = self.data[indx]
3289 _mask = self._mask
3291 def _is_scalar(m):
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1734, in MaskedColumn.data(self)
1731 @property
1732 def data(self):
1733 """The plain MaskedArray data held by this column."""
-> 1734 out = self.view(np.ma.MaskedArray)
1735 # By default, a MaskedArray view will set the _baseclass to be the
1736 # same as that of our own class, i.e., BaseColumn. Since we want
1737 # to return a plain MaskedArray, we reset the baseclass accordingly.
1738 out._baseclass = np.ndarray
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3264, in MaskedArray.view(self, dtype, type, fill_value)
3261 type = dtype
3262 dtype = None
-> 3264 output = super().view(*[a for a in (dtype, type) if a is not None])
3266 # Make sure to reset the _fill_value if needed
3267 if getattr(output, '_fill_value', None) is not None:
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3138, in MaskedArray.__array_finalize__(self, obj)
3136 # Finalize the fill_value
3137 if self._fill_value is not None:
-> 3138 self._fill_value = _check_fill_value(self._fill_value, self.dtype)
3139 elif self.dtype.names is not None:
3140 # Finalize the default fill_value for structured arrays
3141 self._fill_value = _check_fill_value(None, self.dtype)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:516, in _check_fill_value(fill_value, ndtype)
511 except (OverflowError, ValueError) as e:
512 # Raise TypeError instead of OverflowError or ValueError.
513 # OverflowError is seldom used, and the real problem here is
514 # that the passed fill_value is not compatible with the ndtype.
515 err_msg = "Cannot convert fill_value %s to dtype %s"
--> 516 raise TypeError(err_msg % (fill_value, ndtype)) from e
517 return np.array(fill_value)
TypeError: Cannot convert fill_value N/A to dtype int64It does work if I access the .data property of the MaskedColumn, which of course points to the underlying Numpy MaskedArray:
In [34]: np.strings.find(col.data, "foo")
Out[34]:
masked_array(data=[ 0, -1, -1],
mask=False,
fill_value=np.str_('N/A'))which implies that it has something to do with the wrapping of MaskedArray by MaskedColumn. The fact that it goes through the __repr__ is a bit suspicious?
Versions
import astropy
astropy.system_info()platform
--------
platform.platform() = 'macOS-26.6.1-arm64-arm-64bit'
platform.version() = 'Darwin Kernel Version 25.6.0: Sat Jul 11 15:25:26 PDT 2026; root:xnu-12377.161.13~4/RELEASE_ARM64_T6030'
platform.python_version() = '3.12.3'
packages
--------
astropy 8.0.1
numpy 2.5.1
scipy 1.18.0
matplotlib 3.11.1
pandas --
pyerfa 2.0.1.5
Activity
pllim commentedon Aug 21, 2026
jdavies-st commentedon Aug 21, 2026
Yes, same error, same traceback.
jdavies-st commentedon Aug 21, 2026
So after some investigation, it seems this landmine is actually buried in Numpy's
MaskedArray.__array_wrap__method, which propagates a stalefill_valuewith the wrong dtype across dtype-changing ufuncs.https://github.com/numpy/numpy/blob/1f90bbb55e65e7de68921bc3f1a2d94492fdf689/numpy/ma/core.py#L3143-L3201
We find it because
np.strings.findapplied to a string masked array, returns an integer array of matched shape/positions. But thefill_valueis no longer of the correct dtype. So it's sitting there as a landmine, waiting to explode. Internally in Numpy, it seems to never come up, unless you callself.view()on theMaskedArrayinstance, which then tries to validate the stale dtypefill_value.MaskedColumncalls.view()for every.__getitem__. The perils of fiddling with dunder methods of a package you don't control... 🫤I'll try to fix it on this end, and file an issue over at Numpy. I suspect it just means doing an extra check in
__array_wrap__to make sure thefill_valuedtype is valid for the output, and if not, resetting it toNone.jdavies-st commentedon Aug 24, 2026
This is probably related to #14558.jdavies-st commentedon Aug 24, 2026
This broke starting with numpy 2.0.0. So it's been broken a while.
pllim commentedon Aug 24, 2026
Thanks for looking into this!