All masked columns are incompatible with writing to votable

astropy: 3.2.1 numpy 1.16.4

I think the following snippet used to work fine although I cannot pinpoint the version that worked.

from astroquery.gaia import Gaia
r = Gaia.launch_job("select top 1000 * from gaiadr2.gaia_source;").get_results()
r.write('test.vot', format='votable', overwrite=True)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/projects/notebook/problem.py in <module>
      5 from astroquery.gaia import Gaia
      6 r = Gaia.launch_job("select top 1000 * from gaiadr2.gaia_source;").get_results()
----> 7 r.write('test.vot', format='votable', overwrite=True)
      8 print(r['epoch_photometry_url'])
      9

~/miniconda3/lib/python3.7/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
    112         instance = self._instance
    113         with serialize_method_as(instance, serialize_method):
--> 114             registry.write(instance, *args, **kwargs)

~/miniconda3/lib/python3.7/site-packages/astropy/io/registry.py in write(data, format, *args, **kwargs)
    564
    565     writer = get_writer(format, data.__class__)
--> 566     writer(data, *args, **kwargs)
    567
    568

~/miniconda3/lib/python3.7/site-packages/astropy/io/votable/connect.py in write_table_votable(input, output, table_id, overwrite, tabledata_format)
    155
    156     # Create a new VOTable file
--> 157     table_file = from_table(input, table_id=table_id)
    158
    159     # Write out file

~/miniconda3/lib/python3.7/site-packages/astropy/io/votable/table.py in from_table(table, table_id)
    332     votable : `~astropy.io.votable.tree.VOTableFile` instance
    333     """
--> 334     return tree.VOTableFile.from_table(table, table_id=table_id)
    335
    336

~/miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py in from_table(cls, table, table_id)
   3632         votable_file = cls()
   3633         resource = Resource()
-> 3634         votable = Table.from_table(votable_file, table)
   3635         if table_id is not None:
   3636             votable.ID = table_id

~/miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py in from_table(cls, votable, table)
   2891         for colname in table.colnames:
   2892             column = table[colname]
-> 2893             new_table.fields.append(Field.from_table_column(votable, column))
   2894
   2895         if table.mask is None:

~/miniconda3/lib/python3.7/site-packages/astropy/io/votable/tree.py in from_table_column(cls, votable, column)
   1552             kwargs['unit'] = column.info.unit
   1553         kwargs['name'] = column.info.name
-> 1554         result = converters.table_column_to_votable_datatype(column)
   1555         kwargs.update(result)
   1556

~/miniconda3/lib/python3.7/site-packages/astropy/io/votable/converters.py in table_column_to_votable_datatype(column)
   1422                 return {'datatype': 'unicodeChar', 'arraysize': '*'}
   1423         elif isinstance(column[0], np.ndarray):
-> 1424             dtype, shape = _all_matching_dtype(column)
   1425             if dtype is not False:
   1426                 result = numpy_to_votable_dtype(dtype, shape)

~/miniconda3/lib/python3.7/site-packages/astropy/io/votable/converters.py in _all_matching_dtype(column)
   1330     first_shape = ()
   1331     for x in column:
-> 1332         if not isinstance(x, np.ndarray) or len(x) == 0:
   1333             continue
   1334

TypeError: len() of unsized object

This is choking on 'epoch_photometry_url' column which is all masked with empty object.

In [6]: r['epoch_photometry_url']
Out[6]:
<MaskedColumn name='epoch_photometry_url' dtype='object' description='epoch photometry url' length=1000>
 --
 --
 --
 --
 --
 --
 --
 --
 --
 --
 --
 --
 --
 --

Reproducing the gist of the problem:

from astropy.table import Table, MaskedColumn

# All masked, empty string as object: this fails.
# same case as in the above snippet.
try:
    t = Table({'c1':MaskedColumn(
        data=np.array([''.encode()]*5).astype(object),
        mask=np.ones(5).astype(bool))})
    t.write('test1.vot', format='votable', overwrite=True)
except TypeError:
    print("TypeError raised.")

try:
    # All masked, non-empty string as object: this fails.
    t = Table({'c1':MaskedColumn(
        data=np.array(['abc'.encode()]*5).astype(object),
        mask=np.ones(5).astype(bool))})
    t.write('test1.vot', format='votable', overwrite=True)
except TypeError:
    print("TypeError raised.")


# Partially masked, empty string as object: this works.
t = Table({'c1':MaskedColumn(
    data=np.array([''.encode()]*5).astype(object),
    mask=np.array([0,0,0,1,1]).astype(bool))})
t.write('test1.vot', format='votable', overwrite=True)


# All unmasked, empty string as object: this works.
t = Table({'c1':MaskedColumn(
    data=np.array([''.encode()]*5).astype(object),
    mask=np.zeros(5).astype(bool))})
t.write('test1.vot', format='votable', overwrite=True)

@smoh

@smoh , what would be the expected output if this worked?

For it to write the table without raising the TypeError..?

What do you want to see for the masked values in the file?

I think it makes sense to see whatever the raw data actually is. If it's just an array of b'', it should be that.

Hmm. Should also check the VO table standards as set by IVOA. Perhaps they have a section about masking but I can't look into it right now.

@tboch

ping @tboch @tomdonaldson

hmm..I'm finding out that masking some rows seem to throw away the data completely? (I don't know all the ins and outs of MaskedTable)

In [41]: col1 = MaskedColumn(data=['abc']*5, mask=[0,1,1,0,0])

In [42]: col1
Out[42]:
<MaskedColumn dtype='str3' length=5>
abc
 --
 --
abc
abc

In [51]: type(col1.data[1])
Out[51]: numpy.ma.core.MaskedConstant

in fact, the above TypeError is from trying to len this MaskedConstant.
That masking a column (seemingly) irreversibly modifies the raw data is rather weird to me.. but now I can understand that the TypeError is due to trying to figure out what kind of data a given column as from its first element (column[0]), which already became MaskedConstant.

The quirky thing is because the first element is checked, this works

# NOTE: first element not masked
t=Table({'c1':MaskedColumn(
    data=np.array(['abc']*5),
    mask=np.array([0,1,0,1,1]).astype(bool))})
t.write('test1.vot', format='votable', overwrite=True)

and this fails

# NOTE: first element *is* masked
t=Table({'c1':MaskedColumn(
    data=np.array(['abc']*5),
    mask=np.array([1,0,0,1,1]).astype(bool))})
t.write('test1.vot', format='votable', overwrite=True)

And by irreversibly change the data, I mean:

t=Table({'c1':MaskedColumn(
    data=np.array(['abc']*5),
    mask=np.array([0,1,0,1,1]).astype(bool))})
t.write('test1.txt', format='ascii', overwrite=True)

In [62]: !cat test1.txt
c1
abc
""
abc
""
""

The raw data for masked rows is just gone once saved. I guess it has always been this way but I think of masking as annotation to data not modification of it so it's surprising to see them gone. This is not particular to VOTable but what is particular, it seems is e.g., outputing all-masked column to txt works:

t=Table({'c1':MaskedColumn(
    data=np.array(['abc']*5),
    mask=np.array([1,1,1,1,1]).astype(bool))})
t.write('test1.txt', format='ascii', overwrite=True)

In [68]: !cat test1.txt
c1
""
""
""
""
""

Here's what I found after a little investigation:

Given that masked values are changed to some null value when they are output to file, I think it makes sense for votable to do that as well, i.e., I do not expect writing to votable to output 'raw' values but I think raising TypeError is still undesirable.
The TypeError is only raised when dtype of column is object and the first element is masked. As long as one of these conditions is broken, things are fine.
This results in a situation where you can read a dumped votable containing a column of all b'' values as object but not rewrite it.

I'm out of ideas where to look.

Here's the script I am basing this on (Some of the examples I pasted above where the column dtype is string actually works).

from astropy.table import Table
from astropy.table import MaskedColumn
from astroquery.gaia import Gaia

def check_votable_TypeError(table, outname):
    try:
        table.write(outname, format='votable', overwrite=True)
        print(f"VOTable written without any problem.")
    except TypeError:
        print("TypeError was raised in writing votable.")

r = Gaia.launch_job("select top 5 * from gaiadr2.gaia_source").get_results()
print(type(r))
check_votable_TypeError(r, "gaiatop5.vot")


print("Dump response to file; this never parses response to astropy table.")
r = Gaia.launch_job("select top 5 * from gaiadr2.gaia_source",
        dump_to_file=True,
        output_file="gaiatop5dump.vot")
print("Read dumped file.")
t = Table.read("gaiatop5dump.vot", format='votable')

print("Rewrite file as is.")
check_votable_TypeError(t, "rewrite.vot")
# same TypeError raised.

print("Try after unmasking first element.")
t = Table.read("gaiatop5dump.vot", format='votable')
t['epoch_photometry_url'].mask[0] = False
check_votable_TypeError(t, "rewrite.vot")
# as long as first element is unmasked, it's fine.

print("Try after converting dtype.")
t = Table.read("gaiatop5dump.vot", format='votable')
t['epoch_photometry_url'] = t['epoch_photometry_url'].astype('U')
check_votable_TypeError(t, "rewrite.vot")
# as long as dtype is not object, it's fine.


# -------------------------------------------------
import numpy as np
print("-"*79)
print("string column, first element masked")
t = Table(
    {
        "c1": MaskedColumn(
            data=np.array(["abc"] * 5),
            mask=np.array([1, 0, 0, 0, 0]).astype(bool)
        )
    }
)
check_votable_TypeError(t, "dummy.vot")

print("object column, fisrt element masked")
t = Table(
    {
        "c1": MaskedColumn(
            data=np.array(["abc"] * 5).astype(object),
            mask=np.array([1, 0, 0, 0, 0]).astype(bool)
        )
    }
)
check_votable_TypeError(t, "dummy.vot")

print("object column, fisrt element *not* masked")
t = Table(
    {
        "c1": MaskedColumn(
            data=np.array(["abc"] * 5).astype(object),
            mask=np.array([0, 1, 1, 0, 0]).astype(bool)
        )
    }
)
check_votable_TypeError(t, "dummy.vot")

and complete output (stderr redirected just to suppress votable warnings)

$python gaiatop5.py 2>/dev/null
Created TAP+ (v1.0.1) - Connection:
	Host: gea.esac.esa.int
	Use HTTPS: False
	Port: 80
	SSL Port: 443
<class 'astropy.table.table.Table'>
TypeError was raised in writing votable.
Dump response to file; this never parses response to astropy table.
Read dumped file.
Rewrite file as is.
TypeError was raised in writing votable.
Try after unmasking first element.
VOTable written without any problem.
Try after converting dtype.
VOTable written without any problem.
-------------------------------------------------------------------------------
string column, first element masked
VOTable written without any problem.
object column, fisrt element masked
TypeError was raised in writing votable.
object column, fisrt element *not* masked
VOTable written without any problem.

The issue appears to be with the handling of null string values. In the example given in the original report (trimmed to only get a couple rows):

from astroquery.gaia import Gaia
r = Gaia.launch_job("select top 2 * from gaiadr2.gaia_source;").get_results()
r.write('test.vot', format='votable', overwrite=True)

the epoch_photometry_url column, which should contain strings, has values that are numpy.ma.core.MaskedConstant with dtype = float64 and shape = (). Those values trip up the writer logic which assumes that string values will have a length.

The table has those masked values due to a quirk in the votable reader, which is used under the hood by astroquery.gaia to read the results from the server. In this case, the data provider marked the epoch_photometry_url values as null by using the null mask capability of BINARY2 serialization. The reader then populates the astropy Table with masked values for any "null" value from the votable. This behavior is appropriate for numeric FIELDs, but arguably not for strings.

This problem affects only astropy tables created from VOTables that use BINARY2 serialization, have some empty string values, and choose to mark them as null. Prior to VOTable 1.3, there was no way to specify a null string, i.e., there was no way to distinguish an empty string from a null string. (By string, I mean a FIELD with datatype="char" and arraysize=[a number or *].) With VOTable 1.3, the BINARY2 serialization option was added specifically to provide a mask bit for each cell to indicate whether or not the cell is a null value. The other serializations still don't allow distinguishing between null and empty string.

I don't see any code reason why this would be a new problem. More likely the data provider changed how the results were being sent.

One option to address this would be to harden the writer so that it can deal with these masked values on output, writing an empty string. That's OK, but doesn't address the other uses of those masked values. I can't think of any cases where a client would need or prefer masked values over empty strings, and can think of many cases where it complicates things.

Since clients already have to code for the possibility of empty strings, I think it makes sense to keep consistent behavior across all serializations, representing these values as empty strings in the astropy tables.

I'll submit a PR that implements that suggestion.

Thanks for looking into it.

I don't see any code reason why this would be a new problem. More likely the data provider changed how the results were being sent.

It didn't occur to me because I registered in my mind that released data = frozen data but I think that must be it!

So the suggestion is to do away with MaskedColumn entirely for string columns (or for all columns?), am I understanding it right? I think that is very reasonable since after all if someone wants any benefit of MaskedColumn for some purpose, they can create one easily on-the-fly.

I was uncomfortable going as far as doing away with the MaskedColumn entirely. They are used pervasively regardless of what serialization format was used, and I haven't fully learned why yet. Instead of risking wider consequences for now, I'll just prevent those string MaskedColumns from having masked values.

Address issue astropy#8995 by ignoring BINARY2 mask bits for string v…

Merge pull request #9057 from tomdonaldson/masked-column-writing

Address issue astropy#8995 by ignoring BINARY2 mask bits for string v…

The original snippet now works, but some of the test cases later on do not. 1 of the 3 TypeErrors are still around.
My output for the test example for #8995 (comment) this one:

<class 'astropy.table.table.Table'>
WARNING: W50: ?:?:?: W50: Invalid unit string 'log(cm.s**-2)' [astropy.io.votable.tree]
VOTable written without any problem.
Dump response to file; this never parses response to astropy table.
/Users/bsipocz/munka/devel/astroquery/astroquery/utils/tap/taputils.py:294: UserWarning: By default, results in votable_gzip format are returned in compressed format therefore your file gaiatop5dump.vot will be renamed to gaiatop5dump.vot.gz
  warnings.warn('By default, results in ' + ", ".join(
Read dumped file.
Rewrite file as is.
WARNING: W50: ?:?:?: W50: Invalid unit string 'log(cm.s**-2)' [astropy.io.votable.tree]
VOTable written without any problem.
Try after unmasking first element.
WARNING: W50: ?:?:?: W50: Invalid unit string 'log(cm.s**-2)' [astropy.io.votable.tree]
VOTable written without any problem.
Try after converting dtype.
WARNING: W50: ?:?:?: W50: Invalid unit string 'log(cm.s**-2)' [astropy.io.votable.tree]
VOTable written without any problem.
-------------------------------------------------------------------------------
string column, first element masked
VOTable written without any problem.
object column, fisrt element masked
TypeError was raised in writing votable.
object column, fisrt element *not* masked
VOTable written without any problem.

Navigation Menu

Repository navigation

All masked columns are incompatible with writing to votable #8995

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

All masked columns are incompatible with writing to votable #8995

Description

Activity

pllim commented on Jul 15, 2019

smoh commented on Jul 15, 2019

pllim commented on Jul 15, 2019

smoh commented on Jul 15, 2019

pllim commented on Jul 15, 2019

bsipocz commented on Jul 15, 2019

smoh commented on Jul 15, 2019

smoh commented on Jul 16, 2019

tomdonaldson commented on Jul 25, 2019

smoh commented on Jul 26, 2019

tomdonaldson commented on Jul 27, 2019

bsipocz commented on Oct 22, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions