Repository navigation
Is the VOTable parser incorrectly handling variable-length char arrays by stripping the asterisk from the arraysize attribute? #18099
Description
Description
In the process of developing a VOTable binary2 serializer, we've encountered an issue where queries that return a VOTable result with fields that have a variable char datatype with a bounded arraysize seem to fail, in ether a silent way empty table or with an exception.
As an example if we try to run a query that returns just the access_format of our ObsCore table which has the following field definition:
<FIELD name="access_format" datatype="char" arraysize="128*" ucd="meta.code.mime" utype="Access.format">
<DESCRIPTION>Content format of the dataset</DESCRIPTION>
</FIELD>
This results in an empty table. We've confirmed with other clients like TopCat and those do in fact display the table rows.
Upon some quick investigation into how astropy is parsing BINARY2 tables, what seems to be happening is that the binary2 converting process handles bounded variable-length char fields by stripping the asterisk in arraysize="128*" and treating it as a fixed-length field, causing it to attempt reading 128 bytes.
Specifically, the string fields end up using their Char Converter, where the * is stripped from the arraysize and then astropy switches the binary parsing method to _binparse_fixed.
This method reads fixed-length arrays by reading a fixed number of bytes (specified by self.arraysize) which seems incorrect for this field type.
I think what should be used in this case is the _binparse_var method, where we get the length from the 4-byte length prefix and read exactly as many bytes as needed.
This aligns with what we see in the IVOA specification and our understanding that 128* means "up-to 128 length".
The relevant sections are:
astropy/astropy/io/votable/converters.py
Line 299 in 9fce0d4
astropy/astropy/io/votable/converters.py
Line 326 in 9fce0d4
https://github.com/astropy/astropy/blob/9fce0d46c5e1807d7e1030c3cb0b1a9c0a359dd9/astropy/io/votable/converters.py#L376C9-L376C24
astropy/astropy/io/votable/converters.py
Line 376 in 9fce0d4
astropy/astropy/io/votable/converters.py
Line 372 in 9fce0d4
Perhaps someone could confirm if this interpretation of the issue is correct or if we've misunderstood how this should work?
If so I could perhaps put in a quick PR to address this.
Expected behavior
The expectation would be that the the parser reads the 4-byte length prefix (via _parse_length), then reading exactly that many bytes and decoding them.
How to Reproduce
- Parse a VOTable file where fields are variable-length char arrays (e.g. 128*)
from astropy.io.votable import parse as votable_parse
votable = votable_parse('result.xml')A sample VOTable that reproduces the issue:
<?xml version="1.0" encoding="UTF-8"?>
<VOTABLE xmlns="http://www.ivoa.net/xml/VOTable/v1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.4">
<RESOURCE type="results">
<INFO name="QUERY_STATUS" value="OK" />
<INFO name="QUERY" value="SELECT TOP 1 access_format FROM ivoa.ObsCore" />
<TABLE>
<FIELD name="access_format" datatype="char" arraysize="128*" ucd="meta.code.mime" utype="Access.format">
<DESCRIPTION>Content format of the dataset</DESCRIPTION>
</FIELD>
<DATA>
<BINARY2>
<STREAM encoding='base64'>
AAAAACphcHBsaWNhdGlvbi94LXZvdGFibGUreG1sO2NvbnRlbnQ9ZGF0YWxpbms=
</STREAM>
</BINARY2>
</DATA>
</TABLE>
</RESOURCE>
</VOTABLE>
Versions
platform
--------
platform.platform() = 'Linux-6.8.0-58-generic-x86_64-with-glibc2.35'
platform.version() = '#60~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 28 16:09:21 UTC 2'
platform.python_version() = '3.12.7'
packages
--------
astropy 7.2.dev28+g5c2a9f1b22
numpy 2.2.0
scipy --
matplotlib --
pandas --
pyerfa 2.0.1.5
Activity
pllim commentedon May 5, 2025
Are you using dev version of astropy? That line was introduced in #18075 by @ManonMarchand and approved by @bsipocz so hopefully they can clarify. Thanks!
ManonMarchand commentedon May 5, 2025
I added the same line in the Unicodechar converter (by mimicking the char converter). The one in the char converter was already there. It does fix (or at least not refuse) bounded variable size fields by assuming that they are of fixed size corresponding to the maximum size. It's sufficient for normal VOtables, but apparently not for compressed ones.
Thus, since it looks like the line breaks something while reading char fields in binary2, then any fix will need to be applied to unicodechar fields too.
stvoutsin commentedon May 5, 2025
I've just added a section with the package versions, I'm using astropy version:
7.2.dev28+g5c2a9f1b22Agree with @ManonMarchand if this is a confirmed bug it's been there for a while, but perhaps no-one has exercised the parser with binary2 serialized VOTables with bounded variable size chars arrays?
bsipocz commentedon May 5, 2025
Last time I looked at the code the test coverage is rather sparse for some of serialization, so this and other bugs may easily hide there like since forever. (#9433)
stvoutsin commentedon May 6, 2025
Assuming I'm correctly reading this as a bug, a potential fix is in a branch here: main...stvoutsin:astropy:u/stvoutsin/varcharfix
I can PR if it seems like a reasonable fix, after adding some tests for it
ManonMarchand commentedon May 6, 2025
Yes, please do! I tried your fix, it even solves this very old issue #8737
Changed bounded-length arraysize values to '*'
Changed bounded-length arraysize values to '*'
Changed bounded-length arraysize values to '*'