Repository navigation
Extended Table unicode sandwich for varchar data access from votable #8716
Description
As reported by @tomdonaldson , currently, this is the behavior; i.e., variable-length columns in VO table return byte string data. And this behavior was not seen in Python 2 but only in Python 3:
>>> from astropy.table import Table
>>> filename = '.../astropy/io/votable/tests/data/gemini.xml'
>>> t = Table.read(filename)
>>> x = t['access_url'][0]
>>> x
b'http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/...'To obtain a plain text output (not byte string), one needs to do the following. However, converting byte string to unicode also takes up more memory:
>>> x.decode('utf-8')
'http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/...'So, we wouldn't want to automatically convert everything to unicode, but rather do it on-the-fly only to requested data, by extending the "unicode sandwich" implemented in #5700.
Many thanks to @astrofrog for the advice and suggestion at IVOA 2019. cc @bmatthieu3
Appendix
Here is the XML header of the data file in the example above:
<?xml version="1.0" encoding="UTF-8"?>
<VOTABLE xmlns="http://www.ivoa.net/xml/VOTable/v1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.2">
<RESOURCE type="results">
<TABLE>
<FIELD name="ID" datatype="char" ucd="meta.id;meta.main" arraysize="*" />
<FIELD name="access_url" datatype="char" ucd="meta.ref.url" arraysize="*" />
<FIELD name="service_def" datatype="char" ucd="meta.ref" arraysize="*" />
<FIELD name="error_message" datatype="char" ucd="meta.code.error" arraysize="*" />
<FIELD name="semantics" datatype="char" ucd="meta.code" arraysize="*" />
<FIELD name="description" datatype="char" ucd="meta.note" arraysize="*" />
<FIELD name="content_type" datatype="char" ucd="meta.code.mime" arraysize="*" />
<FIELD name="content_length" datatype="long" ucd="phys.size;meta.file" unit="byte" />
<FIELD name="product_type" datatype="char" utype="caom:Artifact.productType" arraysize="*" />
<FIELD name="file_uri" datatype="char" ID="fileURIRef" arraysize="*" />
Activity
astrofrog commentedon May 16, 2019
Just to add some background info, currently variable length string arrays are stored in Numpy object arrays. So I think there are two ways forward for this - the first is to apply the 'unicode sandwich' approach to any Numpy object array when a bytes value is in a cell. Another approach is to make a special Numpy array subclass that would be used in votable to indicate that the array is a variable length bytes array (it would still just be an object array, but it would just be useful for Table to know it's the kind of array where the unicode sandwich applies).