Repository navigation
VOTable export should default to char datatype instead of unicodeChar for string columns #18515
Description
Summary
Currently astropy's VOTable export defaults to unicodeChar datatype for string columns
even when the rows only contain ASCII characters.
I'd argue this is problematic in a few different ways:
- Inefficient storage: 2 bytes per character vs 1 for char
- Some older TAP services and client tools handle char better than unicodeChar
- UnicodeChars are trickier to handle when it comes to binary2 encoding/decoding, especially when it comes to arraysizes.
unicodeCharfor ASCII data violates the principle of least surprise
In recent discussions, it is becoming apparent that VOTable 1.6 will be deprecating unicodeChar in favor of UTF-8 char.
Current Behavior
from astropy.table import Table
t = Table()
t['name'] = ['Hello', 'Pyvo', '123'] # All ASCII
t.write('test.xml', format='votable')
# Results in: <FIELD name="name" datatype="unicodeChar" arraysize="*"/>Proposed Solution
Change the default behavior to:
Use char datatype for string columns by default
Only use unicodeChar when explicitly requested by the user
Ensure we support UTF-8 content in char fields (aligning with VOTable 1.6)
Backward Compatibility
Reading existing VOTable files with unicodeChar should continue to work
Alternative solution:
We could also as an alternative approach parse the rows of the Table and check whether they contain non-ASCII characters in which case we could opt for the unicodeChar type.
However this is likely to introduce a performance hit especially for larger tables.
Activity
mhvk commentedon Aug 13, 2025
Bit far from what I usually work on, but if there's a move to UTF-8 (which is logical), wouldn't it make most sense to just mirror that move and leave the current code as is?
pllim commentedon Aug 14, 2025
@bsipocz advised that we should wait to see what upstream does first before tackling this one.
bsipocz commentedon Oct 15, 2025
OK, so we had a discussion about this with @tomdonaldson , and are thinking that a draft PR might be useful to show if ivoa-std/VOTable#71 is a go ahead or not.
Mark has already prototyped/testing it in STIL/STILTS/TOPCAT and if we don't run into any problems with it for all of binary/binary2, maybe fits, then the upstream change could go ahead much faster.
@stvoutsin - would you have any capacity for opening such a draft PR?
stvoutsin commentedon Nov 15, 2025
Apologies I completely missed this message. Happy to look into drafting a PR for this.