Description
Description
In the process of addressing an issue in the way astropy parses BINARY2 serialized data with bounded variable-length char arrays, I've encountered what seems to be a potential source of confusion in how cases where the data content exceeds the maximum bound are treated.
Specifically:
Astropy issues a warning (W46) when a string exceeds the bound, but it preserves the full string.
In the PR I've also kept this behaviour while handling the equivalent during output serialization.
However, this has the potential to be problematic and lead to compatibility issues for users.
The problematic aspect is that as it stands it may result in a VOTable that doesn't respect the schema contract.
This is likely to lead to unpredictable behaviour or breaking clients, for example in the case where a client allocates a buffer of specific size to read in the content based on the bound value.
Potential Approaches:
Warn only (current behaviour): Issue a warning but preserve the content as is, potentially violating the schema contract
if self.arraysize != "*" and len(value) > self.arraysize:
vo_warn(W46, ("char", self.arraysize), config, pos)
Warn and truncate: Issue a warning and truncate the value to fit the maximum
if self.arraysize != "*" and len(value) > self.arraysize:
vo_warn(W46, ("char", self.arraysize), config, pos)
value = value[:self.arraysize]
Raise an exception: Treat exceeding bounds as a schema violation that requires fixing
if self.arraysize != "*" and len(value) > self.arraysize:
vo_raise(E_NEW_ERROR, ("char", self.arraysize), config, pos)
Auto-adjust arraysize: Dynamically increase the field's arraysize when the content exceeds the max
if self.arraysize != "*" and len(value) > self.arraysize:
vo_warn(W46, ("char", self.arraysize, len(value)), config, pos)
field.arraysize = f"{max(len(value), self.arraysize)}*"
self.arraysize = max(len(value), self.arraysize)
Let the verify param decide the behaviour:
verify="ignore": Silently preserve
verify="warn": Warn and preserve
verify="exception": Raise an exception
My initial thought is that whatever is used, it should be consistent between parsing and serializing to ensure round-trip preserve the data content.
Summarizing with a couple questions:
What should be the standard behaviour when a string exceeds its declared maximum length?
Should the behaviour be consistent between parsing and generating the output?
Should exceeding bounds be considered a serious schema violation (exception) or a minor issue (warning)?
Do we have any interoperability/compatibility concerns with the different approaches?
I'm not aware of the VOTable spec providing any explicit guidance on this case, but perhaps I've missed it?
Activity