Skip to content

How should VOTable handle bounded variable-length arrays that exceed their maximum size? #18167

Open

Description

@stvoutsin

Description

In the process of addressing an issue in the way astropy parses BINARY2 serialized data with bounded variable-length char arrays, I've encountered what seems to be a potential source of confusion in how cases where the data content exceeds the maximum bound are treated.

Specifically:

Astropy issues a warning (W46) when a string exceeds the bound, but it preserves the full string.
In the PR I've also kept this behaviour while handling the equivalent during output serialization.

However, this has the potential to be problematic and lead to compatibility issues for users.
The problematic aspect is that as it stands it may result in a VOTable that doesn't respect the schema contract.
This is likely to lead to unpredictable behaviour or breaking clients, for example in the case where a client allocates a buffer of specific size to read in the content based on the bound value.

Potential Approaches:

Warn only (current behaviour): Issue a warning but preserve the content as is, potentially violating the schema contract

if self.arraysize != "*" and len(value) > self.arraysize:
    vo_warn(W46, ("char", self.arraysize), config, pos)

Warn and truncate: Issue a warning and truncate the value to fit the maximum

if self.arraysize != "*" and len(value) > self.arraysize:
    vo_warn(W46, ("char", self.arraysize), config, pos)
    value = value[:self.arraysize]

Raise an exception: Treat exceeding bounds as a schema violation that requires fixing

if self.arraysize != "*" and len(value) > self.arraysize:
    vo_raise(E_NEW_ERROR, ("char", self.arraysize), config, pos)

Auto-adjust arraysize: Dynamically increase the field's arraysize when the content exceeds the max

if self.arraysize != "*" and len(value) > self.arraysize:
    vo_warn(W46, ("char", self.arraysize, len(value)), config, pos)
    field.arraysize = f"{max(len(value), self.arraysize)}*"
    self.arraysize = max(len(value), self.arraysize)

Let the verify param decide the behaviour:

verify="ignore": Silently preserve
verify="warn": Warn and preserve
verify="exception": Raise an exception

My initial thought is that whatever is used, it should be consistent between parsing and serializing to ensure round-trip preserve the data content.

Summarizing with a couple questions:

What should be the standard behaviour when a string exceeds its declared maximum length?
Should the behaviour be consistent between parsing and generating the output?
Should exceeding bounds be considered a serious schema violation (exception) or a minor issue (warning)?
Do we have any interoperability/compatibility concerns with the different approaches?
I'm not aware of the VOTable spec providing any explicit guidance on this case, but perhaps I've missed it?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    API changePRs and issues that change an existing API, possibly requiring a deprecation periodio.votableneeds-discussion

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions