Repository navigation
Conversation
Contributor
|
Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.
|
neutrinoceros
removed their request for review
August 21, 2026 09:59
1 task done
Member
Author
|
Superseded by #20273 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A VOTable
FIELDdeclares itsdatatypeexplicitly, but reading one into aQTablethrew that away for any integer column that also carried a unit: the column was converted toQuantity, which casts to float64, silently changing every value above 2**53. Object identifiers are routinely that large, so the reported Euclidobject_idcolumn came back with wrong values and no warning, while the same file read into a plainTablewas exact.TableElement.to_table()now sets theQTablepreserve_quantity_dtypeattribute added in #20247, so the declareddatatypesurvives the conversion toQuantity. Alongcolumn withunit="ct"becomes an integerQuantityin counts rather than a float one, which is the behavior discussed in the issue.Fixes #17963.
Based on #20247 — the first commit here is that PR, so this one should be merged after it. Only the second commit is new.
Backwards compatibility
Reading a VOTable into a
QTablechanges the dtype of integer columns that have a unit, from float64 to the declared integer type. That is the bug being fixed, and it is the only value-level change: everything else, including plainTablereads, is untouched. Code that assumed a float column there will now see an integer one — an integerQuantitysupports the same operations, but true division and in-place float assignment behave as they do for any integer array.The other visible change is the extra
__attributes__entry in the meta of every table produced byto_table(), as described above.AI disclosure
This PR is AI-generated using Claude Opus 5. I have examined the code and fully understand the changes in
tree.pyand the tests.Details
Click to expand
Set the
preserve_quantity_dtypetable attribute when converting a VOTable to an astropy table1. Where the cast happens
io.votableitself was never lossy.TableElement.to_table()returns aTablewhose columns hold the exact integer values from the file. The cast happens afterwards, inQTable._convert_col_for_table(), which turns anyColumnwith a unit into aQuantity— andQuantity.__new__defaults todtype=np.inexact. Since the reader is registered forTable, the unified I/O layer produces theQTableitself, without = cls(out, copy=False)intable/connect.py, well after the reader has returned.That is why the fix cannot be a change to the values or dtypes the reader produces — they are already right. What the reader has to do is tell the eventual
QTableconversion to leave the dtype alone, and the only channel that survives from the reader to that conversion is the tablemeta.2. Implementation
Four lines in
TableElement.to_table(), inastropy/io/votable/tree.py:preserve_quantity_dtypeis theTableAttributeadded in #20247; like everyTableAttributeits value lives inmeta['__attributes__'], which is documented behavior ofTableAttributerather than an internal detail being poked at.This goes in
to_table()rather than inconnect.py::read_table_votable()so that it also covers code that goes through the votable API directly:That is the path
pyvoandastroqueryuse —DALResultsTable.to_qtable()in the issue report — so fixing only the unified-I/O entry point would have left the reporter's actual call broken.3. Testing
Two tests in
astropy/io/votable/tests/test_table.py, built on aVOTABLE_INT_WITH_UNITsample modeled on the file in the issue:object_idisdatatype="long" unit="NA"with aVALUES null,countsisdatatype="long" unit="ct",raisdatatype="double" unit="deg".test_qtable_preserves_int_datatype—object_idandcountsare int64 andrais float64; theobject_idvalues compare equal to the exact Python ints;countskeepsu.count; the attribute reads back asTrue; and a plainTableread of the same bytes is unchanged, since it was never affected.test_qtable_preserves_int_datatype_round_trip— writing theQTableback to VOTable and re-reading keeps int64 and the exact values.Two things worth knowing if you touch these tests: reading
unit="NA"emits nothing, but writing it back does emitW50: Invalid unit string 'NA', so only the write is wrapped inpytest.warns. Andunit="count"is not valid VOUnit — it parses toUnrecognizedUnit, so the sample usesct.Also checked by hand against the file from the issue:
long+unit="NA"MaskedQuantity, exact, unitNAlong+unit="ct"MaskedQuantityin countsdouble+unit="deg"parse().get_first_table().to_table()thenQTable(...)Table.readastropy/io/votable,table,io/ascii,io/misc,units,utils, plusdocs/tableanddocs/iodoctests: 10522 passed, 206 skipped, 29 xfailed.4. Trade-offs and what is not addressed
Every table from
to_table()now carriesmeta['__attributes__'] = {'preserve_quantity_dtype': True}, whether or not it has an integer column with a unit. This is visible int.metaand propagates to formats that preserve meta — an ECSV written from a VOTable-read table gains a__attributes__: {preserve_quantity_dtype: true}line. Writing back to VOTable drops it, sinceTableElement.from_table()reads only theID/name/ref/ucd/utype/descriptionkeys.The alternative is to set it only when a column actually is an integer with a unit, which keeps
metaclean in the common case. I went with the unconditional form because the guarantee comes from the format — VOTable always declaresdatatype— rather than from the contents of a particular file. Easy to change if reviewers prefer the narrower version.Not addressed here, from the issue discussion:
unit="NA"still parses to anUnrecognizedUnitand the column still becomes aQuantity. The suggestion of skipping theQuantityconversion entirely forUnrecognizedUnitcolumns is atablechange with wider consequences, and is independent of the dtype question — with this PR the values are correct either way.