Repository navigation
Add new quantity configuration option to keep int dtype - #20273
Conversation
|
Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.
|
|
@mhvk - this is a first cut at the idea discussed at #20247 (comment). I think the key code changes look OK but I'd like your input on the overall approach before going much further. As before, the actual new code footprint is quite small and easy to understand. I haven't scrutinized the rest of the changes / tests so don't worry about that for now. |
mhvk
left a comment
There was a problem hiding this comment.
@taldcroft - I like it! On second thought, I also think the configuration item is the right place for it. I do think, though, that for developers downstream it will be useful to make _keep_integer_dtype() public.
Really, it is mostly about names... For the conf item, I can see the logic of your name, since it is the main effect, but Quantity does also convert object (and should, but does not, convert bool... oops...). How about conf.force_inexact_dtype=["always", "default", "never"]? Or maybe a bit differently, conf.dtype_constraint=["input", "default", "inexact"]?
Somewhat independently, I think _keep_integer_dtype is a bit misnamed, since it does not always do so. So, I would call it something like preserve_dtype_by_default.
|
@mhvk - excellent, glad you are on board! In the meantime Claude and I have made this into a nominally-complete PR. I will change the conf to |
|
I updated the description as well. |
mhvk
left a comment
There was a problem hiding this comment.
It's no big deal, so maybe more for future PRs, but I think assert_array_equal(a, b, [strict=True]) is nicer since if things go wrong, the test will tell you where they were wrong. With strict=True, you also get a dtype check.
My only bigger comment is that we need to refer to the name correctly in the changelog and whatsnew. (There is actually something slightly off with how we define conf in a level at which one presumably should never import...)
I still think it makes some sense to also expose the preserve_dtype_by_default context manager. That implies that one should be added to __all__, so that it becomes with u.preserve_dtype_by_default():
| if expected_kind == "i": | ||
| # Only an integer dtype can hold these values exactly. | ||
| mask = getattr(col, "mask", np.zeros(len(col), dtype=bool)) | ||
| vals = np.asarray(getattr(col, "unmasked", col).value) |
| assert u.Quantity(self.VALS, u.ct, dtype=float).dtype.kind == "f" | ||
| assert u.Quantity([1.5, 2.5], u.ct, dtype=int).dtype.kind == "i" | ||
| assert u.Quantity(self.VALS, u.ct, dtype=None).dtype == np.int64 | ||
| # ``np.inexact`` is an explicit request to upcast, distinct from |
There was a problem hiding this comment.
Nice catch for backward compatibility.
| def test_inherently_float_subclasses(self, cls, unit, value): | ||
| """Subclasses that declare ``dtype=np.inexact`` stay floating point. | ||
|
|
||
| An angle or a distance is inherently a floating point quantity, so these |
There was a problem hiding this comment.
Interesting to here indirectly depend on the unit. I wonder if that will still give pushback, but I do agree it is more logical to have a class have different behaviour than for Quantity behaviour to depend on the unit.
| conf = Conf() | ||
|
|
||
|
|
||
| class _DtypeNotGiven: |
There was a problem hiding this comment.
I'd tend to use np._NoValue for this. Slightly less explicit but a standard we can use that very much has the same purpose.
There was a problem hiding this comment.
Great idea I didn't know about that.
| through a write/read round trip for the formats that serialize ``Quantity`` columns, such | ||
| as ECSV, FITS, HDF5 and parquet. | ||
|
|
||
| To restore the previous behavior, set the new ``units.quantity_convert_int_to_float`` |
There was a problem hiding this comment.
The link is not quite right: units.quantity.conf.<name> (same elsewhere)
| Controlling the Conversion of Integers | ||
| -------------------------------------- | ||
|
|
||
| Whether integer input is upcast is set by the ``quantity_convert_int_to_float`` |
There was a problem hiding this comment.
This should become a link to sphinx docs.
|
@mhvk - thanks for the excellent review. I think I implemented all your suggestions with one commit each. The sphinx change ended up making a new subsection in the API docs. The links all use the fully-qualified |
mhvk
left a comment
There was a problem hiding this comment.
Just a small bit for the docs.
Bit larger: before we had agreed to "change the conf to force_inexact_dtype". This still seems a bit more symmetric, but perhaps what you have is more explicit.
Since this affects Quantity behaviour generally, let me also ping @nstarman (though I suspect he will feel this change is long overdue!).
| a count does not silently lose precision. This is controlled by the | ||
| :ref:`quantity_convert_int_to_float <quantity-convert-int-to-float>` | ||
| configuration item; setting it to ``always`` gives the conversion to ``float`` | ||
| of astropy 7.x and earlier. |
There was a problem hiding this comment.
This should be 8.0 or earlier, right? (may be somewhere else too).
|
|
||
| ``always`` | ||
| Integer input is upcast to float and those overrides are ignored. This gives | ||
| the behavior of astropy 7.x and earlier, in which an integer column with a |
|
I need to review this PR, but first reaction in reading this is wondering why we want this configurable and not just following numpy dtype rules for the value — ie always keeping int-dtype inputs as int-dtype. IMO astropy over-promotes ints to floats since numpy handles upcasting automatically in math operations. Wouldn't it be easier (both in code and for users to understand) and more performant to not have to check configurations on every Q init and just let numpy do its thing? |
I forgot having agreed to that. 😄 I guess my argument for keeping it
In the end of course I'll defer to whatever you and @nstarman decide. |
|
@taldcroft - I'm OK with keeping your choice, I agree it is more descriptive. @nstarman - To me, the present solution is a good compromise, as I think Note that in the present PR, the behaviour of converting to Anyway, with this PR one can now choose to always preserve integers, without it being forced on everyone (and one can choose to stick with how things were). |
That was more to enable float16 vs float64. I do like that this PR allows the
Nice! I guess a speed test should make me happy :)
Wait, why? Exact values should always be safer than inexact ones near branch cuts (assuming you use dtype-upcasting for division) since you can't machine-precision your way into the wrong branch. E.g. Not for this PR, and thus maybe not for this thread. Relatedly, I asked claude to speed-test matching numpy dtype-semantics vs our current setup. Unsurprisingly, it's faster by 0 to 2,100%, depending on the situation. |
|
@nstarman - good catch on asking for performance! The earlier version was introducing up to a 2.6x slowdown for common cases because Config items are really slow (I know that but forgot). As always Claude gets long-winded, but here is the performance now. The fix ended up changing the logic a fair bit, but really for the better because it is faster than main now! PR #20273 — performance checkDefault path only: no
Common user idiom
Control group — does not go through
Table integration
Reading the tableFloat and The control group confirms the measurement is picking up Why integer input pays and float input does notReading a An earlier revision of this PR read the item unconditionally at the top of Method
Environment: Python 3.13.2, numpy 2.2.3, macOS 26.5.2 arm64. The branch is behind |
| # Neither sentinel is a dtype numpy can use, so replace both with None, | ||
| # keeping the original to decide below whether integer input is upcast | ||
| # to float: np.inexact is an explicit request to upcast, np._NoValue | ||
| # means no dtype was given and it is up to the configuration. |
There was a problem hiding this comment.
I notice this comment is not great so I can fix it.
| dtype_arg is np.inexact | ||
| or ( | ||
| dtype_arg is np._NoValue | ||
| and conf.quantity_convert_int_to_float != "never" | ||
| ) | ||
| ): |
There was a problem hiding this comment.
I just noticed this repeated code. I can make this into a private helper function if you prefer.
mhvk
left a comment
There was a problem hiding this comment.
Yes, good catch indeed. I had also forgotten that what looks like a straight attribute access is in fact very slow. But calling it only when actually relevant solves that, great!
474e61d to
cb6de79
Compare
|
For the record, anything that touches dtype should probably run Extra CI in the future. Please see #20318 |
Summary
This PR allows controlling the conversion integers by Quantity via a new configuration
variable. With this in place, the behavior of QTable is updated and an issue reading
files with integer columns with units is fixed.
Controlling the conversion of integers by Quantity
Creating a
Quantityfrom integer input without an explicitdtypeupcasts the valuesto float, which silently changes values that
float64cannot hold. A newunits.quantity_convert_int_to_floatconfiguration item takesdefault,alwaysorneverto control this. The behavior ofQuantityon its own is unchanged, so this partis backward compatible.
Integer columns with a unit keep their dtype in a QTable
An integer
Columnwith a unit added to aQTableis converted to aQuantity, andthat conversion used to cast the values to float. It now keeps the integer
dtype, so a19-digit Gaia
source_idsurvives instead of coming back as a different number. This isan API change; setting the configuration item to
alwaysrestores the previous behavior.Reading files with integer columns that have units
Reading into a
QTablegoes through the same conversion, so an integer column with aunit read from VOTable, FITS, ECSV, HDF5 or parquet now keeps its
dtype, and a writefollowed by a read returns the values that were written. This is an API change but is
really a bug fix: the previous behavior corrupted values that the file itself stored
exactly, with nothing to indicate that anything had been lost.
Fixes #17963.
AI disclosure
This PR includes AI-generated content using Claude Opus 5. I have carefully reviewed the
content and fully understand the changes.
with reviewers.
Details
Click to expand
Make the integer-to-float conversion in
Quantityconfigurable, and haveQTableandtable deserialization opt out of it
Putting an integer column that has a unit into a
QTableconverts it to a floatQuantity. For a column of identifiers or counts this silently changes the values: aGaia
source_idis a 19-digit integer thatfloat64cannot represent, so the numberthat comes out is not the number that went in, and nothing warns about it. The columns
this hits are exactly the ones that arrive with a unit attached from VOTable and FITS
files, which is how the issue was found.
Quantity.__new__defaults todtype=np.inexact, which upcasts integer input to float.That default is right for a
Quantitycreated on its own — most physical quantities arecontinuous — but wrong for a table column, where the dtype was chosen by whoever wrote
the data. This PR adds a
units.quantity_convert_int_to_floatconfiguration item withvalues
default,alwaysandnever, and hasQTableand the table deserializationcode request
neverwhile constructing aQuantityfrom a column. Setting the item toalwaysrestores the previous behavior everywhere, including on files that alreadycontain integers.
Only integer input is affected.
np.inexactupcasts only what is not already inexact, sofloat32,float64and complex columns keep their dtype whether or not this PR isapplied, and
Quantity([1, 2], u.m)on its own still givesfloat64as before.1. Why a configuration item rather than a keyword argument
An earlier attempt (#20247) added a
QTable.preserve_quantity_dtypetable attribute.That works for a
QTablebuilt in memory, but the attribute has to be threaded throughevery entry point that builds one, and
QTable.read(..., preserve_quantity_dtype=True)is not possible because the unified I/O layer passes unknown keywords to the reader
rather than to the table class. It also did nothing for a column reconstructed from a
serialized
Quantity, so an ECSV file written from such a table read back as float.A configuration item is read wherever the conversion actually happens, so the same
decision applies to a table built in memory, a table read from a file, and a column
rebuilt from
__serialized_columns__, with no new keyword arguments.2.
dtype=np.inexactwas doing two jobsQuantity.__new__usednp.inexactboth as the signature default and as an explicitvalue a caller can pass, and the two were indistinguishable. Once the configuration item
exists that matters: an explicit
dtype=np.inexactis a request, documented and testedby
test_float_dtype_promotionas "the minimum precision is float64", and aconfiguration item should not override it.
The default is now
np._NotGiven, so the three cases are distinct:dtypenp.inexactNoneor a real dtypeAngle,DistanceandFunctionQuantitydeclaredtype=np.inexactin their ownsignatures. Under the sentinel these read as a deliberate statement that the class is
inherently floating point — an integer angle is not meaningful — so they keep the
declaration and are unaffected by the configuration. Their signatures are unchanged by
this PR.
3. Implementation
astropy/units/quantity.py:Conf.quantity_convert_int_to_float, aConfigItemwhose default value is the list["default", "always", "never"], so the first is the default and the others are theonly permitted alternatives.
_keep_integer_dtype(), a context manager resolvingdefaulttoneverand passingalwaysandneverthrough unchanged. This is what makesalwaysa real escapehatch rather than a synonym for
default.__new__, the branch that used to befloat_default = dtype is np.inexactis nowthree-way. Note that
dtype = Nonehas to happen for both the sentinel andnp.inexactregardless of the configuration: leavingnp.inexactin place would passit to
np.array(), which resolves it tofloat64and makesneversilently donothing.
astropy/table/table.py—QTable._convert_col_for_tablewraps theQuantityconstruction in
_keep_integer_dtype().astropy/table/serialize.py—_construct_mixins_from_columnswraps the reconstructionloop in
_keep_integer_dtype(). This is the one place shared by every format that uses__serialized_columns__, so it covers ECSV, FITS, HDF5 and parquet together.Reconstructing a serialized column should reproduce what was written rather than
re-deciding a dtype, which is arguably a fix in its own right.
Nothing else changed.
Tableis untouched, and aColumnwith a unit in a plainTableis still a
Column.4. Effect on I/O, and on #20248
With only the
QuantityandQTablechanges, writing was fixed but reading was not: anECSV file correctly received
datatype: int64and was then cast straight back to floaton read. Every format that serializes
Quantitycolumns failed identically, which iswhat pointed at the single call site in
serialize.py:Quantity+QTableonlyserialize.pychangeVOTable was already correct because it stores plain columns with units, so it goes
through
QTable._convert_col_for_table. That also means #20248, which added VOTablespecific handling for this, is not needed: reading the VOTable from #17963 on this branch
gives
int64with the identifier exact, with no changes underastropy/io/votable/.5. Backwards compatibility
Quantityitself is unchanged for existing code: with the shipped configuration,Quantity(...)from integer input still givesfloat64, and explicitdtypeargumentsbehave as before. The signature default is now
np._NoValuerather thannp.inexact, so the rendered signature readsdtype=<dtype not given>; code passingdtype=np.inexactexplicitly still gets the upcast, and now keeps it under anyconfiguration.
QTablechanges behavior. An integer column with a unit becomes an integerQuantitywhere it used to become a float one, whether the table is built in memory or read from a
file. Code that assumed such a column was float may need updating, and integer division
and overflow semantics now apply where floating point ones did. Setting
units.quantity_convert_int_to_floattoalwaysrestores the previous behavioreverywhere, including when reading a file that contains integers.
Changelog fragments are
docs/changes/units/20273.feature.rstanddocs/changes/table/20273.api.rst. Narrative documentation is updated indocs/units/quantity.rst, where the section ondtypeno longer describesnp.inexactas the default, and in
docs/table/mixin_columns.rst, where anattentionblock advisedagainst giving integer columns a unit because they would be converted to float. There are
three
docs/whatsnew/8.1.rstentries: the new configuration item, theQTablechange,and the effect on reading files.