Skip to content

BUG: QTable converts int columns to floats #17963

New issue
New issue
Closed
#20273
Closed
BUG: QTable converts int columns to floats#17963
#20273
Labels
Bugtable

Description

@bsipocz
bsipocz
opened on Mar 31, 2025

Consider this votable, with a long datatype for an object ID column.

Table.read() reads it in properly and object_id is an int64 column, on the other hand QTable converts it into a float which is absolutely incorrect.

>>> from astropy.table import Table, QTable
>>> from io import BytesIO
>>> vot_string = """<?xml version='1.0'?>
<VOTABLE version="1.2"
 xmlns="http://www.ivoa.net/xml/VOTable/v1.2">
<!--
 !  VOTable written by STIL version 4.1-4 (edu.caltech.ipac.table.io.VoTableWriter$VOTableWriterImpl)
 !  at 2025-03-31T23:36:57
 !-->
<RESOURCE>
<TABLE nrows="2">
<FIELD ID="col_0" datatype="long" name="object_id" unit="NA">
<DESCRIPTION>Euclid unique source identifier</DESCRIPTION>
<VALUES null='-9223372036854775808'/>
</FIELD>
<FIELD ID="col_1" datatype="double" name="ra" precision="10" ucd="pos.eq.ra;meta.main" unit="deg">
<DESCRIPTION>Source barycenter RA coordinate (SExtractor ALPHA_J2000) decimal degrees</DESCRIPTION>
</FIELD>
<FIELD ID="col_2" datatype="double" name="dec" precision="10" ucd="pos.eq.dec;meta.main" unit="deg">
<DESCRIPTION>Source barycenter DEC coordinate (SExtractor DELTA_J2000) decimal degrees</DESCRIPTION>
</FIELD>
<FIELD ID="col_3" datatype="long" name="tileid" unit="NA">
<DESCRIPTION>Euclid Tile ID</DESCRIPTION>
<VALUES null='-9223372036854775808'/>
</FIELD>
<FIELD ID="col_4" datatype="float" name="flux_y_templfit" precision="10" unit="uJy">
<DESCRIPTION>NIR Y band source template fitting flux (TPHOT measurement)</DESCRIPTION>
</FIELD>
<FIELD ID="col_5" datatype="float" name="spe_line_snr_gf" precision="10">
<DESCRIPTION>The signal-to-noise ratio of the line from Gaussian fit</DESCRIPTION>
</FIELD>
<FIELD ID="col_6" datatype="float" name="spe_line_snr_di" precision="10">
<DESCRIPTION>The signal-to-noise ratio of the line from direct integration</DESCRIPTION>
</FIELD>
<FIELD ID="col_7" arraysize="*" datatype="char" name="spe_line_name">
<DESCRIPTION>The name of detected line</DESCRIPTION>
</FIELD>
<FIELD ID="col_8" datatype="float" name="spe_line_central_wl_gf" precision="10">
<DESCRIPTION>The estimated central wavelength of the detected line from Gaussian fit</DESCRIPTION>
</FIELD>
<FIELD ID="col_9" datatype="float" name="spe_line_ew_gf" precision="10">
<DESCRIPTION>The estimated equivalent width of the detected line from Gaussian fit</DESCRIPTION>
</FIELD>
<FIELD ID="col_10" datatype="float" name="spe_z_err" precision="10">
<DESCRIPTION>Uncertainty on estimated spectroscopic redshift</DESCRIPTION>
</FIELD>
<FIELD ID="col_11" datatype="float" name="spe_z" precision="10">
<DESCRIPTION>The estimated spectroscopic redshift</DESCRIPTION>
</FIELD>
<FIELD ID="col_12" datatype="float" name="spe_z_prob" precision="10">
<DESCRIPTION>The estimated spectroscopic redshift probability</DESCRIPTION>
</FIELD>
<FIELD ID="col_13" datatype="float" name="spe_line_flux_gf" precision="10">
<DESCRIPTION>The estimated flux of the detected line from Gaussian fit</DESCRIPTION>
</FIELD>
<FIELD ID="col_14" datatype="float" name="spe_line_flux_err_gf" precision="10">
<DESCRIPTION>The error on estimated flux of the detected line from Gaussian fit</DESCRIPTION>
</FIELD>
<FIELD datatype="int" name="ROW_IDX"/>
<FIELD datatype="int" name="ROW_NUM"/>
<DATA>
<TABLEDATA>
  <TR>
    <TD>2741100559643251862</TD>
    <TD>274.11005599493853</TD>
    <TD>64.32518620213646</TD>
    <TD>102158277</TD>
    <TD>14.744326</TD>
    <TD>22.495499</TD>
    <TD>4.9</TD>
    <TD>Halpha</TD>
    <TD>16342.789</TD>
    <TD>-36.514404</TD>
    <TD>4.3738916E-4</TD>
    <TD>1.4891725</TD>
    <TD>0.9999538</TD>
    <TD>3.1438575E-16</TD>
    <TD>7.4882166E-17</TD>
    <TD>0</TD>
    <TD>0</TD>
  </TR>
  <TR>
    <TD>2733456478647137226</TD>
    <TD>273.34564786062185</TD>
    <TD>64.71372260532188</TD>
    <TD>102158277</TD>
    <TD>5.430508</TD>
    <TD>20.98727</TD>
    <TD>0.0</TD>
    <TD>Halpha</TD>
    <TD>16626.322</TD>
    <TD>-90.8508</TD>
    <TD>1.9258176E-4</TD>
    <TD>1.5295725</TD>
    <TD>0.99999934</TD>
    <TD>7.1958313E-16</TD>
    <TD>4.576828E-17</TD>
    <TD>1</TD>
    <TD>1</TD>
  </TR>
</TABLEDATA>
</DATA>
</TABLE>
</RESOURCE>
</VOTABLE>
"""
>>> Table.read(BytesIO(vot_string.encode()), format='votable')
<Table length=2>
       col_0            col_1          col_2       col_3       col_4     ...    col_13       col_14    ROW_IDX ROW_NUM
         NA              deg            deg          NA         uJy      ...                                          
       int64           float64        float64      int64      float32    ...   float32      float32     int32   int32 
------------------- -------------- ------------- --------- ------------- ... ------------ ------------ ------- -------
2741100559643251862 274.1100559949 64.3251862021 102158277 14.7443256378 ... 0.0000000000 0.0000000000       0       0
2733456478647137226 273.3456478606 64.7137226053 102158277  5.4305081367 ... 0.0000000000 0.0000000000       1       1

While

>>> QTable.read(BytesIO(vot_string.encode()), format='votable')
<QTable length=2>
        col_0              col_1          col_2        col_3    ...    col_13       col_14    ROW_IDX ROW_NUM
          NA                deg            deg           NA     ...                                          
       float64            float64        float64      float64   ...   float32      float32     int32   int32 
---------------------- -------------- ------------- ----------- ... ------------ ------------ ------- -------
2.7411005596432517e+18 274.1100559949 64.3251862021 102158277.0 ... 0.0000000000 0.0000000000       0       0
2.7334564786471373e+18 273.3456478606 64.7137226053 102158277.0 ... 0.0000000000 0.0000000000       1       1

Activity

  1. bsipocz
    added
    Bug
    table
    on Mar 31, 2025

  2. mhvk commented on Apr 1, 2025

    @mhvk
    mhvk
    on Apr 1, 2025
    Contributor

    The problem would seem to be that column 0 has a unit "NA", which is interpreted as an actual unit (in VO, anything not known becomes a user-defined unit...). So, I guess we'd need some way to tell that "NA" just means no unit. If using "NA" for that purpose is standard for VO tables, it should be relatively easy to do...

  3. bsipocz commented on Apr 1, 2025

    @bsipocz
    bsipocz
    on Apr 1, 2025
    MemberAuthor

    Thanks @mhvk, we investigate this internally now, too.

  4. bsipocz commented on Apr 1, 2025

    @bsipocz
    bsipocz
    on Apr 1, 2025
    MemberAuthor

    OK, so this is something upstream has to work on, too.

    However, I would have expected a warning when parsing this unknown NA unit?

    Also, the datatype should maybe should be taken into account, and e.g. raise at least a warning when casted from into to float?

  5. mhvk commented on Apr 1, 2025

    @mhvk
    mhvk
    on Apr 1, 2025
    Contributor

    @bsipocz - Warnings are always tricky: so far, we've tried to avoid warning in situations where the user likely has no control, like in reading a file often produced elsewhere, which is why there is no warning about the unit not being recognized even though if one tries to instantiate u.Unit("NA", format="vounit"), one gets an error.

    But internally what QTable does is read in the table first as Column, and only then convert any Column with a unit to Quantity. It would seem to make sense not to do the conversion when a column has UnrecognizedUnit. I worry a bit about columns with unit="" though...

    There has also been discussion on and off about the default conversion to float. There certainly is a logical argument to just leave things as they are in the table, but the problem is that it sets up the user for shooting oneself in the foot if one subsequently tries to do anything inplace (like t["pos"] += 1*u.cm when the unit is km). But maybe that is the lesser problem, not sure...

  6. bsipocz commented on Apr 1, 2025

    @bsipocz
    bsipocz
    on Apr 1, 2025
    MemberAuthor

    I think there are different types of (missing) warnings here. First, the unit is not recognised. E.g we get warnings for that when reading it from a FITS file, so I would have expected the same behaviour when doing it from a votable.

    Then the main issue is really the int--> float conversion. I definitely want to hear about, at least as a warning, potentially as an exception, as that messes up a lot of assumptions (e.g. doing elementwise == to filter for an object ID is a very typical use case). So when the archive provided datatype is being overridden, that's a serious case and at best should be an opt-in silencing mechanism (the units coming out of the database may not make much sense or being standard compliant, but I'm fairly confident the datatype is always consistent with what the database actually stores and provides, so we should not be clever about silently overriding that).

  7. taldcroft commented on Apr 1, 2025

    @taldcroft
    taldcroft
    on Apr 1, 2025
    Member

    So when the archive provided datatype is being overridden, that's a serious case

    Totally agree here. The fact that the long datatype specified in the VOTable is not being respected seems like a real bug. Since Quantity can be integer with a unit, that would be the "normal" expected outcome here for a long with a unit.

    Except in this case NA apparently has a special meaning of "no unit", but if you had a long with unit of count, then that should be an int Quantity with unit of count.

    BTW, is NA called out in the VO spec? If not, then the upstream data provider is doing something wrong.

  8. bsipocz commented on Apr 1, 2025

    @bsipocz
    bsipocz
    on Apr 1, 2025
    MemberAuthor

    BTW, is NA called out in the VO spec? If not, then the upstream data provider is doing something wrong.

    Upstream is on it already, they are fixing the units to be "" (these are not counts but object IDs)

  9. bsipocz commented on Apr 1, 2025

    @bsipocz
    bsipocz
    on Apr 1, 2025
    MemberAuthor

    Btw, I'm not sure, shall I open a separate issue for the problematic non-round tripping of int-->float-->int? It's an upstream numpy behaviour, but again, we run into it here with the silent conversion of data types:

    Illustration:

    ...
>>> result_votable = Irsa.query_tap(adql_query)  # DALResultsTable instance
>>> result_qtable = result_votable.to_qtable()   # QTable instance
>>> result_table = result_votable.to_table()     # Table instance

>>> result_votable[0]['object_id'], result_table['object_id'][0], result_qtable['object_id'][0], result_qtable['object_id'][0].astype(np.int64)
(np.int64(2741100559643251862),
 np.int64(2741100559643251862),
 <MaskedQuantity 2.74110056e+18 NA>,
 <MaskedQuantity 2741100559643251712 NA>)

    Which all ultimately boils down to:

    >>> np.int64(2741100559643251862).astype(np.float64).astype(np.int64)
np.int64(2741100559643251712)

  10. mhvk commented on Apr 1, 2025

    @mhvk
    mhvk
    on Apr 1, 2025
    Contributor

    Looking around, I see there is even a draft PR to adjust the dtype behaviour - #12505. It sounds like it is time to revive it...

  11. taldcroft
    closed this as completedin #20273on Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Bugtable

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions