Skip to content

io.fits should return table data as Astropy Tables #199

Open

Description

@embray

This issue here as a reminder that this work needs to be done at some point (there's also a ticket for this in PyFITS' tracker). Currently PyFITS/astropy.io.fits has its own custom array class for representing FITS table data. Since FITS tables do have a few "special" features it will probably still be necessary to have a special class specifically for FITS tables, though for the sake of consistency it should at least be based on the Astropy Table class (which is a lot cleaner and more usable than what PyFITS currently has anyways).

This won't be doable for the first release--it's a big change and will be very difficult, if not impossible, to implement in a backwards compatible manner. For the first release it's probably better that astropy.io.fits (not to mention the pyfits legacy shim) work as a drop-in replacement for PyFITS.

This work will also have to be backported to PyFITS, so I'll want to tie it to an eventual PyFITS release.

Activity

  1. ghost assigned on Apr 13, 2012
  2. embray commented on Nov 2, 2012

    @embray
    MemberAuthor

    I really need to start working on this soon. I am sick to death of of people having problems with manipulating tables in pyfits. It's not their fault--given a programmable interface to a table they expect it to just work when, for example, they try to add or delete a column, or modify its format. But pyfits just never did this quite right. Or at least there aren't enough tests that the functionality has been well-maintained, if it did ever work. Using a version of the Astropy tables will finally make table manipulation just work (albeit with some inefficiencies for certain operations, but better than nothing).

  3. astrofrog commented on Sep 20, 2013

    @astrofrog
    Member

    I don't think this will be done by 0.3.0

  4. modified the milestones: v1.0.0, v0.4.0 on May 3, 2014
  5. astrofrog commented on Jan 14, 2015

    @astrofrog
    Member

    This won't happen for 1.0 (and is not critical) so removing milestone.

  6. removed this from the v1.0.0 milestone on Jan 14, 2015
  7. added a commit that references this issue on Nov 23, 2016
  8. astrofrog commented on Dec 16, 2016

    @astrofrog
    Member

    Once we find someone who can spend significant effort on io.fits, I think this would be a worthy goal to revive. Since the issue was open, we now have mix-in columns, which means that a FITS Table HDU could be represented by a Table instance (or instance of a sub-class) that includes mix-in columns for e.g. the scaled data. We could also optionally use mix-in columns for bit columns used in e.g. event tables (pretty sure @taldcroft had a prototype for this) or for easier bytes/unicode conversion (#4829)

  9. tepickering commented on May 1, 2018

    @tepickering
    Contributor

    since astropy.table.Table() now supports reading/writing of FITS tables, is this issue still relevant? it seems like io.fits could easily leverage what is already there and provide an option for converting to a Table upon read.

  10. saimn commented on May 1, 2018

    @saimn
    Contributor

    @tepickering - I think the idea here is to use astropy.table.Table directly in io.fits, replacing the FITS_rec class. But this needs quite a big effort...

  11. pllim commented on May 2, 2018

    @pllim
    Member

    This wish is 6 years old now. I propose we close this as won't fix.

  12. added
    Close?Tell stale bot that this issue/PR is stale
    on May 2, 2018
  13. bsipocz commented on May 2, 2018

    @bsipocz
    Member

    well, I feel this is still relevant and would definitely be useful. And basically as a feature request can stay open indefinite.

  14. pllim commented on May 2, 2018

    @pllim
    Member

    OK.

  15. added and removed
    Close?Tell stale bot that this issue/PR is stale
    on May 2, 2018
  16. saimn commented on May 2, 2018

    @saimn
    Contributor

    I also think it is useful to keep this kind of issue open, with more long-term ideas/vision, even if it is unlikely to happy soon given the amount of work that this requires.

  17. taldcroft commented on May 2, 2018

    @taldcroft
    Member

    I think one important consideration is back-compatibility and not breaking code in a major way. This is mentioned in the original issue description, so it would seem any path forward would involve a new interface which is inpsired by the existing direct io.fits API but using astropy table Column objects or appropriately tweaked mixin objects (as suggested earlier by @astrofrog). At that point the "classic" interface could be deprecated but likely left in for a long time. It'll be a big job but gives an opportunity for a fresh start on API and full integration with astropy table.

    One day there will be a major celebration when issue #199 gets closed! Just like our #3000 celebration.

  18. SaOgaz commented on May 22, 2018

    @SaOgaz
    Contributor

    I'd like to point out that there are still some problems with round-tripping and the headers that should probably be fixed up before this happens. i.e. the comments on keywords getting clipped, as well as regular strings in the headers getting re-ordered (should probably file an actual issue for this one, although I'm not sure it's something that needs to be "fixed" or it's just the way things are). Just thought I'd mention those here :D Or would this potentially leave the headers as is, and just change the data part of the HDU to an astropy Table?

  19. added 2 commits that reference this issue on Jun 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions