Skip to content

Conversation

@astrofrog
Copy link
Member

@astrofrog astrofrog commented Nov 5, 2018

Edit: THIS REQUIRES #9009, keep open while that is addressed.

Before this fix, the following failed:

>>> from astropy.time import Time
>>> from astropy.table import Table
>>> time = Time(['2016-03-22T12:30:31', '2016-03-22T12:30:38', '2016-03-22T12:34:40'])
>>> table = Table()
>>> table['time'] = time
>>> table.add_index('time')
>>> table.write('test.fits', overwrite=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/table.py", line 2592, in write
    io_registry.write(self, *args, **kwargs)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/registry.py", line 560, in write
    writer(data, *args, **kwargs)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/fits/connect.py", line 388, in write_table_fits
    table_hdu = table_to_hdu(input, character_as_bytes=True)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/fits/convenience.py", line 470, in table_to_hdu
    table, hdr = time_to_fits(table)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/fits/fitstime.py", line 533, in time_to_fits
    newtable.replace_column(col.info.name, Column(jd12, unit='d'))
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/table.py", line 1796, in replace_column
    raise ValueError('cannot replace a table index column')
ValueError: cannot replace a table index column

The solution is to remove the index columns.

@taldcroft - is there a better way to just remove all indices from a table?

@astropy-bot
Copy link

astropy-bot bot commented Nov 5, 2018

Check results are now reported in the status checks at the bottom of this page.

@astropy astropy deleted a comment from astropy-bot bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot bot Nov 5, 2018
@saimn
Copy link
Contributor

saimn commented Nov 5, 2018

It would be more generic to handle this case in Table.replace_column instead of raising an error. Why not dropping the index there, maybe with a keyword or warning ?

@astrofrog
Copy link
Member Author

@taldcroft - do you have any thoughts on this?

@astrofrog
Copy link
Member Author

astrofrog commented Nov 7, 2018

@saimn - on further thoughts, I guess my thinking is that Table has already made a choice to raise a clear exception in this use case of replace_column, and this PR fixes a bug in the FITS code that uses that. Changing the behavior in Table would actually be an API change. So I personally think we should just treat this as a bug fix and not modify Table.

I guess we could also add a keyword argument to control this but again this would be a change in API and I'd like to get this fix in v3.1.

@taldcroft
Copy link
Member

This looks fine as a bug fix. Ideally the right answer is to serialize the index so it round-trips, which would make generating the index more useful since it would persist. In theory the machinery is mostly there, but it probably would not be trivial in the end.

@saimn
Copy link
Contributor

saimn commented Nov 7, 2018

@astrofrog - I can understand your reasoning, but for me it is a Table thing, not FITS or Time. Looking back in time, #4090 (comment) shows - I think - that there is no real reason for raising an error in this case.
I don't know what would the good solution when replacing a column with an index (recreating all the indexes using this column may be tricky), but a simple solution could be to add a drop_index keyword, which would not change the default behavior.
And maybe another option would be to remove the use of replace_colum, I'm not sure if it is necessary here ? At least replacing it with newtable[col.info.name] = jd12 does not cause a failure in the unit tests.

@saimn
Copy link
Contributor

saimn commented Nov 7, 2018

@taldcroft - the serialization is not the issue (it seems that indexes are dropped when writing tables to fits), the crash happens when using replace_column here:

# The following is necessary to deal with multi-dimensional ``Time`` objects
# (i.e. where Time.shape is non-trivial).
jd12 = np.array([col.jd1, col.jd2])
# Roll the 0th (innermost) axis backwards, until it lies in the last position
# (jd12.ndim)
jd12 = np.rollaxis(jd12, 0, jd12.ndim)
newtable.replace_column(col.info.name, Column(jd12, unit='d'))

@codecov
Copy link

codecov bot commented Nov 8, 2018

Codecov Report

Merging #8077 into master will decrease coverage by 0.03%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8077      +/-   ##
==========================================
- Coverage   86.96%   86.92%   -0.04%     
==========================================
  Files         408      383      -25     
  Lines       61428    57826    -3602     
  Branches     1096     1056      -40     
==========================================
- Hits        53422    50267    -3155     
+ Misses       7383     6945     -438     
+ Partials      623      614       -9
Impacted Files Coverage Δ
astropy/io/fits/fitstime.py 84.5% <100%> (+0.15%) ⬆️
astropy/convolution/__init__.py 62.5% <0%> (-37.5%) ⬇️
astropy/wcs/__init__.py 50% <0%> (-16.67%) ⬇️
astropy/visualization/wcsaxes/coordinates_map.py 86.88% <0%> (-10.18%) ⬇️
astropy/modeling/mappings.py 84.31% <0%> (-8.79%) ⬇️
astropy/convolution/src/convolve.c 80.66% <0%> (-8.36%) ⬇️
astropy/wcs/src/unit_list_proxy.c 53.96% <0%> (-6.91%) ⬇️
astropy/visualization/units.py 68.08% <0%> (-5.5%) ⬇️
astropy/modeling/separable.py 90.62% <0%> (-5.26%) ⬇️
astropy/constants/__init__.py 94.87% <0%> (-5.13%) ⬇️
... and 277 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 253ea74...af611a3. Read the comment docs.

@astrofrog astrofrog modified the milestones: v3.1, v3.2 Nov 13, 2018
@astrofrog
Copy link
Member Author

Ok let's delay this for now, I don't have time to work on it more for 3.1

@astrofrog astrofrog modified the milestone: v3.2 Nov 22, 2018
Copy link
Member

@taldcroft taldcroft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in discussion comment.

@astropy-bot
Copy link

astropy-bot bot commented Jun 15, 2019

Hi humans 👋 - this pull request hasn't had any new commits for approximately 7 months. I plan to close this in a month if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

@astropy-bot
Copy link

astropy-bot bot commented Jul 17, 2019

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.

@taldcroft
Copy link
Member

I re-opened this as I think that this approach is OK and fixes a real issue.

The only outstanding problem is the comment above that the current code actually corrupts the original table by removing indices on newtable. That's basically a bug that I plan to fix in #9009.

@bsipocz
Copy link
Member

bsipocz commented Jul 17, 2019

@astrofrog - could you please rebase and address the comment to make it ready to be merged?

@taldcroft
Copy link
Member

@bsipocz - I think you just missed that this would be best done by waiting on a fix for #9009, at which point the table copy (with copy=False) will do the right thing.

However, there definitely should be a test that this operation does not corrupt the original table. That will fail right now, but will start working once #9009 is fixed.

@bsipocz
Copy link
Member

bsipocz commented Jul 17, 2019

Oh, indeed. Then I add the keep-open here, as either that or a rebase that makes sure this doesn't get closed again when the bot runs next time.

@bsipocz bsipocz modified the milestones: v3.2.2, v4.0 Sep 27, 2019
@taldcroft taldcroft force-pushed the fix-table-io-time-index branch from 60d051e to 1ef279e Compare October 22, 2019 17:41
@taldcroft taldcroft force-pushed the fix-table-io-time-index branch from 1ef279e to af611a3 Compare October 22, 2019 17:45
@taldcroft
Copy link
Member

@astrofrog et al. - I think should be good now, please review. The crux lines of code depend on table improvements in 4.0-dev, so I don't recommend trying to backport.

@bsipocz bsipocz removed the keep-open label Oct 22, 2019
@bsipocz
Copy link
Member

bsipocz commented Oct 22, 2019

@taldcroft - There are no more backports except critical fixes like the one we need for iers :)

@bsipocz
Copy link
Member

bsipocz commented Oct 22, 2019

@taldcroft - #9009 still seems to be relevant, is that not an issue for this any more?

@taldcroft
Copy link
Member

#9009 is still relevant because it will make it possible to improve the implementation here to avoid making any copies. Right now it does copy any mixin columns that have indices. So I put in a breadcrumb in #9009 to come back here and have a look.

@taldcroft taldcroft merged commit 04e6d66 into astropy:master Oct 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants