Skip to content

Support CDS/MRT format writing for Time columns [GSoC21b] - #12027

Closed
Suyog7130 wants to merge 9 commits into
astropy:mainfrom
Suyog7130:time-cols-in-cdsMRT
Closed

Suyog7130 wants to merge 9 commits into
astropy:mainfrom
Suyog7130:time-cols-in-cdsMRT

Conversation

@Suyog7130

@Suyog7130 Suyog7130 commented Aug 9, 2021 •

Copy link
Copy Markdown
Contributor

Description

This Pull Request is part of the 2021 Google Summer of Code work I am doing together with mentors @aaryapatil and @hamogu.
It is a continuation of the modifications in #11897 which primarily aims to address issue #11257.

Note that the count appended at the end of the PR title, [GSoC21a], [GSoC21b] and so on, denote the sequence in which rebasing has been (will be) done to the commits. It is preferred if the merging also takes place in this order. Also, because the previous PRs are not merged yet, the later ones contain numerous unrelated commits at the time of opening the PR.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see "When to rebase and squash commits".
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the Extra CI label.
  • Is a change log needed? If yes, did the change log check pass? If no, add the no-changelog-entry-needed label.
  • Is a milestone set? Milestone must be set but astropy-bot check might be missing; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate backport-X.Y.x label(s) before merge.

Checklist for PR author

  • Give time column format and scale in Byte-By-Byte description.
  • Work on other edit suggestions.

@Suyog7130

Suyog7130 commented Aug 9, 2021 •

Copy link
Copy Markdown
Contributor Author

Both the CDS and MRT format do not have any particular standard for Time valued columns.
See https://vizier.u-strasbg.fr/doc/catstd-3.1.htx and https://journals.aas.org/mrt-standards/

#11897 has added MRT format table writing to Astropy. This PR directly appends to it and provides support for internally converting Time columns or columns with Time values. All Time values are converted to Modified Julian Date (MJD) value with a precision of 12 digits after decimal. Conversion to MJD has been chosen instead of say, writing in datetime format or in decimal, because both CDS and MRT standard do not have any restrictions on this and because MJD values are relatively more precise. This is essentially also the reason for not implementing division of datetime objects into its constituent components. Column label and description are altered when necessary (have a look at the examples in the tests for this). If some column contains only a few Time values, then no Time conversion is done, instead the whole column is modified to have only string values, as done for other mix-in columns.

The commits relating to the present PR start after the one titled add roundtrip test for MRT tables

@Suyog7130

Copy link
Copy Markdown
Contributor Author

Two further additions that can be done are:

  • Support for TimeDelta values. They can be made to convert to seconds if the duration is <1000 sec, or to Julian Days if otherwise.
  • Support for datetime objects. Columns for year, month etc. can be extracted out.

However, both of these are not specific to CDS/MRT standard and there may also be fewer use cases for having any default behaviour.

Suyog7130 added a commit to Suyog7130/astropy that referenced this pull request Aug 9, 2021
@hamogu

hamogu commented Aug 16, 2021

Copy link
Copy Markdown
Member

I would leave the possible additions for later. Let's see how people use that and what they tell us.

@Suyog7130

Copy link
Copy Markdown
Contributor Author

All Time values are converted to Modified Julian Date (MJD) value with a precision of 12 digits after decimal. Conversion to MJD has been chosen instead of say, writing in datetime format or in decimal, because both CDS and MRT standard do not have any restrictions on this and because MJD values are relatively more precise. This is essentially also the reason for not implementing division of datetime objects into its constituent components. Column label and description are altered when necessary (have a look at the examples in the tests for this). If some column contains only a few Time values, then no Time conversion is done, instead the whole column is modified to have only string values, as done for other mix-in columns.

However, the CDS format instructions on Labels say to use complete Julian Dates instead of MJD etc.? (see comments in http://vizier.u-strasbg.fr/doc/catstd-3.3.htx)

Julian dates should be complete – no truncated Julian date, no Modified Julian Date (MJD) which are the source of many confusions.

In contrast to this MRT standard has a MJD example on their page. (see the example towards the end in https://journals.aas.org/mrt-standards/)

Suyog7130 added a commit to Suyog7130/astropy that referenced this pull request Sep 1, 2021
@Suyog7130

Copy link
Copy Markdown
Contributor Author

There is only a single line that is not covered by the tests, however the codecov/project test shows below satisfactory coverage!?

@Suyog7130

Copy link
Copy Markdown
Contributor Author

Apart from that, @aaryapatil and @hamogu, this PR is complete for the time being.

@hamogu

hamogu commented Sep 2, 2021

Copy link
Copy Markdown
Member

Don't worry about the coverage failure. We cam merge even with that failure. I tried to see more, but I cannot access the coverage logs (alternating between "too many requests from github" and "you are not authorized "- either way, I can't see the details).

Comment thread docs/io/ascii/write.rst
the writer does not do any internal conversion or modification. These columns should be
converted to regular columns with proper ``unit`` and ``name`` attribute before writing
the table. Thus::
For columns with `~astropy.time.Time` values and for columns that are thus, the values are

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this sentence. Do you mean "columns that contain Time objects are are a time mix-in column?
How about

Suggested change
For columns with `~astropy.time.Time` values and for columns that are thus, the values are
For columns holding `~astropy.time.Time` values, the values are

which can be read as "one (= the column is a Time object) or more (the column is a list of Time objects)".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, that's what I meant there. Your suggestion is far more spot on.

@hamogu

hamogu commented Sep 2, 2021

Copy link
Copy Markdown
Member

Just supporting your comment above: CDS explicitly warns against MJD, while MRT uses it. For now, we write MRT tables, but something to keep in mind when splitting off CDS tables.

This PR also still needs a 12027.feature.rst file. Otherwise, I think this is almost ready to go. That's the advantage of having smaller PRs - easier to write, easier to review, and faster to merge ;-)

@aaryapatil aaryapatil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments specific to round-tripping of Time values. This is based on my previous knowledge of how time is dealt with in Astropy, so feel free to correct me if I'm wrong.

Comment thread astropy/io/ascii/cds.py
Example::

--------------------------------------------------------------------------------
================================================================================

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why you have changed the header line?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That example dated back to the time we were the pycdsreadme template for the ReadMe.
Changed to = for header since the example on MRT standard homepage uses it.

Comment thread astropy/io/ascii/cds.py
if col.name is not None:
name = col.name
try:
self.cols[i] = Column([tval.mjd for tval in col],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be changed to tval.to_value('mjd', 'long') for longdouble precision, and go beyond the format='.12f'.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see that.
However, is having greater precision than 12 decimal digits is a requirement?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. Unfortunately, I don't think we can have a one-size-fits-all answer. Imaging you make a long-term lightcurve over years for a variable star, going back from historical records in 1894 to today. In that case, precision to a day is enough and a precision of 12 digits would in fact be misleading. For the historical records, you won't know the time of the observation better than an hour (or a minute, if they were very good in writing down their notes and all notes survived to today), even for modern observations, the exposure time is often a few min to an hour for faint objects, so there is really no point to be more precise than a minute or so. Everything above that is misleading. On the other hand, the time column could hold the data for pulsar timing, where we want to measure the spin-up or spin-down of the pulsar. Typical pulse rates are 300 Hz or more, so if we want to measure spin-down, we need to be precise on the mirco or nano-second level.

I'm in fact not sure how to address that, since the "time" does not know about the precision needed by the user. Internally, it's just a variable with two floats, but for printing out for human consumption (which is what we do here, even if the format is called machine-readable), we do need that extra bit of information to make sense. For "normal" columns, the user can set the "format" to decide how many digits they want. I don't think time columns have that, though? Maybe @taldcroft has an idea how to address that?

I don't know much about the mix-in columns and don't have time to dig into the code right now to see if this information can be attached by the user in some way, but on top of my head I think we might need to use the "formats" dictionary for the write function. If no format is given for the time column, we pick some default (I suggest 4 decimal digits - a few second precision) and of someone wants more or less, they can change that. (That would profit from an example in the narrative docs. Can be short, just text like "The default is to print time in days as recommended by the MRT standard [link here] with four decimal digits. The precision can be changed like this: tab.write(..., formats={'name of time colum'}: '%15.12f')." - no need to have an example table and an executable example here, just a line of inline text woudl be enough, but we may need a test for this.)

Comment thread astropy/io/ascii/cds.py
try:
self.cols[i] = Column([tval.mjd for tval in col],
name = name,
unit = u.day,

@aaryapatil aaryapatil Sep 2, 2021 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This goes back to @taldcroft's comment

The u.day unit is still not appropriate, and probably worse than u.yr. The point here is that a Time does not have a unit. A value of 2019.0 is a decimal representation of a point in time that is equivalent to 2019-01-01, which is a string representation of that point in time. Neither of those have a meaningful unit.

This is applicable here too because Modified Julian Date is a "date" especially when we talk about it in terms of a Time column. TimeDelta would probably be more appropriate for using units, but I am not so sure about units being used here. However, this might be overly pedantic and maybe the MRT format doesn't talk about it. For now we can go ahead with this, but I will try to find out whether the format suggests a unit for time coordinates, and why does it not ask for a scale since it is important for interpreting and round-tripping precise times.

Also, when the time scale is UTC, one day is not exactly 86400 seconds but u.day is.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Will make the change. Thanks for the input. :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaryapatil became quite the expert in time systems in her GSOC, so we can trust here here!

@Suyog7130 Suyog7130 Sep 4, 2021 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is applicable here too because Modified Julian Date is a "date" especially when we talk about it in terms of a Time column.

Ah, I do recall having thought over this while assigning the unit as u.day. The example on the MRT standard's page gives the unit of the BJD column as such. (https://journals.aas.org/mrt-standards/)

Again, I think the dates would have the unit of a day since their resolution is a day, in spite of the fact that the defination of the day may differ.
However, likewise can be argued for date value given in other formats and for the decimal year!?

cc: @taldcroft

Comment thread astropy/io/ascii/cds.py
if isinstance(col, Column):
if col.description is not None:
description = col.description + ' (Modified Julian Date)'
if col.name is not None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though you are using isinstance(col, Column) and col.name and col.description can be directly accessed for a column, but would it be better to make this the "mixin-safe" col.info.name and so on. Don't think it is possible to have a Quantity column with Time values, but maybe there is some "mixin-specific" case that we are missing...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked up the QTables and Quantity columns. For now, I suppose using the if statement suffices our purpose. I can work on Quantity with Time values and other QTable specific scenario in a separate PR later.

@Suyog7130

Copy link
Copy Markdown
Contributor Author

This PR also still needs a 12027.feature.rst file. Otherwise, I think this is almost ready to go. That's the advantage of having smaller PRs - easier to write, easier to review, and faster to merge ;-)

It is already added in commit add changelog for #12027

@taldcroft

taldcroft commented Sep 5, 2021 •

Copy link
Copy Markdown
Member

I haven't had a chance to read through everything in detail, but my quick thought is to represent a Time object in the same way that the user wants. I.e. if tm is a Time object and tm.format == 'mjd' then output MJD, but if they choose to have tm.format == 'isot' then they get ISO-8601. In this case then tm also has a precision attribute that controls the output precision for string-based formats like isot. This is easy and happens already by default for io.ascii output of Time objects.

Since MRT and CDS both do not specify an requirements for time formatting, I think it is best to use what the user has set for their Time object.

And I will repeat again my conviction that Time objects do not have a meaningful unit. Astropy needs to be strict about the interpretation of units and so the unit should be empty. TimeDelta object do have a meaningful unit.

@taldcroft

Copy link
Copy Markdown
Member

For example, see the simple patch below. I think the Time special handling here could consist of putting the time format and time scale into the description or else an associated note. That's the only official place in MRT to store this kind of metadata.

diff --git a/astropy/io/ascii/cds.py b/astropy/io/ascii/cds.py
index 77740bdbbf..c2bfb1df5f 100644
--- a/astropy/io/ascii/cds.py
+++ b/astropy/io/ascii/cds.py
@@ -27,6 +27,7 @@ from astropy import units as u
 
 from astropy.table import Table
 from astropy.table import Column, MaskedColumn
+from astropy.time import Time
 from string import Template
 from textwrap import wrap
 
@@ -647,6 +648,10 @@ class CdsHeader(core.BaseHeader):
 
                 to_pop.append(i)   # Delete original ``SkyCoord`` column.
 
+            elif isinstance(col, Time):
+                col = Column(col.value, name=col.info.name, description=col.info.description)
+                self.cols[i] = col
+
             # Convert all other ``mixin`` columns to ``Column`` objects.
             # Parsing these may still lead to errors!
             elif not isinstance(col, Column):

With this:

In [2]: t = simple_table()

In [3]: t['tm'] = Time([1,2,3], format='cxcsec')

In [4]: ascii.write(t, format='cds')
Title:
Authors:
Table:
================================================================================
Byte-by-byte Description of file: table.dat
--------------------------------------------------------------------------------
 Bytes Format Units  Label     Explanations
--------------------------------------------------------------------------------
    1  I1     ---    a       [1/3] Description of a     
 3- 5  F3.1   ---    b       [1.0/3.0] Description of b 
    7  A1     ---    c       Description of c           
 9-11  F3.1   ---    tm      [1.0/3.0] Description of tm
--------------------------------------------------------------------------------
Notes:
--------------------------------------------------------------------------------
1 1.0 c 1.0
2 2.0 d 2.0
3 3.0 e 3.0

In [5]: t['tm'].format = 'isot'

In [6]: ascii.write(t, format='cds')
Title:
Authors:
Table:
================================================================================
Byte-by-byte Description of file: table.dat
--------------------------------------------------------------------------------
 Bytes Format Units  Label     Explanations
--------------------------------------------------------------------------------
    1  I1     ---    a       [1/3] Description of a    
 3- 5  F3.1   ---    b       [1.0/3.0] Description of b
    7  A1     ---    c       Description of c          
 9-31  A23    ---    tm      Description of tm         
--------------------------------------------------------------------------------
Notes:
--------------------------------------------------------------------------------
1 1.0 c 1998-01-01T00:00:01.000
2 2.0 d 1998-01-01T00:00:02.000
3 3.0 e 1998-01-01T00:00:03.000

@Suyog7130

Copy link
Copy Markdown
Contributor Author

Thank you once again for your helpful suggestions.
The only thing I would perhaps want in the writer is default writing behaviour to MJD (with a precision of 4). The format keyword can still be used for any other formats.
What do others think?

@hamogu

hamogu commented Sep 6, 2021

Copy link
Copy Markdown
Member

I like @taldcroft suggestion. I think that time always has some format set, because you have to give it a format to put any values in. (That can be implicit. If you put in the string '2019-04-04T12:34:45' it will recognize that that's an ISO date, and thus know you initialized this object with an ISO date). So, this is pretty close to what I said before when I argues that it has to be user-configurable. This is even better, because the time object already knows the format the user set, so no extra parameter is needed in the tab.write(...) call, and the user has a clear and easy way to change the format - simlpy set the format on your time column to a different time format, like you would set the format on the float column to %3.1f.
If time has not format set, then MJD or ISOT would be reasonable defaults, but, as I said, I'm not sure you can ever have a time object with no format set.

@taldcroft

Copy link
Copy Markdown
Member

Time always has a format.

@Suyog7130

Suyog7130 commented Sep 6, 2021 •

Copy link
Copy Markdown
Contributor Author

Since we are providing the special provision for cases when columns are Time objects or they have values that are Time objects, the values will always have a format. So, I suppose, no need for conversion to MJD!

@Suyog7130

Copy link
Copy Markdown
Contributor Author

I think the Time special handling here could consist of putting the time format and time scale into the description or else an associated note. That's the only official place in MRT to store this kind of metadata.

tm.format has the short form of format name saved to it, like mjd. @aaryapatil, do you know a way to get the full name of the time format? Or should I simply append the values returned by tm.format and tm.scale to the Byte-By-Byte description?

@hamogu

hamogu commented Sep 6, 2021

Copy link
Copy Markdown
Member

But if this is a column ob objects, where each object is a time object, then each of them could have a different format. In that case, should be convert all to the format of the first row? Or just leave it to the user to make sure they all have compatible formats set (e.g. not JD in row q and MJD in row 2)?

@hamogu

hamogu commented Sep 6, 2021

Copy link
Copy Markdown
Member

I think appending to the description is fine. The formats we pretty understandable, and, as I always say, the user or editor is free to edit the text description if they want to be more verbose.

@Suyog7130

Suyog7130 commented Sep 6, 2021 •

Copy link
Copy Markdown
Contributor Author

But if this is a column ob objects, where each object is a time object, then each of them could have a different format. In that case, should be convert all to the format of the first row?

Fine tuning to check up format for each value may not be the best choice for long columns.

Or just leave it to the user to make sure they all have compatible formats set (e.g. not JD in row q and MJD in row 2)?

We can write about this in the Docs and leave it upto the user, assuming the column has the format of the first value.

@Suyog7130

Suyog7130 commented Sep 6, 2021 •

Copy link
Copy Markdown
Contributor Author

About the name of the Time columns when no name is present, MRT shows columns names as MJD/BJD. However, it doesn't have any predefined labels for them. The only predefined label associated with time values is Epoch. Also, column name and description as in the following do not have much appeal and can be a bit confusing. So, perhaps, column name should be the name of the Time format only for MJD/BJD and for others we can give Time and Description of Time (format:xx, scale:xx)?

-  '37-54  F18.12 d      MJD     [48136.51/48136.52] Modified Julian Date',
+  '37-59  A23    ---    ISO     (format:iso, scale:utc)',

The above is from the current test output, which expects the internal conversion to MJD. Thus the different in the column format values.

@Suyog7130

Copy link
Copy Markdown
Contributor Author

Uhm... When I try to set the precision as tm.precision = 12 the following error comes up. Here tm is same as given by @taldcroft above.

t['obs'].precision = 12
  File "/media/suyog/DATA/Python/astropy_fork/astropy/time/core.py", line 609, in precision
    raise ValueError('precision attribute must be an int between '
ValueError: precision attribute must be an int between 0 and 9

This means that when the columns is initialized as Time object as say, t['obs'] = Time('1990-9-2 12:23:6.125468') then when subsequently it's format is changed to one having float value, say t['obs'].format = 'mrt', then when the precision cannot be set (since t['obs'].precision = 12 doesn't work for more than 9 digits), ascii.write will output an arbitary number of digits after the decimal. This would create a difference in the output table for different operating systems/computers, a problem we did face earlier.

@taldcroft

Copy link
Copy Markdown
Member

@Suyog7130 - something like mjd will always be a 64-bit float regardless of OS/computer, and I think that the string representation of that 64-bit float will be the same. And even if that is wrong, I'll give you some of my favorite words to live by: perfect is the enemy of good; and simple is better than complex.

In other words, if you have a simple solution that works well in 99% of real-world cases, then think about the impact / consequence of the 1%. Is it going to be a bug that makes an exception? Is it going to give an actual wrong result? If the answer to those is no then the "good" solution is probably fine.

@taldcroft

Copy link
Copy Markdown
Member

BTW, the precision attribute only applies to string formats in Time like iso. Formats that are numeric don't currently have a way to control the output representation.

@hamogu

hamogu commented Sep 6, 2021

Copy link
Copy Markdown
Member

There can be an issue with testing, but -in this case- that should not drive the code design. If this is good and passes on our normal architectures, I'll find a a way to make it pass on the 32 bit system, too. In the worst case, we just skip that test. Last time that came up, we also thought about just finding a test case of "special" values that will pass, or changing the comparison in the test to only compare the first few digits with the output. I know you find a more general solution there, which I liked, but I don't think that's necessary here. The output format is given by the time object and that's tested already, so we don't have to worry about it here. We just need to make sure the byte-by-byte description gives the correct column width; if the last few digits are different that's fine as long as the byte-by-byte and the table are consistent in every case.

@github-actions github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Jan 30, 2022
@github-actions

Copy link
Copy Markdown
Contributor

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

@github-actions github-actions Bot added the closed-by-bot Closed by stale bot label Mar 2, 2022
@github-actions

github-actions Bot commented Mar 2, 2022

Copy link
Copy Markdown
Contributor

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Close? Tell stale bot that this issue/PR is stale closed-by-bot Closed by stale bot io.ascii unified-io

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants