Repository navigation
Conversation
|
Both the CDS and MRT format do not have any particular standard for Time valued columns. #11897 has added MRT format table writing to Astropy. This PR directly appends to it and provides support for internally converting The commits relating to the present PR start after the one titled add roundtrip test for MRT tables |
|
Two further additions that can be done are:
However, both of these are not specific to CDS/MRT standard and there may also be fewer use cases for having any default behaviour. |
|
I would leave the possible additions for later. Let's see how people use that and what they tell us. |
|
All However, the CDS format instructions on Labels say to use complete Julian Dates instead of MJD etc.? (see comments in http://vizier.u-strasbg.fr/doc/catstd-3.3.htx)
In contrast to this MRT standard has a MJD example on their page. (see the example towards the end in https://journals.aas.org/mrt-standards/) |
8f26339 to
95c6494
Compare
convert to `mjd` if whole column is `Time` object
95c6494 to
ecadabc
Compare
|
There is only a single line that is not covered by the tests, however the codecov/project test shows below satisfactory coverage!? |
|
Apart from that, @aaryapatil and @hamogu, this PR is complete for the time being. |
|
Don't worry about the coverage failure. We cam merge even with that failure. I tried to see more, but I cannot access the coverage logs (alternating between "too many requests from github" and "you are not authorized "- either way, I can't see the details). |
| the writer does not do any internal conversion or modification. These columns should be | ||
| converted to regular columns with proper ``unit`` and ``name`` attribute before writing | ||
| the table. Thus:: | ||
| For columns with `~astropy.time.Time` values and for columns that are thus, the values are |
There was a problem hiding this comment.
I don't understand this sentence. Do you mean "columns that contain Time objects are are a time mix-in column?
How about
| For columns with `~astropy.time.Time` values and for columns that are thus, the values are | |
| For columns holding `~astropy.time.Time` values, the values are |
which can be read as "one (= the column is a Time object) or more (the column is a list of Time objects)".
There was a problem hiding this comment.
Ya, that's what I meant there. Your suggestion is far more spot on.
|
Just supporting your comment above: CDS explicitly warns against MJD, while MRT uses it. For now, we write MRT tables, but something to keep in mind when splitting off CDS tables. This PR also still needs a |
aaryapatil
left a comment
There was a problem hiding this comment.
Left a few comments specific to round-tripping of Time values. This is based on my previous knowledge of how time is dealt with in Astropy, so feel free to correct me if I'm wrong.
| Example:: | ||
|
|
||
| -------------------------------------------------------------------------------- | ||
| ================================================================================ |
There was a problem hiding this comment.
Is there a reason why you have changed the header line?
There was a problem hiding this comment.
That example dated back to the time we were the pycdsreadme template for the ReadMe.
Changed to = for header since the example on MRT standard homepage uses it.
| if col.name is not None: | ||
| name = col.name | ||
| try: | ||
| self.cols[i] = Column([tval.mjd for tval in col], |
There was a problem hiding this comment.
I wonder if this should be changed to tval.to_value('mjd', 'long') for longdouble precision, and go beyond the format='.12f'.
There was a problem hiding this comment.
Let me see that.
However, is having greater precision than 12 decimal digits is a requirement?
There was a problem hiding this comment.
Good question. Unfortunately, I don't think we can have a one-size-fits-all answer. Imaging you make a long-term lightcurve over years for a variable star, going back from historical records in 1894 to today. In that case, precision to a day is enough and a precision of 12 digits would in fact be misleading. For the historical records, you won't know the time of the observation better than an hour (or a minute, if they were very good in writing down their notes and all notes survived to today), even for modern observations, the exposure time is often a few min to an hour for faint objects, so there is really no point to be more precise than a minute or so. Everything above that is misleading. On the other hand, the time column could hold the data for pulsar timing, where we want to measure the spin-up or spin-down of the pulsar. Typical pulse rates are 300 Hz or more, so if we want to measure spin-down, we need to be precise on the mirco or nano-second level.
I'm in fact not sure how to address that, since the "time" does not know about the precision needed by the user. Internally, it's just a variable with two floats, but for printing out for human consumption (which is what we do here, even if the format is called machine-readable), we do need that extra bit of information to make sense. For "normal" columns, the user can set the "format" to decide how many digits they want. I don't think time columns have that, though? Maybe @taldcroft has an idea how to address that?
I don't know much about the mix-in columns and don't have time to dig into the code right now to see if this information can be attached by the user in some way, but on top of my head I think we might need to use the "formats" dictionary for the write function. If no format is given for the time column, we pick some default (I suggest 4 decimal digits - a few second precision) and of someone wants more or less, they can change that. (That would profit from an example in the narrative docs. Can be short, just text like "The default is to print time in days as recommended by the MRT standard [link here] with four decimal digits. The precision can be changed like this: tab.write(..., formats={'name of time colum'}: '%15.12f')." - no need to have an example table and an executable example here, just a line of inline text woudl be enough, but we may need a test for this.)
| try: | ||
| self.cols[i] = Column([tval.mjd for tval in col], | ||
| name = name, | ||
| unit = u.day, |
There was a problem hiding this comment.
This goes back to @taldcroft's comment
The u.day unit is still not appropriate, and probably worse than u.yr. The point here is that a Time does not have a unit. A value of 2019.0 is a decimal representation of a point in time that is equivalent to 2019-01-01, which is a string representation of that point in time. Neither of those have a meaningful unit.
This is applicable here too because Modified Julian Date is a "date" especially when we talk about it in terms of a Time column. TimeDelta would probably be more appropriate for using units, but I am not so sure about units being used here. However, this might be overly pedantic and maybe the MRT format doesn't talk about it. For now we can go ahead with this, but I will try to find out whether the format suggests a unit for time coordinates, and why does it not ask for a scale since it is important for interpreting and round-tripping precise times.
Also, when the time scale is UTC, one day is not exactly 86400 seconds but u.day is.
There was a problem hiding this comment.
Sure. Will make the change. Thanks for the input. :)
There was a problem hiding this comment.
@aaryapatil became quite the expert in time systems in her GSOC, so we can trust here here!
There was a problem hiding this comment.
This is applicable here too because Modified Julian Date is a "date" especially when we talk about it in terms of a
Timecolumn.
Ah, I do recall having thought over this while assigning the unit as u.day. The example on the MRT standard's page gives the unit of the BJD column as such. (https://journals.aas.org/mrt-standards/)
Again, I think the dates would have the unit of a day since their resolution is a day, in spite of the fact that the defination of the day may differ.
However, likewise can be argued for date value given in other formats and for the decimal year!?
cc: @taldcroft
| if isinstance(col, Column): | ||
| if col.description is not None: | ||
| description = col.description + ' (Modified Julian Date)' | ||
| if col.name is not None: |
There was a problem hiding this comment.
Even though you are using isinstance(col, Column) and col.name and col.description can be directly accessed for a column, but would it be better to make this the "mixin-safe" col.info.name and so on. Don't think it is possible to have a Quantity column with Time values, but maybe there is some "mixin-specific" case that we are missing...
There was a problem hiding this comment.
I haven't looked up the QTables and Quantity columns. For now, I suppose using the if statement suffices our purpose. I can work on Quantity with Time values and other QTable specific scenario in a separate PR later.
It is already added in commit add changelog for #12027 |
|
I haven't had a chance to read through everything in detail, but my quick thought is to represent a Since MRT and CDS both do not specify an requirements for time formatting, I think it is best to use what the user has set for their And I will repeat again my conviction that |
|
For example, see the simple patch below. I think the With this: |
|
Thank you once again for your helpful suggestions. |
|
I like @taldcroft suggestion. I think that time always has some format set, because you have to give it a format to put any values in. (That can be implicit. If you put in the string '2019-04-04T12:34:45' it will recognize that that's an ISO date, and thus know you initialized this object with an ISO date). So, this is pretty close to what I said before when I argues that it has to be user-configurable. This is even better, because the time object already knows the format the user set, so no extra parameter is needed in the |
|
Time always has a format. |
|
Since we are providing the special provision for cases when columns are |
|
|
But if this is a column ob objects, where each object is a time object, then each of them could have a different format. In that case, should be convert all to the format of the first row? Or just leave it to the user to make sure they all have compatible formats set (e.g. not JD in row q and MJD in row 2)? |
|
I think appending to the description is fine. The formats we pretty understandable, and, as I always say, the user or editor is free to edit the text description if they want to be more verbose. |
Fine tuning to check up format for each value may not be the best choice for long columns.
We can write about this in the Docs and leave it upto the user, assuming the column has the format of the first value. |
|
About the name of the Time columns when no name is present, MRT shows columns names as MJD/BJD. However, it doesn't have any predefined labels for them. The only predefined label associated with time values is The above is from the current test output, which expects the internal conversion to MJD. Thus the different in the column format values. |
|
Uhm... When I try to set the precision as This means that when the columns is initialized as |
|
@Suyog7130 - something like In other words, if you have a simple solution that works well in 99% of real-world cases, then think about the impact / consequence of the 1%. Is it going to be a bug that makes an exception? Is it going to give an actual wrong result? If the answer to those is no then the "good" solution is probably fine. |
|
BTW, the |
|
There can be an issue with testing, but -in this case- that should not drive the code design. If this is good and passes on our normal architectures, I'll find a a way to make it pass on the 32 bit system, too. In the worst case, we just skip that test. Last time that came up, we also thought about just finding a test case of "special" values that will pass, or changing the comparison in the test to only compare the first few digits with the output. I know you find a more general solution there, which I liked, but I don't think that's necessary here. The output format is given by the time object and that's tested already, so we don't have to worry about it here. We just need to make sure the byte-by-byte description gives the correct column width; if the last few digits are different that's fine as long as the byte-by-byte and the table are consistent in every case. |
|
Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then. In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary. If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock. If you believe I commented on this pull request incorrectly, please report this here. |
|
I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks! If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here. |
Description
This Pull Request is part of the 2021 Google Summer of Code work I am doing together with mentors @aaryapatil and @hamogu.
It is a continuation of the modifications in #11897 which primarily aims to address issue #11257.
Note that the count appended at the end of the PR title, [GSoC21a], [GSoC21b] and so on, denote the sequence in which rebasing has been (will be) done to the commits. It is preferred if the merging also takes place in this order. Also, because the previous PRs are not merged yet, the later ones contain numerous unrelated commits at the time of opening the PR.
Checklist for package maintainer(s)
This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.
Extra CIlabel.no-changelog-entry-neededlabel.astropy-botcheck might be missing; do not let the green checkmark fool you.backport-X.Y.xlabel(s) before merge.Checklist for PR author
formatandscalein Byte-By-Byte description.