Skip to content

MRT metadata input and subsequent writing [GSoC21c] - #12039

Closed
Suyog7130 wants to merge 52 commits into
astropy:mainfrom
Suyog7130:mrt-metadata
Closed

Suyog7130 wants to merge 52 commits into
astropy:mainfrom
Suyog7130:mrt-metadata

Conversation

@Suyog7130

Copy link
Copy Markdown
Contributor

Description

This Pull Request is part of the 2021 Google Summer of Code work I am doing together with mentors @aaryapatil and @hamogu.
It is a continuation of the modifications in #11897 and #12027 which primarily aims to address issue #11257.

Note that the count appended at the end of the PR title, [GSoC21a], [GSoC21b] and so on, denote the sequence in which rebasing has been (will be) done to the commits. It is preferred if the merging also takes place in this order. Also, because the previous PRs are not merged yet, the later ones contain numerous unrelated commits at the time of opening the PR.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see "When to rebase and squash commits".
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the Extra CI label.
  • Is a change log needed? If yes, did the change log check pass? If no, add the no-changelog-entry-needed label.
  • Is a milestone set? Milestone must be set but astropy-bot check might be missing; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate backport-X.Y.x label(s) before merge.

Suyog7130 added 30 commits June 27, 2021 23:21
these together write the CDS Data Table without the ReadMe header.
[skip ci]
this is done via `test_write_data` func in `test_cds.py`
[skip ci]
including `test_write_ByteByByte_units`
[skip ci]
Also got the `cdsdict` argument passing working. The user can pass it using `ascii.write(Writer=ascii.Cds, cdsdict={})` or `table.write(format='cds', cdsdict={})`.
[skip ci]
and for ByteByByte description of columns with Null values
[skip ci]
also edited last line of ReadMe template
[skip ci]
Correct expression for `nsplit`.
Add `test_write_byte_by_byte_bytes_col_format` and `test_write_byte_by_byte_wrapping` functions.
[skip ci]
correctly place `# noqa: W291` at end of every closing triple quotes.
This works fine now, so no need for `# flake8: noqa: W291` to not checking codestyle of whole file.

reduce output lenght of `test_bbbWrap` func and other corrections.

add MRT_TEMPLATE

[skip ci]
add a diff second col in `coord_cols` test output
copy table before writing to CDS format
[skip ci]
add exception for column with partial `SkyCoord` values.
add `else` statement for `SkyCoord` columns not in `SphericalRepresenation`

convert other `mixin` and partial `SkyCoord` col vals to str
and have column label for them as `Unknown`

add test `test_write_mixin_and_broken_cols`

[skip ci]
add `test_write_extra_SkyCoord_cols` test
delete ReadMe template file
[skip ci]
put all imports in `test_cds.py` to file start
make `SkyCoord` import local

use only `if not isinstance(col, Column)`

checking values of `col.info.format` and `col.format`
in the cases when no `formats` argument is passed.

[skip ci]
BBB was not correctly taking the format passed in the `formats` argument.

minor change in the format issuing part

some extra useless trailing zeros are appended sometimes
to the table column float values, which is why `col.meta.size` being calculated is sometimes different from what is obtained from `len(max(col.str_vals))`

edit exception in `_split_float_format`

[skip ci]
edit coord tests to reflect default format
[skip ci]
add `test_write_mrt_metadata_notes_for_mixin_cols` test
add `test_write_mrt_metadata_repeated_notes` test
add `test_write_mrt_metadata_wrapping` test
add `test_write_mrt_metadata_global_notes` test
add `test_write_mrt_metadata` test
@Suyog7130

Suyog7130 commented Aug 10, 2021 •

Copy link
Copy Markdown
Contributor Author

This PR is the work done to add getting MRT metadata from the user for CDS/MRT format table writing.
Changes pertaining to this PR start after the commit titled add changelog for #12027

@Suyog7130

Suyog7130 commented Aug 10, 2021 •

Copy link
Copy Markdown
Contributor Author

Brief summary and some comments:

  • Global notes can be passed using a direct notes keyword for the Cds class.
  • If a SkyCoord column has any notes, then all its component columns after extraction should have the same note with the same note number appended in the column description.
  • Same note should not be repeated for two columns, instead all the columns having same note should reference the same note number.
  • Global notes should be at the end of the notes section in the ReadMe. Column description should not reference the Global notes!
  • Add feature to add authors from a list of authors.
  • Should the author name be sorted according to First/Last name, like it is done in BibTex?
    Na, I don't think this is required. Too much extra work for what can be a not too useful feature.
  • What if the reference column for repeated notes is a column with mix-in values? Since the original column is not used then, will col.meta.notes or col.note_ref attributes be available?
    This works! It should since the column with mix-in values will have to be a Column object in order to have meta.notes attribute anyway. If this is the case, then these columns are almost the same as other columns.
    • I can prob write about this in the docs!
  • Edit the docstring of Cds class to list out the metadata keywords.
  • Have used Note (G) for global notes in the ReadMe, since CDS decrees something similar. MRT doesn't have any particular opinion about this, so CDS must be followed for these optional attributes.
  • Global notes are important specially for the cases when a single file has numerous Byte-By-Bytes. Like given here: https://cdsarc.unistra.fr/viz-bin/ReadMe/IX/22?format=html&tex=true
    But our case is only for a single table, thus containing a single Byte-By-Byte in the ReadMe.
    • Maybe I should write about this in the documentation too.

@Suyog7130

Suyog7130 commented Aug 10, 2021 •

Copy link
Copy Markdown
Contributor Author

Have done a clever trick of using the column name in col.meta.notes to inform that any particular column's note will be the same as that of another column. Voila! A col.note_ref attribute is used internally for this purpose.

The above would note :) work when the reference is made before the notes for the previous column have been defined.
So, it should be clear in the documentation that when one wants to repeat a note for multiple columns, the first such column should the note defined, while the following ones referencing it.

Else, the notes will be repeated in the ReadMe even if the col.meta.notes are same.
Also, catching the AttributeError in these cases, doesn't automatically repeat the note. It puts the name of the reference col in the notes instead.

In other words, Reference to the same column note can be assigned by passing the name of the first column whose note has to be repeated, to the col.meta.notes attribute.
An error occurs while assigning note_ref if the reference column itself doesn't have a note_ref assigned yet. This can occur, for instance, when the reference column is further down in the Table. The Notes will be repeated for such cases, with the note reference number also being different.

@Suyog7130

Suyog7130 commented Aug 10, 2021 •

Copy link
Copy Markdown
Contributor Author

Notes and notes referencing doesn't work for mix-in columns because they are not Column objects and thus do not have the meta attribute. So, col.meta.notes cannot be assigned. They can be converted to Column objects beforehand by the user, if notes for them are required. The Notes should work then for columns that are Column objects but have coord values, after splitting into the component cols is done.
However, the writing works for tables that have notes and have mix-in columns as well.

@Suyog7130

Copy link
Copy Markdown
Contributor Author

A further point about note-taking in MRT/CDS,

@github-actions github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Jan 12, 2022
@github-actions

Copy link
Copy Markdown
Contributor

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

@hamogu

hamogu commented Jan 19, 2022

Copy link
Copy Markdown
Member

@Suyog7130 Do you think you'll have time to continue this (and the other PRs)? If now, should @aaryapatil and I see if we can take it over? We'd love you to be a regular contributor to astropy, but it would also be bad to see this PR just closed and the work that already went into it forgotten. We've gone so far already with the first PRs you did, I'd love to see these finished, too.

@github-actions

Copy link
Copy Markdown
Contributor

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.

@github-actions github-actions Bot added the closed-by-bot Closed by stale bot label Feb 12, 2022
@github-actions github-actions Bot closed this Feb 12, 2022
@hamogu

hamogu commented Feb 13, 2022

Copy link
Copy Markdown
Member

@Suyog7130 Shoudl we re-open this?

@Suyog7130

Copy link
Copy Markdown
Contributor Author

Yes, that would be great. I will push new commits to the PR soon.

@hamogu hamogu reopened this Feb 13, 2022
@pep8speaks

Copy link
Copy Markdown

Hello @Suyog7130 👋! It looks like you've made some changes in your pull request, so I've checked the code again for style.

Line 387:36: E201 whitespace after '('
Line 387:69: E202 whitespace before ')'
Line 424:41: E201 whitespace after '('
Line 424:78: E202 whitespace before ')'
Line 694:42: E201 whitespace after '('
Line 694:66: E202 whitespace before ')'
Line 751:43: E201 whitespace after '('
Line 751:59: E202 whitespace before ')'

@pllim pllim removed Close? Tell stale bot that this issue/PR is stale closed-by-bot Closed by stale bot labels Feb 14, 2022
@github-actions github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Feb 14, 2022
@github-actions

Copy link
Copy Markdown
Contributor

Hi humans 👋 - this pull request hasn't had any new commits for approximately 6 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

@github-actions github-actions Bot added the closed-by-bot Closed by stale bot label Mar 18, 2022
@github-actions

Copy link
Copy Markdown
Contributor

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Close? Tell stale bot that this issue/PR is stale closed-by-bot Closed by stale bot io.ascii unified-io

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants