Skip to content

TST: improve ArrayWrapper roundtrips (preserve dtype byteorder) - #18139

Merged
dhomeier merged 3 commits into
astropy:mainfrom
neutrinoceros:table/tst/18127/improve-arraywrapper-roundtrips
May 16, 2025
Merged

dhomeier merged 3 commits into
astropy:mainfrom
neutrinoceros:table/tst/18127/improve-arraywrapper-roundtrips

Conversation

@neutrinoceros

@neutrinoceros neutrinoceros commented May 13, 2025 •

Copy link
Copy Markdown
Contributor

Description

Fixes #18127. The primary goal was to fix compatibility of a test with CPython 3.14, but in discussing the root issues we found ways to simplify the test for every supported versions instead.

I'm assuming that astropy.table.table_helper isn't public API, in which case this can be treated as a dev-only bugfix to the test suite and be backported.

follow up to #11545

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@github-actions github-actions Bot added the table label May 13, 2025
@github-actions

Copy link
Copy Markdown
Contributor

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

@neutrinoceros
neutrinoceros marked this pull request as ready for review May 13, 2025 11:22

@dhomeier dhomeier left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fix. I had not even looked into why init_from_class was in the non-byteorder-preserving group, but having that fixed as well is of course yet better!

I have not found any pickle wrappers within our code, so this can indeed be considered a testing-only issue – there is one fixture relevant in

@pytest.fixture(params=[0, 1, -1])

so it might not hurt to extend this to params=[0, 1, 4, -1].

Still could be useful to note the different protocols in the docs, though again the only mention of pickle is in cosmology.io. Could suggest there to use pickle.dump(Planck18, file, protocol=5), although the format is most definitely not recommended for exchanging files between different machines, much less different architectures – @nstarman what do you think?

Comment thread astropy/table/tests/test_mixin.py Resolved
@dhomeier dhomeier added the Bug label May 13, 2025
@nstarman

Copy link
Copy Markdown
Member

I think that's a good idea.
https://docs.astropy.org/en/stable/cosmology/io/details.html#cosmology-in-a-pickle says that pickle is not safe and should mostly be used for temporary storage, e.g. in multiprocessing.

@dhomeier

Copy link
Copy Markdown
Contributor

The more reason perhaps to avoid any byteswapping where possible, although I have not benchmarked the two protocols

@neutrinoceros
neutrinoceros force-pushed the table/tst/18127/improve-arraywrapper-roundtrips branch from 1134ec8 to c394946 Compare May 13, 2025 16:38
@neutrinoceros
neutrinoceros requested a review from a team as a code owner May 13, 2025 16:38
@neutrinoceros
neutrinoceros requested a review from nstarman May 13, 2025 16:38
@neutrinoceros

neutrinoceros commented May 13, 2025 •

Copy link
Copy Markdown
Contributor Author

Thank ! I updated the PR with your suggestions.
Let me know if you'd like these commits split to a separate PR, anyone.

@github-project-automation github-project-automation Bot moved this from Triage to Reviewer approved in Cosmology, the Expansion May 13, 2025

@taldcroft taldcroft left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me.

@pllim

pllim commented May 14, 2025

Copy link
Copy Markdown
Member

Since this is marked as a bug and is going to land on v7.1.0 if backported, it should have a change log.

@neutrinoceros

Copy link
Copy Markdown
Contributor Author

@dhomeier what do you think ? Do you expect anyone is using this internal API downstream ?

@dhomeier dhomeier left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really know how "private" the test helper functions are considered, but in any case the "most advanced" protocol included there has already changed once (4->5 with Python 3.8) since this fixture was introduced in #1947, so downstream users already should have noticed (or never cared; I was hardly aware there are different pickle protocols at all). I think this deserves a bit more extensive documentation and should indeed get a changelog entry as well.

Comment thread astropy/tests/helper.py Resolved
Comment thread docs/cosmology/io/details.rst Resolved
@dhomeier dhomeier added the Extra CI Run cron CI as part of PR label May 15, 2025

@dhomeier dhomeier left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the formatting help; looks good now, but I think this deserves a test on the big-endian machine!

@pllim pllim modified the milestones: v7.1.1, v7.1.0 May 15, 2025
Comment thread astropy/tests/helper.py Resolved
@neutrinoceros
neutrinoceros force-pushed the table/tst/18127/improve-arraywrapper-roundtrips branch from e491aad to b60eb34 Compare May 15, 2025 16:36
Comment thread astropy/tests/helper.py Outdated
def pickle_protocol(request):
"""
Fixture to run all the tests for protocols 0 and 1, and -1 (most advanced).
Fixture to run all the tests for protocols 0, 1, 4 and 5 (most advanced).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since "5" is already mentioned in bullet point below under "-1", I think we should change this back to "-1" to be less confusing.

Suggested change
Fixture to run all the tests for protocols 0, 1, 4 and 5 (most advanced).
Fixture to run all the tests for protocols 0, 1, 4 and -1 (most advanced).

@neutrinoceros
neutrinoceros force-pushed the table/tst/18127/improve-arraywrapper-roundtrips branch from b60eb34 to 3c7acf8 Compare May 15, 2025 16:42

@pllim pllim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now. Thanks! @dhomeier , can this be merged when CI passes?

@dhomeier
dhomeier merged commit f442813 into astropy:main May 16, 2025
@github-project-automation github-project-automation Bot moved this from Reviewer approved to Done in Cosmology, the Expansion May 16, 2025
meeseeksmachine pushed a commit to meeseeksmachine/astropy that referenced this pull request May 16, 2025
@dhomeier

Copy link
Copy Markdown
Contributor

Thanks all; should have referred to "-1" in the bullet point right away!

@neutrinoceros
neutrinoceros deleted the table/tst/18127/improve-arraywrapper-roundtrips branch May 16, 2025 13:37
pllim added a commit that referenced this pull request May 16, 2025
@neutrinoceros neutrinoceros modified the milestones: v7.1.0, v7.1.1 Jul 25, 2025
@neutrinoceros

Copy link
Copy Markdown
Contributor Author

I found that this wasn't actually included in the 7.1.0 release, so, updating the milestone now.

neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Jul 25, 2025
…undtrips (preserve dtype byteorder))
neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Jul 25, 2025
…undtrips (preserve dtype byteorder))
@neutrinoceros

Copy link
Copy Markdown
Contributor Author

Ah, actually this was partially included in 7.1.0. Specifically, one in three commits were auto backported as #18156
I don't remember seeing this happen before. In any case, the other two commits are backported in #18461

pllim added a commit that referenced this pull request Jul 25, 2025
Backport PR #18139 on `v7.1.x` (TST: improve `ArrayWrapper` roundtrips (preserve dtype byteorder))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

TST: a failing table test (test_info_preserved_pickle_copy_init) on CPython 3.14

5 participants