TST: replace mpl380 tox factor with oldestdeps and solidify reproducibility of image tests - #19115

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

I think we are maybe not talking about the same thing. Each figure test env uploads figures (in a folder named for the tox env) here: https://github.com/astropy/astropy-figure-tests/tree/astropy-main/figures at the moment the old envs stay around even after we stop updating them, but if they are all named oldestdeps then it would overwrite the figures when they are regenerated with a newer mpl version.

oh, I get it now. Isn't this repo used as a submodule here ? if so, it should trivially be pinned to a git commit hash, which would make this problem moot. Alternatively, we could also add git tags there ?

I don't think it is a Git submodule and I do not wish to reintroduce that after we managed to get rid of astropy-helpers

I think this is fine as long as we accept that bumping oldestdeps will probably mean a full figure regeneration.

This does seem okay to me too. Overall I don't see anything blocking here, so can we move forward with it ?

full figure regeneration

Let me try to understand, so we will need a lot more PRs like this?

• TST: Update mpldev hashes #19195

(rebased to resolve a merge conflict)

friendly reminder @Cadair

I have a concern related to what @Cadair raised above. Let's say we release astropy 8.0 which has a certain set of versions for oldestdeps. Now let's suppose that in main we bump the minimum version of matplotlib. At this point the version of matplotlib used for oldestdeps will have diverged between the release branch and main, which will cause issues when running tests on the release branch.

The only way I can think of to get around this if we want to avoid having to rename CI jobs is that when we construct the reference image URL, we inject the matplotlib version into it (like we used to).

I couldn't find how it used to be done in the repo history. Any pointers ?

Something like this:

https://github.com/astropy/astropy/blob/v3.0.x/astropy/tests/image_tests.py

Just wondering, is this really worth it though? We only bump the minimum MPL version once or twice a year, and not sure if it's worth over-engineering this?

We only bump the minimum MPL version once or twice a year

right, but this PR isn't really about removing this (admittedly marginal) maintenance burden: the goal is rather to eliminate a failure mode, which the current state of tox.ini is a testimony to (we have a redundant constraint on pyparsing). Currently we have no control over the frequency of this failure, and this PR would set an upper bound to the frequency at which we actively bump our minimal requirement for matplotlib.

and not sure if it's worth over-engineering this?

One could argue the existing setup is over-engineered already. I'd like to give a shot at figuring out a simpler solution that eliminates the defect you and Stuart pointed out.

I think we could probably retain the existing env names and remove metadata duplication by using recent tox features like extend=true to define mpl380 (https://tox.wiki/en/stable/config.html#configuration-reference)... will dig in some more.

I think I'll need to migrate tox.ini to tox.toml to benefit from these newest features, but this is temporarily blocked by scientific-python/cookie#739

apparently tox configuration is in a weird state right now: tox.toml is the current recommended configuration file, but does not support all features from tox.ini, including conditional env factors, so I think we cannot actually migrate yet. I'm now hoping I was wrong earlier when I concluded we needed to use this format to leverage new features, otherwise it would mean that tox.toml's feature sets is not even a subset of tox.ini's.

Apparently this "design" was actually intentional:
https://github.com/tox-dev/tox/pull/3353/changes#r1784021175

I'm at a loss for words.

tox.toml now officially supports everything that tox.ini did (though migration might not be trivial), and will be the only target for new features. I'll open a separate issue to track this.

Hi humans 👋 - this pull request hasn't had any new commits for approximately 5 months. I plan to close this in a month if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.