DEP: in support of greater forbearance in bumping optional dependencies, beyond SPEC 0

I'm seeing a tendency to drop support for any dependency as soon as SPEC 0 allows for it, wether mandatory or optional.

I would like to make a plea for hardening our policy beyond what adhering to SPEC 0 allows us to do, specifically about optional dependencies, also known as extras.

Recent examples of bumping optional dependencies early include

Aggressive upgrades of this sort may create problems for end users; because the dependencies are optional, we cannot provide any guarantee they are installed at the same time as astropy, using the proper extra specification (typically as, e.g., pip install 'astropy[recommended]'). Crucially, and in contrast with direct dependencies such as numpy, we cannot actually restrict what versions of, e.g., scipy or matplotlib are installed with astropy, save for the most disciplined part of our user base who

know about extras
how to use them
... and when to use them.

I would argue that this is probably a very small fraction of the population, but the point is, many, if not most of our users should not be expected to be knowledgeable in this area. Considering most scientific Python users still run long-lived Python environments (and who can blame them ? alternative workflows are only starting to emerge now), it is especially likely that someone might, for instance, install matplotlib 3.6, then astropy 7.2.0 (which, at the time of opening, is at a rc stage, and -optionally- requires matplotlib 3.8), and end up with a broken environment, even though we had everything needed to support this valid use case, but decided to remove the backward compatibility layer

To be clear, I am not pointing fingers at the people who on-boarded the aforementionned bumps. I mean to point to a systematic failure to recognize optional dependencies as a different kind of contract with our users: we should only be allowed to make a dependency optional, or to aggressively bumping our requirements on it, but not both.

The solution to this, is to

apply forbearance when bumping our optional dependencies (a backward compatibility layer is temporary code any way, and not necessarily a burden to carry around for a bit longer)
promote or demote the ones we need to bump to either hard requirements, or dev-only dependencies

Incidentally, I believes this clarifies an existing, but so far unclear incentive for #17623

An immediate action could be to revert #18164 and #18492 for the time being. Tentatively triaging this issue to the 7.2.0 milestone with this point in mind, but I expect the ticket not to be closed without a broader discussion and possibly with some policy change.

We haven't dropped support for dependencies just to make numbers go up. We support our dependencies for at least two years, like SPEC 0 requests, and we drop them when they start causing problems.

@astrofrog

Errr RC2 is already out and I expect that to be the last RC as I haven't seen any downstream report a blocker thus far. I don't see how we would halt a release for this issue at this point for v7.2.0. cc @astrofrog

Can you clarify what problems these dependencies caused ? I didn't find anything beyond "getting rid of compatibility code" and "avoiding warnings in CI" (btw, #18852). These are not problems that affect users, but the solution that was applied is costing them. I don't think that we should be externalising friction to users when it can easily be avoided.

I don't see how we would halt a release for this issue at this point for v7.2.0

I don't mean this to be a blocker. I just wanted to make sure it was at least considered before the final release was out.

Bumping matplotlib requirement was a prerequisite for dropping support for numpy 1.23. It was convenient to use SPEC 0 as guidance to avoid spending time figuring out how much exactly matplotlib should have been bumped.

h5py versions we dropped support for used numpy.product() which was deprecated in numpy 1.25 and removed in numpy 2.0, so they wouldn't even work with most major versions of numpy we support.

I suspect that in practice people either upgrade their packages or they don't. If they do they are not affected by any of this because they will not be running bleeding edge astropy together with ancient versions of other packages. If they don't then they will not be running bleeding edge astropy together with ancient versions of other packages.

I will quote a sentence from the SPEC 0 Motivation

Adoption of this SPEC will ensure a consistent support policy across packages, and reduce the need for individual projects to divise similar policies.

@eerovaher

I can see the case here, especially assuming we have an oldest dependency CI run with all optional dependencies that simply breaks for the cases that @eerovaher described (as I think happened).

I guess there some separate situations,

Minimum version update without any compatibility code removed -> I don't see any point (and I don't think we have been doing those...).
Version update with very minimal compatibility code removed -> I'd tend to accept it, but this is probably where the argument is.
Version update because our required dependencies no longer are compatible -> I think this has to be done (is enforced by CI?).
No version update but instead introducing minimal compatibility code -> seems a bad idea
No version update but lots of compatibility code -> no way, I think.

Bumping matplotlib requirement was a prerequisite for dropping support for numpy 1.23.

I may be missing something here, because the only reasonable interpretation I'm seeing now is that matplotlib 3.6.0 is fundamentally incompatible with numpy 1.24, which to the best of my knowledge, isn't the case.
Why was it a requirement ? #18164 appears to be motivated by "removing more compatibility code", which in my opinion isn't a goal worth pursuing in itself. Was there a greater motivation there ?

h5py versions we dropped support for used numpy.product() which was deprecated in numpy 1.25 and removed in numpy 2.0, so they wouldn't even work with most major versions of numpy we support.

I should know: I took care of updating h5py on this issue, but h5py 3.8 would still work with our oldest supported versions of numpy, so I can see it both ways. However, h5py is less of a concern to me since the bump didn't involve removing a backward compatibility layer, so I'm willing to let this one go;

we have an oldest dependency CI run with all optional dependencies that simply breaks

I'd like to make sure we have a common understanding of what "broken CI" means in this context, because I suspect you may be using a pretty liberal definition where any warning (because we promote it to an error) is considered "breaking". So, to be very clear: I don't think warnings, especially in the context of running with oldestdeps (#18853), are enough of a justification to drop working compatibility code. There are very easy, cheap to maintain, ways around it (first and foremost, selective warning filtering), that come at no cost for end users, that I think should be considered long before removing code.

I will quote a sentence from the SPEC 0 Motivation

SPEC 0 never mentions the difference between direct and transitive dependencies
I understand the motivation, even agree with it to some extent, but I don't think it's a valid argument to shut all further discussions on the ground that a previous agreement was supposed to make them unneeded. The current approach creates actual problems, we should at least be able to discuss amending it.

Bumping matplotlib requirement was a prerequisite for dropping support for numpy 1.23.

I may be missing something here, because the only reasonable interpretation I'm seeing now is that matplotlib 3.6.0 is fundamentally incompatible with numpy 1.24, which to the best of my knowledge, isn't the case. Why was it a requirement ?

There was some problem in the oldest dependencies CI job in #18160 that I solved by bumping dependencies. I can't look up what the problem was because the CI logs have expired and I didn't bother investigating it thoroughly at the time because SPEC 0 allowed bumping the requirements instead. The problem might not have been a fundamental incompatibility between matplotlib 3.6 and numpy 1.24, it might have been some problem in our compatibility code or test setup. If you want more details you will have to investigate yourself.

I will highlight that dropping support for old dependencies instead of investigating compatibility problems is not abusing some loophole in SPEC 0. Instead SPEC 0 is explicitly designed to permit that. The SPEC 0 Motivation section clearly states:

Limiting the scope of supported dependencies is an effective way for
packages to limit maintenance burden. Combinations of packages need
to be tested, which impacts also on continuous integration times and
infrastructure upkeep. Code itself also becomes more complicated when
it has to be aware of various combinations of configurations.

#18164 appears to be motivated by "removing more compatibility code", which in my opinion isn't a goal worth pursuing in itself. Was there a greater motivation there ?

The greater motivation was to not get stuck supporting numpy 1.23 forever.

...are enough of a justification to drop working compatibility code.

When I was working on #18160 I discovered that in some cases astropy was hiding numpy deprecation warnings from users (see b26362d), so clearly we did not have working compatibility code. Recently I discovered that with numpy 1.24 running

NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=0 pytest

causes errors during test collection even though there is compatibility code that serves no other purpose than to enable calling exactly the command above. These examples prove that in many cases our CI succeeds not because of working compatibility code but because of lack of test coverage. The problem of providing compatibility for many different dependency versions is clearly too complicated for us and to improve the situation we will have to simplify the problem by limiting the number of dependency versions we support.

If you want more details you will have to investigate yourself.

I feel like this should not be required: the context should be better documented. I'll open a revert PR just to see what it can tell me.
update: see #18971

I will highlight that dropping support for old dependencies instead of investigating compatibility problems is not abusing some loophole in SPEC 0. Instead SPEC 0 is explicitly designed to permit that.

again, SPEC 0 not ever addressing the distinction between mandatory and optional deps is the problem here. And now that you mention it, I do think it's a loophole. If this was mentioned explicitly I would reject the SPEC entirely, but it's not mentioned at all, even in passing, so I do feel it's abusive to assert that it was designed this way.

The greater motivation was to not get stuck supporting numpy 1.23 forever.

Who said anything about "forever" ? I do agree with you guys that compatibility code is meant to be temporary, but we can't debate seriously with straw man arguments.

These examples prove that in many cases our CI succeeds not because of working compatibility code but because of lack of test coverage. The problem of providing compatibility for many different dependency versions is clearly too complicated for us and to improve the situation we will have to simplify the problem by limiting the number of dependency versions we support.

Your examples are about numpy, which is a direct dependency to us. I intentionally and explicitly scoped this issue about bumping optional dependencies, so these examples, as valid as they are in that other context, are not relevant here.

Results are in: https://github.com/astropy/astropy/actions/runs/19512707755/job/55856342175?pr=18971:
I think we can safely ignore failures about assert_array_equal's strict argument, that's a clear artifact of my quick and dirty revert process.
Beyond that, I don't see anything unexpected in CI. So what am I missing ? any chance the tests that surfaced the problem were removed or updated in the mean time ?

@neutrinoceros

@neutrinoceros - I agree in principle that for optional dependencies there is no strong case for automatic updates of the minimum version, and I am really happy that you are trying to get us to a state where we have a well-defined CI run which is guaranteed to use our minimum dependencies, so that we know when something breaks. But I don't think any of us is arguing we just update their minimum versions as SPEC 0 allows.

The real question to me remains the third one in my list, and is related to required dependencies: whether, if we increase the minima for our required dependencies and find from CI that causes problems, are we free to just increase the minimum dependencies for the optional packages to make the problem go away? At that point there is a real cost to anything but an automatic update to the minimum that SPEC 0 would give (cost in time spent to find alternative lowest minimum or write a work-around).

p.s. The distinction between required and optional is blurry in astropy: we actually have required, recommended and optional. Many have argued that the "recommended" ones (scipy, matplotlib, narwhals) should become required dependencies since quite a lot of the code in astropy relies on them (and indeed they are installed by default with a conda install, and would be with pip too if pip had a mechanism to allow that being the default and still easily be able to install minimal).

p.s.2 I guess we may have to agree to disagree on another aspect: in my experience, compatibility code has a real cost, if only in any time I look at it, having to think why it is there, and whether I'm breaking something. But I'll readily admit this is mostly for numpy, since the code I tend to look at doesn't care about optional dependencies.

If you want more details you will have to investigate yourself.

I feel like this should not be required: the context should be better documented. I'll open a revert PR just to see what it can tell me. update: see #18971

The detailed logs have expired, but GitHub still retains the fact that the oldest dependencies CI job for 8aa1dd2 failed and none of the reviewers thought that #18164 was poorly motivated at the time.

I will highlight that dropping support for old dependencies instead of investigating compatibility problems is not abusing some loophole in SPEC 0. Instead SPEC 0 is explicitly designed to permit that.

again, SPEC 0 not ever addressing the distinction between mandatory and optional deps is the problem here. And now that you mention it, I do think it's a loophole. If this was mentioned explicitly I would reject the SPEC entirely, but it's not mentioned at all, even in passing, so I do feel it's abusive to assert that it was designed this way.

The problem is not really support for individual packages but support for combinations of packages, as SPEC 0 clearly states, and if there is an incompatibility between specific versions of two packages then the problem doesn't go away if you relabel an optional dependency as required or the other way around.

The greater motivation was to not get stuck supporting numpy 1.23 forever.

Who said anything about "forever" ? I do agree with you guys that compatibility code is meant to be temporary, but we can't debate seriously with straw man arguments.

It is true that I exaggerated when I used the word "forever".

These examples prove that in many cases our CI succeeds not because of working compatibility code but because of lack of test coverage. The problem of providing compatibility for many different dependency versions is clearly too complicated for us and to improve the situation we will have to simplify the problem by limiting the number of dependency versions we support.

Your examples are about numpy, which is a direct dependency to us. I intentionally and explicitly scoped this issue about bumping optional dependencies, so these examples, as valid as they are in that other context, are not relevant here.

I can't provide as detailed examples about compatibility code written for other packages not working because I have not bothered investigating them in such detail, but I have no reason to suspect that our numpy compatibility code is systematically less robust than the compatibility code for other packages, or that the test coverage for supported numpy versions is less complete.

Results are in: https://github.com/astropy/astropy/actions/runs/19512707755/job/55856342175?pr=18971: ... I don't see anything unexpected in CI. So what am I missing ?

You should have tested with numpy 1.24 not 1.23.

none of the reviewers thought that #18164 was poorly motivated at the time.

this is what I meant to shield off by

I am not pointing fingers at the people who on-boarded the aforementionned bumps. I mean to point to a systematic failure to recognize optional dependencies as a different kind of contract with our users

You should have tested with numpy 1.24 not 1.23.

I take your point, and I'm hoping that you are able to branch out at the correct point of your fork's history to run a more reliable experiment, because it seems I cannot; even with the commit still being publicly available from the GH UI, I don't seem to be able to check it out.

@mhvk

The real question to me remains the third one in my list, and is related to required dependencies: whether, if we increase the minima for our required dependencies and find from CI that causes problems, are we free to just increase the minimum dependencies for the optional packages to make the problem go away?

@mhvk here I think you are touching on an adjacent topic that I also opened a dedicated issue for: stealth upgrades. Quoting the litteral bottom line of this issue:

the correct course of action when a difference is detected would need to be decided on a case-by-case basis. This issue is purely about ensuring that human maintainers are made aware of these upgrades when they happen.

and to answer you question directly; it depends on the problem. Making dependency graph unresolvable or hitting actual runtime incompatibilities are absolutely good reasons to also bump optional deps, and deprecation warnings shouldn't be. Beyond these 3 classes of problem, more nuance may be needed on a case by case basis. Does this make sense ?

cost in time spent to find alternative lowest minimum or write a work-around

finding an alternative minimum should be free actually: resolving #18965 will provide an automated and accurate solution to this problem. Work arounds not so much, but then what good is it to have paid engineering time if we can't use it to solve problems that impact users ? Again, the gain VS risk of working on such workarounds should absolutely be subject to evaluation, and assessed on a case by case basis.

The distinction between required and optional is blurry in astropy: we actually have required, recommended and optional.

I'm going to take a radical stance here and say it doesn't matter:

I'm only talking about the technical distinction. recommended deps are still part of project.optional-dependencies
most users don't know what extras are and won't use them
even knowingly choosing not to use them is a valid use case in my book

I guess we may have to agree to disagree on another aspect: in my experience, compatibility code has a real cost, if only in any time I look at it, having to think why it is there, and whether I'm breaking something. But I'll readily admit this is mostly for numpy, since the code I tend to look at doesn't care about optional dependencies.

btw, I understand your position, and am frequently sharing the same feeling of discomfort when I need to work on code that has compatibility layers, but I'll take personal discomfort any day if disrupting users is the alternative. Any way, case in point, you are talking about numpy here, which isn't the focus of this discussion. I just wanted to share that I know how it feels.

@neutrinoceros

@neutrinoceros - I'm all for your automation process, and great if that can also help to find out how to bump to avoid conflicts. Ideally, it would even create the PR that does it!

About costs: so far, I would estimate the discussion here has taken between half and a full work-day between all participants, or around US$1000 at a standard rate of $150/hour. To me, it is not obvious this is best use of time (and money, if astropy funds are used for that). I think it behooves us to come up with a procedure that avoids similar costs whenever an upgrade is made. SPEC 0 provides one such path, and it has the advantage that it minimizes compatibility code.

But arguably the main issue for me remains that I'm really not sure there is an actual problem to be solved -- it is not like we're getting issues raised about incompatibilities.

Anyway, I think I'm just repeating myself, so will stop. I think we'd need a concrete proposal of an alternative to SPEC 0 that makes the case why this is important and has concrete, minimal work guidelines.

I can formalize this out into an APE. Will wait to know the fate of my Cycle 5 FR first, though.

I'm all for your automation process, and great if that can also help to find out how to bump to avoid conflicts. Ideally, it would even create the PR that does it!

Do note that automating a PR pipeline is both much harder and less reliable. My proposal is to inform humans about changes in resolved environments, not abstract requirements. Like most information destructive processes, going from the former to the latter is much easier than the way back.

This topic is always controversial and I think have resurfaced from time to time in various forms. We have to keep in mind that:

the underlying problem of Python Dependency Hell ™️ is not something astropy itself can solve, and
the pip stuff is not our only problem; we also have to deal with dependency limitations in conda, Debian, and whatever distros out there that decides to distribute our package, AND we cannot control how they choose to manage our dependencies.

Other topics that might be related (just those I can recall, so if I missed something, feel free to add):

All in all, I think there is a general agreement that we would not drop support for an older version just to bump the number (besides those specifically laid out in APE 18). So I propose that going forward, we should be mindful of this but deal with each bump on a case by case basis. Due to the sheer number of variety of situations that could arise to make us think of bumping a minversion, I don't think it is reasonable to generalize all of them into a single piece of document (that people would bother reading).

If you insist on formalizing it, then a companion APE to APE 18 (as mentioned above) would be the way to go, but please do not do it before consulting with astropy-dev and/or Astropy Coordination Committee.

Thank you, all!

I am removing the milestone because I don't think this can be resolved any time soon and is not a release blocker.

Navigation Menu

Repository navigation

DEP: in support of greater forbearance in bumping optional dependencies, beyond SPEC 0 #18967

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DEP: in support of greater forbearance in bumping optional dependencies, beyond SPEC 0 #18967

Description

Activity

eerovaher commented on Nov 18, 2025

pllim commented on Nov 18, 2025

neutrinoceros commented on Nov 18, 2025

neutrinoceros commented on Nov 18, 2025

eerovaher commented on Nov 18, 2025

mhvk commented on Nov 19, 2025

neutrinoceros commented on Nov 19, 2025

eerovaher commented on Nov 19, 2025

neutrinoceros commented on Nov 19, 2025

neutrinoceros commented on Nov 19, 2025

mhvk commented on Nov 19, 2025

eerovaher commented on Nov 19, 2025

neutrinoceros commented on Nov 20, 2025

neutrinoceros commented on Nov 20, 2025

neutrinoceros commented on Nov 20, 2025

mhvk commented on Nov 20, 2025

neutrinoceros commented on Nov 20, 2025

neutrinoceros commented on Nov 20, 2025

pllim commented on Nov 24, 2025

pllim commented on Nov 24, 2025

pllim commented on Nov 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions