Make scipy a required dependency

I would like to suggest making scipy a required dependency of the core package (or some alternative approaches mentioned below).

There are a number of sub-packages in astropy where we use scipy where it's not obvious that would be the case and where users will encounter unexpected ImportErrors if it is not installed. For instance the following code snippets from the docs require scipy:

astropy.coordinates

>>> c = SkyCoord(ra=ra1*u.degree, dec=dec1*u.degree, distance=distance1*u.kpc)
>>> catalog = SkyCoord(ra=ra2*u.degree, dec=dec2*u.degree, distance=distance2*u.kpc)
>>> idx, d2d, d3d = c.match_to_catalog_3d(catalog)  # REQUIRES SCIPY

astropy.cosmology

>>> from astropy.cosmology import WMAP9 as cosmo
>>> cosmo.H(0)
<Quantity 69.32 km / (Mpc s)>
>>> cosmo.kpc_proper_per_arcmin(3)  # REQUIRES SCIPY
<Quantity 472.97709620405266 kpc / arcmin>

astropy.units

>>> from astropy.cosmology import WMAP9
>>> z = 1100 * cu.redshift
>>> z.to(u.K, cu.with_redshift(WMAP9))
<Quantity 3000.225 K>
>>> z.to(u.Mpc, cu.with_redshift(WMAP9, distance="luminosity"))   # REQUIRES SCIPY
<Quantity 15418438.76317008 Mpc>

astropy.convolution

>>> gauss_oversample = Gaussian1DKernel(3, mode='oversample', factor=10)
>>> gauss_integrate = Gaussian1DKernel(3, mode='integrate')   # REQUIRES SCIPY

astropy.modeling

scipy is also used extensively in modeling!

In all of these examples, there is no way the user would know in advance that scipy is required, and they will stumble on these issues and have to install scipy to get them to work. Now that scipy is needed for some of the functionality in at least five sub-packages, I think it is time to consider requiring it. In total, from scipy. appears 76 times in the astropy core package, and doctest-requires:: scipy appears 36 times!

As packaging has improved tremendously since the start of the project, and wheels and conda packages are available for scipy for many platforms, I think it would be reasonable to start requiring scipy and would result in fewer surprise errors for users.

The philosophy here is the same as why we added PyYAML as a required dependency - in many cases users couldn't have suspected this dependency.

To give a counter example of something I think doesn't need to be a required dependency, astropy.visualization provides lots of functionality to expand matplotlib, but in many cases matplotlib needs to be installed anyway for the user to want to use the functionality. Take a look at the following example:

    from astropy import units as u
    from astropy.visualization import quantity_support
    quantity_support()
    from matplotlib import pyplot as plt
    plt.figure(figsize=(5,3))
    plt.plot([1, 2, 3] * u.m)

Here matplotlib is imported explicitly anyway, and a user wouldn't be trying to use quantity_support without it installed.

Another counter-example would be the table reader/writer for HDF5 - in this case it's ok to error if h5py is not installed with an explicit error that says 'h5py is required to read/write HDF5' files.

Perhaps this requires an APE or a policy document describing some basic rules/tests we can use to determine if a dependency should be required.

Alternatives

I suspect there will be resistance to including scipy as a required dependency given that it may not be easy to install on some more exotic platforms. So here are some more options:

The option mentioned above, add scipy to install_requires
In a lot of cases we might only be using a very limited subset of scipy, eg integration. We could try and replace scipy with another lighter weight dependency in these cases and require that.
We could make scipy a required dependency only on the main common platforms where wheels are available (this can be done in the requirements with environment markers) and make sure that the test suite still passes without scipy installed
We could require scipy as in option 1. but document how users can install astropy without it (--no-deps and explicitly specifying our minimal required dependencies) and continue to include some test builds without scipy
We could make it so that sub-packages don't import at all if scipy isn't installed - for example cosmology uses scipy enough that we could just disable importing the whole subpackage if scipy is not installed (rather than giving surprise errors for half the methods).

It would be very convenient if scipy were a runtime dependency.
From what I understand, the major reason why it hasn't already been made one is the issue of installation on various systems. Perhaps it could be tabulated what systems astropy runs on versus where scipy can be installed...

Update: Okay, I traced the scipy build problem to cibuildwheels automatically trying to build for musllinux. Disabling that as follows fixed the problem. So I am neutral again.

CIBW_SKIP: '*-musllinux_*'

I think the case for making it an install_requires dependency is convincing enough, but I don't work on systems where scipy is difficult to install. Before we presuppose pushback, maybe it's worth sending a question to astropy-dev to see if people in those situations anticipate real issues?

Posted to astropy-dev: https://groups.google.com/g/astropy-dev/c/T5gskNMJpx8

Scipy adds 85 MB to the installation while astropy itself is 19 MB and numpy is 28 MB (well, the debian packages are). That does feel like the tail wagging the dog! So, I'm not too keen.

By the way, units does not require scipy - the example above uses cosmology... But at some level that proves the point - it is not obvious from the example that scipy would be needed! But anyway, I'd say that cosmology, convolution and modeling are the ones where scipy is truly needed.

Note that only a little while ago, I added a "recommended" category in setup.cfg, and updated our installation instructions to suggest one do

pip install astropy[recommended]

Unfortunately, I don't know if there is a similarly easy way to get a more minimal installation if we were to add scipy to the installation requirements.

One possible approach is to have a set of 'core' dependencies, and then a set of required dependencies per sub-module. So e.g. pip install astropy[cosmology] or pip install astropy[convolution] would install scipy, but pip install astropy wouldn't.

@dstansby

@dstansby - that sounds dangerously like something we have at times considered, which is to package astropy in parts that somewhat respect the internal hierarchy, i.e., pip install astropy-cosmology which pulls in astropy-core as well as scipy (and where astropy-core in turn may depend on astropy-units, as units & constants basically do not depend on anything else in astropy). But obviously very unclear if this is worth the likely considerable effort!

Scipy adds 85 MB to the installation while astropy itself is 19 MB and numpy is 28 MB (well, the debian packages are). That does feel like the tail wagging the dog! So, I'm not too keen.

That was one of my first 3 thoughts on this issue as well. Installed size I get on macOS is 70 / 35 / 31 MB, so that's not quite as bad.
The other 2 were, to me it seems so weird not using any Scipy functionality it may as well be always installed;
then I felt it would expand the dependency chain quite a bit beyond our original Numpy-only situation.
But actually there are no other Python runtime dependencies, and the only potential heavy-weight requirements are probably a BLAS, which is already required by Numpy, and Suitesparse.
OTOH there are certainly still many users out there who are using Astropy chiefly for io.fits or similar, so for them doubling effective install size of an already relatively large package is not negligible.
At least we may probably trust that there is some awareness of install size issue on the Scipy side as well – scipy/PROPACK#2.

Debian doesn't bundle our compiled libraries (like libcfitsio and libwcs), which may explain part of the difference. It is good that scipy is thinking about how to reduce its footprint. Although it doesn't seem test files at least in Debian, but rather just cython compiled libraries... Just to be sure, I checked and at least the Debian installation of astropy is pretty good: the largest file by far is the IERS table.

Anyway, this is getting a bit off-topic. Apologies!

@dstansby

@dstansby - that sounds dangerously like something we have at times considered, which is to package astropy in parts that somewhat respect the internal hierarchy,

I rather understood the suggestion as labels for various bundles, i.e. astropy[cosmology], astropy[convolution] or astropy[modeling] would just include the additional dependencies for the respective sub-packages, so astropy[recommended] would become something like a superset of them all.
Splitting the core package itself into partial installations also sounds like a dangerous can of worms to me.

@astrofrog

I think adding scipy as a hard dependency is OK at this point. I have not seen a system in years where an extra < 100 MB install would be a problem. I install scipy into my environments for almost any project anyway, because sooner or later, I always need the interpolation or integration or one of the special functions from scipy.

I think I'm a fairly typical user (laptop or linux dektop with sufficient disk space, using conda environments, no HPC or exotic systems) and to me it seems that in @astrofrog list only two options are worthwhile:

keeps a is now
make scipy a full dependency

Anything in between strikes me a a lot of extra effort and mental overhead especially for casual developers (scipy is required, but we still need to make sure tests pass without?) for limited gain.

Note: astropy.stats.poisson_conf_interval(... interval=‘kraft-burrows-nousek’) has a dependency of either scipy or mpmath, where mpmath is actually preferred because of the higher precision you can get. So, that is an example where "rewrite to use a light-weight library instead of scipy" is in place already; but unfortunately, mpmath won't do all the things we need scipy for.

units is just when used in combination with cosmology example, so it would (and should) be caught on import of cosmology. coordinates has only the matching. I think only cosmology and modeling really rely in scipy, with convolution a more half-way example.

p.s. Really the problem is that pip is a really lousy package manager, where it is not possible to have a minimal install that removes some of the dependencies. Now it does seem the whole packaging is being rethought; maybe we should just wait on that.

Even modeling is not sold hard on scipy; > 60 % of its tests are still running with not HAS_SCIPY.
That may not be very helpful though in the scheme of only disabling certain modules – it's probably not practical to have the import fail selectively even below the submodule level, like on astropy.modeling.optimizers.

I don't think that failing import of entire sub-packages is useful. The current fine-grained approach of disabling specific functionality and raising an exception that clearly states that scipy needs to be installed seems reasonable and not confusing.

I think in practice the vast majority of astropy users end up having scipy installed anyway so the package size probably doesn't matter for a typical user. But could there be niche users on a severely disk-limited systems that are impacted? Generally speaking of course 85 Mb is nothing these days.

I'm also curious about any impact on CI since that means many more downloads of the large scipy package. On the flip side, if a lot of tests require scipy and are not done in the normal test matrix (for current req'd deps) then we are potentially missing bugs.

The benefit of having scipy as a required dependency would also be to simplify the code, no more need to check if scipy is available, no HAS_SCIPY, no conditional tests. Having a way to recommend some dependency does not solve that since the import should still be optional in the code and tests.

So maybe something we could do is to make scipy required by some subpackage, e.g. modeling. When importing astropy.modeling it would raise a nice and informative error if scipy is missing, and then we could remove the checks in the submodules and tests. (And even if many modeling tests currently run without scipy, the subpackage itself is pretty useless without scipy).

On other hand, I guess astropy is used (mostly for io.fits ?) by observatories on some low level workstations and/or archive servers where admins would prefer to keep the number of dependencies small ?

@saimn

@saimn - indeed, I was suggesting the error on importing modules that use scipy heavily so that we can get the code simplification without having to add the requirement for all of astropy. One could use the same for visualization without matplotlib.

Note that at the end on the long thread, there had been convergence on how to enable indicating smaller subsets in pip, and the suggestion to write a PEP -- though in December the main author noted that he would have no time and preferred to be a co-author -- https://discuss.python.org/t/adding-a-default-extra-require-environment/4898/138

@taldcroft

Related: I just got a message from someone at RIKEN who asked whether it was allowed to pull astropy.units out to avoid the cost of all the rest of astropy. (Of course, it is, and I'll write to say so, noting that it means giving up on bug fixes, etc.) Maybe this is an example of users already finding that too much is bundled -- right now of course we do not hear from those who have reasons for a minimal system and would be like to use just quantities or tables+io or just the actual basic astronomy stuff, Time and SkyCoord.

Anyway, I guess there are 3 options:

Keep things as they are;
Require scipy;
Raise for some modules if scipy is not present.

I think in the end it really depends on the problem we're trying to solve. I don't think we are making things particularly difficult for users, since, as @taldcroft noted, most have scipy installed anyway. And it would seem both that, and complications inside modules, would be solved just as well by (3) as by (1) - and (3) would mean no changes needed if pip ever allowed one to remove dependencies from the default list.

p.s. I think the by-type-of-machine route is more likely to lead to confusion, but perhaps a fourth option is to install scipy automatically on mac and windows, since linux users (including CI) can be assumed to need less hand-holding (maybe include conda for automatic install too).

I just got a message from someone at RIKEN who asked whether it was allowed to pull astropy.units out to avoid the cost of all the rest of astropy.

Anecdotally, I've heard the opposite sentiment from people (mostly students and postdocs), who have said that they love that Astropy comes out of the box with so much of the functionality they need. If we asked about this to a broad user group, I suspect we would find users at all points of the spectrum from "split out all subpackages to let us install minimally what we need" to "merge all affiliated packages into astropy main so I don't have to install 50 different packages to do what I want". I'm not sure who is thinking about a user survey these days, but maybe this would be a good thing to ask about?

I think the by-type-of-machine route is more likely to lead to confusion, but perhaps a fourth option is to install scipy automatically on mac and windows, since linux users (including CI) can be assumed to need less hand-holding

If the goal is to simplify code within Astropy, I don't think the solution should be to add conditional installs! So I don't like this option.

FWIW, as a user, if I installed a package and several of the major subpackages errored on import because of a missing dependency, I would be annoyed that the maintainers didn't add that dependency to the required installs!

Note that we do have pip install astropy[recommended] or pip install astropy[all] for those who need everything. Perhaps that should just be more prominent (including in the import errors)? Similarly, before making astropy into a new iraf, we could have pip install astropy-all that also pulls in all the affiliated packages; absolutely nothing stopping someone from creating that!

p.s. As I wrote before, I'd be all for making scipy a requirement if pip install astropy[minimal] actually existed; we're having a very long discussion mostly because python's package system is bad. Something that will likely be fixed soon enough, since it is even worse for packages that link to compiled libraries.

p.s. Splitting out astropy.units has come up multiple times; e.g., #10606
It certainly isn't particularly astronomy specific.

@mhvk

I agree with @mhvk here on not requiring scipy. I think there are very different groups of people with very different requirements. A typical user might want everything already there with a single pip call, but if you are deploying stuff using astropy on the cloud, storage is precious and doubling the install size would be bad.

ref: #12448 (comment)

@astrofrog

@astrofrog, I think the ease of installation of SciPy depends on what BLAS you want to use and whether it available from your OS or PyPI / Anaconda etc. When I used to install all these things from source a decade or two ago, I recall ATLAS being a bit of a pain to compile and also SciPy itself requiring C, ForTran and C++ compilers to build (the last one being an undocumented build requirement at the time). From a quick poke around, it looks like the SciPy on PyPI (someone please complete the rhyme) bundles its own copy of OpenBLAS, so installation should be straightforward as long as you're happy with that -- presumably you will be if you just want AstroPy to be installable, though perhaps users should be aware that it might not have optimized performance out of the box (I don't have up-to-date info. on that).

Note that if I try to install scipy==0.8.0 from 2010 with pip on CentOS 8, the installation fails with a BlasNotFoundError. So I think the answer is that installing it from PyPI is easier than it used to be, because the required dependencies are bundled nowadays, rather than sought from the OS (at the expense of hiding optimization options from the user).

p.s. Splitting out astropy.units has come up multiple times; e.g., #10606
It certainly isn't particularly astronomy specific.

Quite naturally, it would make more sense to have it available as an external dependency building directly on numpy or a generalised array interface. Most recently under discussion, I think, in https://discuss.scientific-python.org/t/physical-units-summit/390

Careful now... there is a dedicated balance between these 3 scenarios:

Breaking everything back into their own separate packages. Remember that we made a conscious decision to put "core" functionality in the same package so users don't have to install a million things to get stuff working.
Bringing back namespace packages (gasp).
Recreating IRAF but in Python.

Navigation Menu

Repository navigation

Make scipy a required dependency #12448

36 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make scipy a required dependency #12448

Description

Activity

nstarman commented on Nov 10, 2021

pllim commented on Nov 18, 2021

pllim commented on Nov 18, 2021

adrn commented on Dec 30, 2021

pllim commented on Dec 30, 2021

mhvk commented on Dec 30, 2021

dstansby commented on Dec 30, 2021

mhvk commented on Dec 30, 2021

dhomeier commented on Dec 30, 2021

mhvk commented on Dec 30, 2021

dhomeier commented on Dec 31, 2021

hamogu commented on Jan 3, 2022

36 remaining items

mhvk commented on Jun 21, 2023

mhvk commented on Jun 21, 2023

dhomeier commented on Jun 21, 2023

taldcroft commented on Jun 22, 2023

saimn commented on Jun 22, 2023

mhvk commented on Jun 22, 2023

mhvk commented on Jun 22, 2023

adrn commented on Jun 22, 2023

mhvk commented on Jun 22, 2023

mhvk commented on Jun 22, 2023

pllim commented on Jun 22, 2023

jehturner commented on Jun 22, 2023

jehturner commented on Jun 22, 2023

dhomeier commented on Jun 22, 2023

pllim commented on Jun 22, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions