Repository navigation
Move astropy to src directory #13678
Description
Description
Having the following package structure is considered much safer.
astropy/
├── pyproject.toml
├── ...
├── src/
│ └── astropy/
Additional context
https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-the-package-files
https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
Reacted by Stuart Mumford
Activity
nstarman commentedon Sep 16, 2022
Maybe the release manager can do this in the v5.2 release? It doesn't take too much work, just a few files like
setup.cfgneed to be updated.pllim commentedon Sep 16, 2022
Oh, dear... Won't this create some backport nightmare? I think @Cadair brought this up several times in the past but I don't think we have the energy to deal with the amount of work this will generate.
Nothing is broken (yet), why is this needed now?
dhomeier commentedon Sep 16, 2022
I don't see real context in the link, other than the given example using a
srcsubdirectory – is there some reference why that should be safer (other than possibly allowing to run python in the top dir withoutimport astropyimporting directly from the repo, which numpy for example expressly forbids)?nstarman commentedon Sep 16, 2022
Posted the wrong link. Meant to do https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
nstarman commentedon Sep 16, 2022
The section on flat layouts outlines why src layouts are preferred.
nstarman commentedon Sep 16, 2022
Instead of v5.2, could we instead do this when the new LTS is out. That should solve back ports.
Also, I'm not 100% sure of this, but if a commit just moves a file and then another commit is rebased on that 1st commit, I think it still works. Meaning, I don't think this breaks backports.
dhomeier commentedon Sep 16, 2022
I don't see a clear recommendation of either layout over the other there, so we'll need to weigh the pros and cons of both ourselves.
nstarman commentedon Sep 16, 2022
Some highlighted issues with flat-layout:
These aren't insurmountable, but src-layout has no issues. If we could transition in a non-painful way, it could be nice.
33 remaining items
namurphy commentedon Mar 27, 2024
I've been working on PlasmaPy/PlasmaPy#2598 to switch PlasmaPy to an
srclayout with a separate top-level directory fortests. I'm putting a summary of my thoughts this comment, in case Astropy eventually switches to ansrclayout.Overall:
tox.ini🥲namurphy commentedon May 7, 2024
To follow up on my previous post, switching to an
srclayout in PlasmaPy ended up causing very few disruptions, either to myself or others. Overall it's been a pretty smooth transition! 🎉🎂💖🎆🪩🌌🥦Because we used
git mvwhen changing the layout, we were able to apply the layout changes to the PR branches with agit pull upstream main. I had no merge conflicts associated with the change for all of the ∼10 PRs I had open at the time.The impacts we've noticed have been:
plasmapy/directory alongside thesrc/plasmapy/directory, and emptytests/subdirectories.The caveat is that because Astropy has a much larger contributor base, the chance of at least some noteworthy disruptions happening is higher. But, at least for PlasmaPy, it's been a quite smooth process.
Update: we did run into one trickier situation recently where an older PR created new files under the old directory structure, so we had to resolve a merge conflict there. If Astropy does eventually switch to an
srclayout, it may be worthwhile for a core maintainer to do agit pull upstream mainandgit pushto all active PRs just after thesrcpull request is merged.astrofrog commentedon Sep 26, 2026
@adrn might also want to comment on this as he advocated for it at the coordination meeting this year (this was not discussed in a breakout or plenary session though)
My personal opinion is that I can see arguments both ways, and I think having things in the
src/layout may be a bit cleaner, but if we did move I would advocate for keeping the tests in the source tree rather than moving them out - it's very useful to have the tests be part of the installed version.nstarman commentedon Sep 27, 2026
+1 on moving to src layout.
IMO there are good reasons to move the test directory, but I think we can have the test directory discussion independently of this one.
mhvk commentedon Sep 27, 2026
Continued 👎 to refactoring for very minor benefits.
taldcroft commentedon Sep 27, 2026
nstarman commentedon Sep 27, 2026
@taldcroft I'd like to hear more about that, because I must be missing something critical. From my understanding the whole point of worktrees is to isolate changes, e.g. allowing development on multiple branches simultaneously because each is in a different worktree — a particularly nice pattern I use everyday with AI since it can crunch on those separate trees in the background. A common editable install sounds like its breaking that isolation, which seems to defeat the whole point of the worktree, and sounds equivalent to developing on multiple branches on the same code. How does this not crash together? Is this to try to save env space? If you use
condathen it's an issue, but withuvits global cache means you’re not actually reinstalling dependencies, so you get the isolation point of worktrees at minimal (some symlinks) extra disk space. (all my codes, some with 100+ worktrees (I know, I need to prune 😆), usesrc-layout +uvand it all works great.)@mhvk, I'm sure this is on your radar, but https://git-scm.com/docs/directory-rename-detection describes the git mv mechanism, and more importantly how if we did
git mv astropy src/astropyvery few branches — active, in-development, or moldering on an old laptop — should need to be rebased. Migrating to asrcdirectory seems like a largely infrastructural choice with little impact on subpackages ... except that it's verifiably safer, esp for testing: https://hynek.me/articles/testing-packaging.taldcroft commentedon Sep 27, 2026
@nstarman - I use conda for several reasons, so yes, making lots of new full installs gets expensive. Claude was able to figure out a clever way to use worktrees with a single editable install and they are indeed isolated. I can give you Claude's explanation if you care, but probably not. 😄
The more important point is that the astropy developer community is large and they have evolved many ways of doing things based on the current flat layout. Breaking those current workflows would require strongly compelling reasons. The ones presented seem marginal at best. If there are compelling issues then why have so few other major packages changed their layout?
In theory yes, but in practice I'm not aware of it being a real issue.
Astropy doesn't use auto-discovery and there is only one top-level package.
mhvk commentedon Sep 27, 2026
@nstarman - my problem is finger memory. Yes, it will slowly adapt to every single time I open a file in an editor or run
pytest ...having to types<TAB>/a<TAB>instead of a directa<TAB>, but it is a negative.I suggest we do not revisit layout until there is a general move in the scientific python ecosystem towards a different one.
nstarman commentedon Sep 27, 2026
I think one of the few workflows that would break is this Claude workaround you're describing, which is also a new development, giving that AI post November 2024 is what popularized git worktrees. Most other workflows are building an editable install, modifying files, and pushing, which a src-layout doesn't break, just protects.
Presumably because it can be a bit of a hassle to check the infrastructure / CI change. Which to me begs the different question, how much of a hassle is it for us? IMO if we're planning to do a migration to Meson et al then this is a comparatively minor change.
But it also depends on which "major" packages, you mean, e.g. https://github.com/psf/requests is about as big as it gets and switched many years ago to
src-layout. I've never been convinced that a scientific-python bubble that sometimes doesn't closely track the rest of python is a good idea; if we want to be small and not tap into Python everywhere, we'd be Julia.src-layouts are minor in those set of things, e.g. I think Pydantic and other things are proverbially larger potatoes, but still.I also think we have a different onus of responsibility than numpy and pandas. We used to have a package template because Astropy is also trying to help the astronomy community to develop open source packages. We don't maintain that package template anymore and defer to resources like https://packaging.python.org/en/latest/ or https://www.pyopensci.org/python-package-guide/. IMO it's b bad that we don't practice what we recommend — e.g. see https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/ and https://www.pyopensci.org/python-package-guide/tutorials/get-to-know-hatch.html#step-3 — where those resources pretty clearly favour src layouts. We basically tell people: "do as we aught, not as we do".
mhvk commentedon Sep 27, 2026
Well, in terms of recommendations, there is still https://packaging-guide.openastronomy.org/en/latest/ -- which does not use a src layout.
Anyway, I don't think we'll convince each other here. I do not really want to spend more time on this, but my vote is for no change unless scientific python as a whole decides to change.
nstarman commentedon Sep 27, 2026
That's the latest iteration of our outdated template. It still recommends
distutils.https://giphy.com/gifs/foxtv-the-simpsons-fox-hSulTsYX8yatJMVtiQ
adrn commentedon Sep 27, 2026
FWIW I do support changing to the
srclayout, but appreciate your reasons for not wanting to. Perhaps this is something that will require a vote at some point.I was reminded of this during the coordination meeting (and hence discussed with @astrofrog) because I was trying to check if a failing test was also failing in the latest stable version - I installed v8.0.1 but running the test in my environment grabbed the dev version (from the cloned repo). I've gotten used to being protected from issues like this in other packages I develop that use
src.The Open Astronomy packaging guide seems to be a bit out of date relative to the vast majority of recommendations that recommend the
srclayout (@nstarman linked to some already):I agree with that part at least 😅.
namurphy commentedon Sep 28, 2026
Let's say we're in the top-level directory of the astropy repository, and run
pip install .. We openpythonand then doimport astropy. Is theastropythat gets imported going to be the installed version, or the version contained in theastropy/directory? When we runpytest, are we running tests on the installed version, or the files contained inastropy/? If we add a new.pyfile but neglect to include it inMANIFEST.in, will it be included when runningpytest?The reason why flat-layouts make me cringe is because flat-layouts introduce subtle ambiguities that can confuse contributors, and cause subtle packaging errors that are detected in CI but that don't get reproduced locally when running
pytest.But given the reticence, I'm wondering if an effective way to encourage the scientific pythoniverse to adopt src-layouts would be to write a SPEC. 🤔
mhvk commentedon Sep 28, 2026
Just for the record -- if we started astropy now, it would indeed have made sense to do a src layout. The discussion is whether it makes sense to change, where I'm with @taldcroft, who wrote
Anyway, I agree that if there is a SPEC, which presumably means that most of the scientific stack is happy to change, astropy should definitely follow that. In the meantime, though, I'll use the little spare time I have to do reviews.