Repository navigation
Friendly system wise compilation needed #11188
Description
Hi there
it is a Royal Mess to compile recent version of astropy under a unix system due to its obsessive bundling of every possible library and bugs all over the setup system (It used to be fine for all versions <= 4.0.1).
One of your contributor "Olebole" is an expert in this and it is only thanks to him that I succeeded a build for current 4.2 version. I have been using its patches, published on the Debian repository, more than 12 are required to just not getting a crash while running python setup.py build! At some point I was wondering if I was compiling IRAF-2.16 or python3-astropy ;)
Please merge them, or ask him consultancy if you need to understand how shared/system libraries works.
Thank you.
Best,
Chris.
Activity
mwcraig commentedon Dec 23, 2020
Thanks for the suggestion @dirteat. If you or @olebole have some concrete suggestions in a PR we could take a look at them. Generally I would say that our goal is to make installation easy but that we delegate out responsibility for making packages for specific OS. Debian is an example of that -- as you point out, they have some strict rules about not bundling binaries that available as separate packages. Some other parts of our packaging ecosystem (conda packages) are also moving in that direction.
One of the things that prevents us from completely unbundling, though, is the lack of support in pip for non-Python packages. Since
pipis the canonical tool for installing Python packages I cannot imagine unbundling everything for the foreseeable future.That said, it could perhaps be made easier in the build process to choose to build with system, instead of bundled, libraries. We also overhauled the build system in 4.1, so there may well be some things that were broken, or whose breakage became visible, in the transition.
A more specific bug report including what system you are running on, the error log you are hitting, etc, would be very helpful.
It may be that we have some pre-packaged binaries for that system that we could point you towards.
dirteat commentedon Dec 23, 2020
Thanks for the answer. The reason why I am building it is for distributing it over a linux distribution. I don't need specific help on an error message in particular, but for instance, it is impossible to purely build it against system libraries in spite of setting all the relevant environment variables. Most of the patches of Olebole are fixing issues related to that, just have a look to them, you'll see what they fix ( I would need to open a few bug reports here to summarize, I have not the courage).
Although I have interest in astronomy, I am not using astropy intensively, so if you guys decide to go to pip/conda only, I suspect you're targeting professional only; then we will just stop distributing it. I've opened this feature request just for you to be aware of having a (nice) piece of software not compiling fine in a system environment means it will most likely be abandoned by distributions at some point.
It is certainly not the place here to discuss why pip blind downloading is not very clever, neither very respectful of people not having fast internet connexions. Not talking about the fact that X users on the same system will install X times the same packages etc...
Anyway, I am also distributing iraf-community edition, and right now, it is actually easier to compile. Quite an irony :)
mhvk commentedon Dec 23, 2020
@dirteat - Thanks for the further explanation. As @mwcraig notes, we generally rely on different distributions to package, with help from @olebole and @sergiopasra for debian and redhat, resp. They both already helped to reduce dependencies, remove patches, etc., but it is not a fast process, nor super-easy, since we also have to be sure that we can install properly on Windows, MacOS, etc. PRs that make this process even a little bit better are definitely welcome.
Note that right now, for Debian like systems, we actually test that compilation using system libraries is possible in weekly cron, using the code below (we combine this with a test for non-amd64 architectures, but should work for all). In principle, we could document that, though I would worry that it would soon get out of date (but perhaps just pointing to the below code would help). Certainly, the idea for that is that it sets up a state where
pipneeds to pull in very little indeed. Again, suggestions/PRs very welcome.astropy/.github/workflows/ci_cron_weekly.yml
Lines 98 to 118 in 9c79f08
olebole commentedon Dec 24, 2020
I usually forward patches to Astropy, if I think that they are useful to be included in the upstream source. However, most patches are really specific for Debian, and mainly ensure that the tests succeed under our circumstances. Specifially:
Summarized, I don't see the big problem here -- two patches may be discussed further, but the rest is really dealing with Debian peculiarities. If you (@dirteat) see this different, feel free to open an issue on Astropy, please put me in Cc in this case.
What is the distribution you working on, BTW?
Concerning IRAF: There I am my own upstream (I mantain both iraf-community and the Debian package, and even then I have a number of patches specific for the Debian package -- not really less than for Astropy. This is just a common problem of complex software to have it well-build on a platform.
mhvk commentedon Dec 24, 2020
@olebole - thanks so much for the full list! It would be nice if we can make the use of the packages bundled in
externeasier (or perhaps move to just relying onpip/condato install it as needed); for the largest of theexternbundled packages,jquery, there is already an issue open - #10420.embray commentedon Jan 16, 2021
@olebole For the intersphinx mapping, I wonder if there might be some way to package the documentation for the other projects it links to (e.g. Python) and run the doc build with some kind of local web proxy which handles this. Probably not worth the effort though.
olebole commentedon Jan 16, 2021
@embray, you are right; and often these are already packaged. However, this creates additional dependencies, and I am too lazy to maintain them. I must say that I am not sure how useful the local documentation is at all, as it is available online as well (and I personally just use the online docs here). And the
-docpackages are installed rarely; for astropy f.e. we have >1400 package installations (Python 3), but only 36 installations reported.But I would accept patches to do this properly ;-)
embray commentedon Jan 16, 2021
Local documentation has its uses, though I agree it's rare :)
dirteat commentedon Jan 16, 2021
Hi there,
thank you for the follow-up. I had to use these ones to have a successful compilation against system libs:
Use-system-ply-instead-of-local-copy.patch
Use-system-wide-configobject-validate.patch
python-astropy-system-configobj.patch
Best,
Chris.
olebole commentedon Jan 16, 2021
@dirteat the third one does not exist for astropy_4.2-1 in Debian.
For the others two patches: Astropy should work without them. If not, it is a bug in Astropy. Can you give a proper description what fails without these patches? Which sources did you use (pypi? github?)?