Use system python packages with --enable-system-site-packages by orlitzky · Pull Request #36141 · sagemath/sage

orlitzky · 2023-08-26T00:48:18Z

Rebased branch of #29665

Platform-specific testing:

for example TARGETS_PRE="build-local" tox -e docker-ubuntu-focal-standard-sitepackages -- build ptest

…nfigure. This new option is experimental and disabled by default. When it is enabled, the string "--system-site-packages" is appended to a new environment variable SAGE_VENV_FLAGS that is then passed to the sage-venv command during the build using autoconf substitution. Thusly the correct setting finds its way into local/pyvenv.cfg.

This new macro encapsulates the check for system installations of python packages. It uses setuptools.version's pkg_resources to check if the contents of install-requires.txt are satisfied for the given package; and if not, tells the build system to install the SPKG instead.

This includes a SAGE_SPKG_DEPCHECK([gsl glpk suitesparse],... because those C dependencies wind up linked into shared libraries like, /usr/.../cvxopt/glpk.cpython-39-x86_64-linux-gnu.so that could potentially conflict with (for example) the glpk SPKG.

This includes a SAGE_SPKG_DEPCHECK([zeromq],... because libzmq winds up linked into shared libraries like, /usr/.../zmq/backend/cython/message.cpython-39-x86_64-linux-gnu.so that potentially conflict with the zeromq SPKG.

This includes a SAGE_SPKG_DEPCHECK([openblas],... because blas/lapack wind up linked into shared libraries like, /usr/.../numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so that potentially conflict with the openblas SPKG.

This includes a SAGE_SPKG_DEPCHECK([openblas],... because blas/lapack wind up linked into shared libraries like, /usr/.../scipy/linalg/_flinalg.cpython-39-x86_64-linux-gnu.so that potentially conflict with the openblas SPKG.

This includes a SAGE_SPKG_DEPCHECK([bzip2 freetype libpng qhull],... because those dependencies wind up linked into shared libraries like, /usr/lib/.../matplotlib/ft2font.cpython-39-x86_64-linux-gnu.so that potentially conflict with their SPKG counterparts.

orlitzky · 2023-09-01T11:24:03Z

Please take a look if you agree with the build system changes that I pushed

One nit: despite what the GNU sed man page says, sed -E is not actually in the latest POSIX spec. AIX, HP-UX, and Solaris look like they're missing it. Would cut -f2 -d. work as well to parse the python minor version?

orlitzky · 2023-09-01T11:29:18Z

I had to check, but there is an ABI guarantee for python, so we should be OK rebuilding everything only when python's minor version changes.

mkoeppe · 2023-09-01T21:43:51Z

Would cut -f2 -d. work as well to parse the python minor version?

Sure, that would work too and is of course more elegant

dimpase · 2023-09-01T21:46:53Z

On Fri, Sep 01, 2023 at 02:44:02PM -0700, Matthias Köppe wrote: > Would `cut -f2 -d.` work as well to parse the python minor version? Sure, that would work too and is of course more elegant

it's a Python bug that the minor version is not available from pkg-config, IMHO

Use the "cut" utility to parse the python minor version component instead of sed -E. Despite what the GNU sed manpage says, the "-E" flag does not actually appear in any version of POSIX, while "cut" does. The cut version is also a bit more readable.

orlitzky · 2023-09-01T23:04:59Z

it's a Python bug that the minor version is not available from pkg-config, IMHO

At least get_python_version() is documented to output something usable. You can rely on $PATH to execute it. There are far far worse situations.

Anyway, I pushed the cut change. Everything else LGTM. Thanks for the review and the help. This should keep me busy for a while.

vbraun · 2023-09-01T23:27:52Z

merge conflict with #36112

mkoeppe · 2023-09-01T23:45:38Z

Merged #36112

mkoeppe · 2023-09-01T23:46:28Z

Tests at https://github.com/mkoeppe/sage/actions/runs/6054949711

mkoeppe · 2023-09-02T01:02:56Z

debian-bullseye (https://github.com/mkoeppe/sage/actions/runs/6054949711/job/16433159643#logs)

[categorie] updating environment: [new config] 223 added, 0 changed, 0 removed
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/sage/src/sage/repl/ipython_kernel/__main__.py", line 1, in <module>
    from ipykernel.kernelapp import IPKernelApp
  File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 33, in <module>
    from traitlets.utils import filefind
ImportError: cannot import name 'filefind' from 'traitlets.utils' (/usr/lib/python3/dist-packages/traitlets/utils/__init__.py)
[plot3d   ] Extension error:
[plot3d   ] Notebook execution failed (exception: Kernel died before replying to kernel_info)

github-actions · 2023-09-02T02:07:05Z

Documentation preview for this PR (built with commit de831a5; changes) is ready! 🎉

mkoeppe · 2023-09-02T02:23:37Z

fedora-33:

pkg_resources.ContextualVersionConflict: (sphinxcontrib-htmlhelp 1.0.3 (/usr/lib/python3.9/site-packages), Requirement.parse('sphinxcontrib-htmlhelp>=2.0.0'), {'sphinx'})

dimpase · 2023-09-02T06:32:12Z

https://docs.fedoraproject.org/en-US/releases/lifecycle/ Fedora 36 and earlier are past EOL. Let's not waste time on them.

…

On 2 September 2023 05:23:48 EEST, "Matthias Köppe" ***@***.***> wrote: fedora-33: ``` pkg_resources.ContextualVersionConflict: (sphinxcontrib-htmlhelp 1.0.3 (/usr/lib/python3.9/site-packages), Requirement.parse('sphinxcontrib-htmlhelp>=2.0.0'), {'sphinx'}) ``` -- Reply to this email directly or view it on GitHub: #36141 (comment) You are receiving this because you commented. Message ID: ***@***.***>

mkoeppe · 2023-09-02T18:59:23Z

Once more I'll explain that old Fedora is proxy for RHEL.

dimpase · 2023-09-02T20:06:36Z

We are not married to RHEL, and a proxy is just a proxy. It is very broken.

To Hell with RHEL, if they are not willing to provide any useful CI.

sagemathgh-36141: Use system python packages with --enable-system-site-packages Rebased branch of sagemath#29665 Platform-specific testing: - for example TARGETS_PRE="build-local" tox -e docker-ubuntu-focal- standard-sitepackages -- build ptest Fixes sagemath#29665 URL: sagemath#36141 Reported by: Michael Orlitzky Reviewer(s): Dima Pasechnik, Matthias Köppe, Michael Orlitzky

sagemathgh-36218: src/bin/sage-env: set PYDEVD_DISABLE_FILE_VALIDATION=1 This fixes a doctest with newer versions of debugpy, as seen in in sagemath#36141 and sagemath#36129. Explanation in the commit message. URL: sagemath#36218 Reported by: Michael Orlitzky Reviewer(s): Matthias Köppe

sagemathgh-36240: Fix Python package metadata generated by `bootstrap`    After sagemath#36141, pyproject.toml and setup.cfg files are no longer generated correctly.   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36240 Reported by: Matthias Köppe Reviewer(s): Michael Orlitzky

@tobiasdiez

sagemathgh-36405: `bootstrap-conda`: Refactor, generate versioned environment files   - Environment files are now versioned with the Python minor version, as proposed in (and cherry-picked from) sagemath#35986 by @tobiasdiez - CI for conda now uses `--enable-system-site-packages` from sagemath#36141 by @orlitzky Split out from    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36405 Reported by: Matthias Köppe Reviewer(s): Tobias Diez

Matthias Koeppe and others added 30 commits August 21, 2023 20:10

build/pkgs/pythran: Update to 0.13.1

9c9de7b

build/pkgs/meson_python: Update to 0.13.2

e70ddfc

build/pkgs/pybind11: Update to 2.11.1

20c90ca

build/pkgs/pyproject_metadata: Update to 0.7.1

b7227e3

build/pkgs/gast: Update to 0.5.4

ddf8efe

build/pkgs/meson_python/patches: Remove

a291fee

build/pkgs/meson: Update to 1.2.1

cb21d9f

build/pkgs/ninja_build: Update to 1.11.1

d0be1a4

Merge branch 'cython-0.29.33' into scipy_1_11

a97e3de

build/pkgs/cmake: Update to 3.27.3

2461b37

build/pkgs/scipy/install-requires.txt: Allow 1.11.x

548160a

build/pkgs/meson/spkg-configure.m4: Require meson >= 1.1.0

3d17e4a

Trac sagemath#29665: standard python spkg-configure.m4 for py.

19fa7c8

Trac sagemath#29665: standard python spkg-configure.m4 for wheel.

dc39acc

Trac sagemath#29665: new spkg-configure.m4 for pyzmq.

a172e1b

This includes a SAGE_SPKG_DEPCHECK([zeromq],... because libzmq winds up linked into shared libraries like, /usr/.../zmq/backend/cython/message.cpython-39-x86_64-linux-gnu.so that potentially conflict with the zeromq SPKG.

Trac sagemath#29665: standard python spkg-configure.m4 for requests.

a69bc99

Trac sagemath#29665: standard python spkg-configure.m4 for six.

9a81424

Trac sagemath#29665: standard python spkg-configure.m4 for packaging.

40a17dd

Trac sagemath#29665: new spkg-configure.m4 for numpy.

0820e25

This includes a SAGE_SPKG_DEPCHECK([openblas],... because blas/lapack wind up linked into shared libraries like, /usr/.../numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so that potentially conflict with the openblas SPKG.

Trac sagemath#29665: new spkg-configure.m4 for scipy.

b902f1b

This includes a SAGE_SPKG_DEPCHECK([openblas],... because blas/lapack wind up linked into shared libraries like, /usr/.../scipy/linalg/_flinalg.cpython-39-x86_64-linux-gnu.so that potentially conflict with the openblas SPKG.

Trac sagemath#29665: standard python spkg-configure.m4 for pluggy.

04cd285

Trac sagemath#29665: standard python spkg-configure.m4 for pickleshare.

eec1c60

Trac sagemath#29665: standard python spkg-configure.m4 for html5lib.

2bf735f

Trac sagemath#29665: standard python spkg-configure.m4 for networkx.

394e7e8

Trac sagemath#29665: standard python spkg-configure.m4 for bleach.

6647de2

Trac sagemath#29665: standard python spkg-configure.m4 for decorator.

ebb763a

vbraun added the s: needs work label Sep 1, 2023

Merge branch 'scipy_1_11' into system-python-packages

de831a5

orlitzky mentioned this pull request Sep 3, 2023

Notebook 7, ipykernel 6.27, ipython 8.17 #36129

Merged

5 tasks

orlitzky mentioned this pull request Sep 8, 2023

src/bin/sage-env: set PYDEVD_DISABLE_FILE_VALIDATION=1 #36218

Merged

vbraun merged commit 7bb1e6b into sagemath:develop Sep 10, 2023

github-actions bot removed the s: positive review label Sep 10, 2023

tornaria mentioned this pull request Sep 10, 2023

Update Cython to 3.0.2 #36110

Merged

3 tasks

orlitzky deleted the system-python-packages branch September 22, 2023 00:16

Uh oh!

Comments

Conversation

orlitzky commented Aug 26, 2023 • edited by mkoeppe Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

orlitzky commented Sep 1, 2023

Uh oh!

orlitzky commented Sep 1, 2023

Uh oh!

mkoeppe commented Sep 1, 2023

Uh oh!

dimpase commented Sep 1, 2023 via email

Uh oh!

orlitzky commented Sep 1, 2023

Uh oh!

vbraun commented Sep 1, 2023

Uh oh!

mkoeppe commented Sep 1, 2023

Uh oh!

mkoeppe commented Sep 1, 2023

Uh oh!

mkoeppe commented Sep 2, 2023

Uh oh!

github-actions bot commented Sep 2, 2023

Uh oh!

mkoeppe commented Sep 2, 2023

Uh oh!

dimpase commented Sep 2, 2023 via email

Uh oh!

mkoeppe commented Sep 2, 2023

Uh oh!

dimpase commented Sep 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

orlitzky commented Aug 26, 2023 •

edited by mkoeppe

Loading

dimpase commented Sep 2, 2023 •

edited

Loading