Repository navigation
[Bug]: netBSD packaging build broken for 3.9 #28260
Description
Bug summary
Hi! I've tried updating the pkgsrc package for matplotlib to 3.9.0 and ran the self tests to confirm it works, but I got an error about a missing symbol. I think the symbol should come from the included external agg24-svn library. I see the following weird output in the build log:
[1/40] Generating lib/matplotlib/mpl-data/matplotlibrc with a custom command
[2/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_vpgen_segmentator.cpp.o
[3/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_trans_affine.cpp.o
[4/40] Compiling C++ object extern/ttconv/libttconv.a.p/ttutil.cpp.o
[5/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_bezier_arc.cpp.o
[6/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_vcgen_dash.cpp.o
[7/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_image_filters.cpp.o
[8/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_curves.cpp.o
[9/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_vcgen_stroke.cpp.o
[10/40] Compiling C++ object extern/agg24-svn/libagg.a.p/src_agg_vcgen_contour.cpp.o
[11/40] Linking static target extern/agg24-svn/libagg.a
ar: extern/agg24-svn/libagg.a.p/src_agg_bezier_arc.cpp.o: plugin needed to handle lto object
[12/40] Compiling C++ object extern/ttconv/libttconv.a.p/pprdrv_tt.cpp.o
[13/40] Compiling C++ object extern/ttconv/libttconv.a.p/pprdrv_tt2.cpp.o
[14/40] Linking static target extern/ttconv/libttconv.a
ar: extern/ttconv/libttconv.a.p/pprdrv_tt2.cpp.o: plugin needed to handle lto object
I'm not sure what that wants to tell me.
Code for reproduction
Try to build matplotlib 3.9.0, run self tests.Actual outcome
___________________________________________________________________________ ERROR collecting lib/matplotlib/testing ___________________________________________________________________________
<frozen importlib._bootstrap_external>:995: in exec_module
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
lib/matplotlib/__init__.py:159: in <module>
from . import _api, _version, cbook, _docstring, rcsetup
lib/matplotlib/cbook.py:32: in <module>
from matplotlib import _api, _c_internal_utils
E ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
____________________________________________________________________________ ERROR collecting lib/matplotlib/tests ____________________________________________________________________________
/usr/pkg/lib/python3.12/site-packages/pluggy/_manager.py:169: in register
hook._maybe_apply_history(hookimpl)
/usr/pkg/lib/python3.12/site-packages/pluggy/_hooks.py:581: in _maybe_apply_history
res = self._hookexec(self.name, [method], kwargs, False)
/usr/pkg/lib/python3.12/site-packages/pluggy/_manager.py:120: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
lib/matplotlib/testing/conftest.py:27: in pytest_configure
matplotlib.use('agg', force=True)
E AttributeError: module 'matplotlib' has no attribute 'use'
_____________________________________________________________________ ERROR collecting lib/mpl_toolkits/axes_grid1/tests ______________________________________________________________________
<frozen importlib._bootstrap_external>:995: in exec_module
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
lib/mpl_toolkits/axes_grid1/__init__.py:1: in <module>
from . import axes_size as Size
lib/mpl_toolkits/axes_grid1/axes_size.py:15: in <module>
from matplotlib.axes import Axes
lib/matplotlib/axes/__init__.py:1: in <module>
from . import _base
lib/matplotlib/axes/_base.py:14: in <module>
from matplotlib import _api, cbook, _docstring, offsetbox
lib/matplotlib/cbook.py:32: in <module>
from matplotlib import _api, _c_internal_utils
E ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
_____________________________________________________________________ ERROR collecting lib/mpl_toolkits/axisartist/tests ______________________________________________________________________
<frozen importlib._bootstrap_external>:995: in exec_module
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
lib/mpl_toolkits/axisartist/__init__.py:1: in <module>
from .axislines import Axes
lib/mpl_toolkits/axisartist/axislines.py:46: in <module>
import matplotlib.axes as maxes
lib/matplotlib/axes/__init__.py:1: in <module>
from . import _base
lib/matplotlib/axes/_base.py:14: in <module>
from matplotlib import _api, cbook, _docstring, offsetbox
lib/matplotlib/cbook.py:32: in <module>
from matplotlib import _api, _c_internal_utils
E ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
_______________________________________________________________________ ERROR collecting lib/mpl_toolkits/mplot3d/tests _______________________________________________________________________
<frozen importlib._bootstrap_external>:995: in exec_module
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
lib/mpl_toolkits/mplot3d/__init__.py:1: in <module>
from .axes3d import Axes3D
lib/mpl_toolkits/mplot3d/axes3d.py:21: in <module>
from matplotlib import _api, cbook, _docstring, _preprocess_data
lib/matplotlib/cbook.py:32: in <module>
from matplotlib import _api, _c_internal_utils
E ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
=================================================================================== short test summary info ===================================================================================
ERROR lib/matplotlib/testing - ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
ERROR lib/matplotlib/tests - AttributeError: module 'matplotlib' has no attribute 'use'
ERROR lib/mpl_toolkits/axes_grid1/tests - ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
ERROR lib/mpl_toolkits/axisartist/tests - ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
ERROR lib/mpl_toolkits/mplot3d/tests - ImportError: cannot import name '_c_internal_utils' from 'matplotlib' (/scratch/graphics/py-matplotlib/work/matplotlib-3.9.0/lib/matplotlib/__init__.py)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================================== 5 errors in 0.38s ======================================================================================
Expected outcome
Successful test.
Additional information
No response
Operating system
NetBSD 10.99.10/amd64
Matplotlib Version
3.9.0
Matplotlib Backend
fails due to missing symbol
Python version
3.12.3
Jupyter version
n/a
Installation
from source (.tar.gz)
Activity
tacaswell commentedon May 19, 2024
I'm not seeing a missing symbol in the logs, I'm seeing that one of our c-extensions did not get built/installed correctly.
The module that is failing is https://github.com/matplotlib/matplotlib/blob/main/src/_c_internal_utils.cpp which does not use Agg (it is mostly some windows related code and checking if we have any sort of running X or wayland server available).
We switched to building with meson for 3.9, I suspect that the packaging in netbsd will need to be updated to patch
meson.optionsrather than mplsetup.cfg to use the system provided qhull + freetype (see https://github.com/matplotlib/matplotlib/blob/main/meson.options)https://matplotlib.org/stable/install/dependencies.html for the current dependencies. I looked at https://github.com/NetBSD/pkgsrc/tree/trunk/graphics/py-matplotlib which I think is what you are using? I'm not familiar enough with netBSD's build tooling to diagnose what is wrong (I assume one of the
includesin the make file pulls in the actual build/install commands?).[-][Bug]: [/-][+][Bug]: netBSD packaging build broken for 3.9[/+]0-wiz-0 commentedon May 19, 2024
I've put the update package at https://github.com/NetBSD/pkgsrc-wip/tree/master/py-matplotlib
I've already updated it for the tool switch, and it's using qhull and freetype2 from pkgsrc.
I think it's related to agg because of this:
which has in the middle a couple lines like
tacaswell commentedon May 20, 2024
Maybe turn off LTO? We turn it on by default.
0-wiz-0 commentedon May 20, 2024
Thank you, turning off LTO fixes the missing symbol problem!
I still can't get the tests to run though. In both cases I install matplotlib. When I run py-test in the build directory, I get (see above):
in the collection step (5 times); when I run it somewhere else (not in the build directory), I get:
Here's the list of installed files matching
image:0-wiz-0 commentedon May 20, 2024
(That's with the pypi matplotlib 3.9.0 source files.)
QuLogic commentedon May 21, 2024
You need to install the test data; please read the release notes for the build change.
0-wiz-0 commentedon May 21, 2024
Thank you for the pointer. I changed the meson arguments as suggested and many more files were installed.
Now I get (after installation, running it from outside the build directory):
The installed file has:
and the non-installed file (why is that even found?) is the same
Any ideas?
QuLogic commentedon May 24, 2024
Loading from the current directory is how pytest works by default; if you want to test the installed copy, you need to ask it with
--pyargs: https://docs.pytest.org/en/8.2.x/how-to/usage.html#specifying-which-tests-to-run.QuLogic commentedon Jun 3, 2024
I just noticed this in the output; you probably need to set
AR=gcc-aror similar in the environment so that thearwill have loaded the LTO plugin. Then you won't have to disable LTO. That's something specific to netBSD though, so I'm not sure the exact binary.0-wiz-0 commentedon Jun 6, 2024
Thanks for the help.
I don't particularly care using the installed tests, I would have preferred to test using the data from the source/build directories instead, but I couldn't get that to work.
With
--pyargsI can run the tests:some complain about missing dependencies like
pikepdf,ghostscript,TeX, but most say that the images are not close.About
gcc-ar: that does not exist on NetBSD, butaris justarfrom GNU binutils 2.39.Thanks for the help!