Repository navigation
Description
Description
The problem was initial reported as part of #15028, but has a different root cause, so I'm making a separate ticket.
Expected behavior
I should be able to write code that uses Astropy, and compile my code using Nuitka.
How to Reproduce
Create minimalTest.py with the following text:
import astropy.units
import astropy.coordinates as ascord
print('hello world')
Confirm you got the fix for #15028. Should output 'hello world.'
python3 -OO minimalTest.py
This is only necessary until the fix is in the released version.
Install Nuitka.
git clone https://github.com/Nuitka/Nuitka
python3 setup.py install
Lines above can be replaced with pip3 install nuitka once the patch from Nuitka/Nuitka#2313 is in their release branch.
Compile using Nuitka and run the resulting executable.
pip3 install patchelf
python3 -m nuitka --standalone minimalTest.py
./minimalTest.dist/minimalTest.bin
Output of minimalTest.bin:
ERROR: No token list is defined
Traceback (most recent call last):
File "/Testing/minimalTest.dist/astropy/units/format/generic.py", line 587, in _do_parse
File "/Testing/minimalTest.dist/astropy/units/format/generic.py", line 483, in _parse_unit
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Testing/minimalTest.dist/astropy/units/core.py", line 2098, in __call__
File "/Testing/minimalTest.dist/astropy/units/format/generic.py", line 571, in parse
File "/Testing/minimalTest.dist/astropy/units/format/generic.py", line 590, in _do_parse
File "/Testing/minimalTest.dist/astropy/utils/decorators.py", line 747, in __get__
File "/Testing/minimalTest.dist/astropy/units/format/generic.py", line 72, in _parser
File "/Testing/minimalTest.dist/astropy/units/format/generic.py", line 448, in _make_parser
File "/Testing/minimalTest.dist/astropy/utils/parsing.py", line 148, in yacc
File "/Testing/minimalTest.dist/astropy/extern/ply/yacc.py", line 3282, in yacc
astropy.extern.ply.yacc.YaccError: Unable to build parser
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Testing/minimalTest.dist/minimalTest.py", line 3, in <module>
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/Testing/minimalTest.dist/astropy/units/__init__.py", line 25, in <module astropy.units>
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/Testing/minimalTest.dist/astropy/units/astrophys.py", line 66, in <module astropy.units.astrophys>
File "/Testing/minimalTest.dist/astropy/constants/constant.py", line 42, in wrapper
File "/Testing/minimalTest.dist/astropy/units/quantity.py", line 1011, in unit
File "/Testing/minimalTest.dist/astropy/utils/decorators.py", line 837, in __get__
File "/Testing/minimalTest.dist/astropy/constants/constant.py", line 208, in _unit
File "/Testing/minimalTest.dist/astropy/units/core.py", line 2121, in __call__
ValueError: 'm / (s)' did not parse as unit: Unable to build parser If this is meant to be a custom unit, define it with 'u.def_unit'. To have it recognized inside a file reader or other code, enable it with 'u.add_enabled_units'. For details, see https://docs.astropy.org/en/latest/units/combining_and_defining.html
This error message looks very similar to #10361, but I think the fix for 15037 should have resolved the related issue.
For those following the previous thread, Nuitka fixed their optimize flag problem, and that did not resolve my issue.
Kayhayen has some suggestions in this thread that I don't know exactly how to implement, but I'm not the best Python programmer: Nuitka/Nuitka#2313
Versions
Linux-5.18.7-1.el7.elrepo.x86_64-x86_64-with-glibc2.17
Python 3.9.10 (main, Jun 29 2023, 21:55:15)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
astropy 6.0.dev396+g7e9f51e (need this to get -OO fix, but get similar errors with stable release)
Numpy 1.25.0
pyerfa 2.0.0.3
Traceback (most recent call last):
File "/generator/Testing/minimalTest.py", line 9, in
import scipy; print("Scipy", scipy.version)
ModuleNotFoundError: No module named 'scipy'
Activity
pllim commentedon Jul 13, 2023
Well, at least we got past the coordinates import error for
__doc__. 😬pllim commentedon Jul 13, 2023
The person mentioned a piece of code in Nuitka/Nuitka#2313 that I believe was added in #11227 by @bmerry for thread-safety; so perhaps bmerry can shed some light on this? 🙏
bmerry commentedon Jul 13, 2023
I'm not familiar with Nuitka at all, but from glancing at the discussion, the code that's particularly problematic (
get_caller_module_dict) is in the vendored ply module (and there are two copies of it, one in lex.py and one in yacc.py). My changes in #11227 does some patching of those functions, but it wasn't clear to me whether that patching is what's causing problems for Nuitka, or the way ply does introspection (the patching was done to avoid introducing changes in the vendored copy of the ply package).MatCat776 commentedon Jul 14, 2023
No solution yet, but some interesting facts that may help with debugging.
I printed out the value of pdict after astropy/astropy/extern/ply/yacc.py line 3245. The value of pdict is slightly different depending on whether I'm running normal python or a bin compiled by Nuitka. Note that in the normal python method, tokens is defined towards the bottom, but this section is missing from the compiled bin version.
Normal python use:
debug out: pdict
Combiled bin use:
debug out: pdict
Search for 'Timeout' in both error strings. (Timeout is not significant, but it gets you to the right place in the string. You can't search for a missing term.) Notice that the first significant deviation is for the value 'quit'.
Any chance Nuitka does something odd with the value 'quit'?
MatCat776 commentedon Jul 14, 2023
Alternate solution to consider: If Astropy is planning to update to the new version of Ply, we should probably wait for that upgrade before troubleshooting this.
Current version of Ply is available here. https://github.com/dabeaz/ply
Ply/Astropy discussions are spread across a few threads:
#15028
#13399
I looked into updating Ply and it is a bit over my head. I'm happy to help with this transition if this is likely a good solution, but I'm not going to dive in just yet.
Update: I looked at the source code for the current Ply and its usage of get_caller_module_dict looks quite similar to the old code.
pllim commentedon Jul 14, 2023
#13399 has been open for a year, so I am not sure if it is high priority for us, especially when it is not trivial.
MatCat776 commentedon Jul 19, 2023
I'm out of time to debug this. Unfortunately our solution for now is to not use the Astropy library. I do hope to come back at some point.
golfinq commentedon May 20, 2025
If someone could look at the work I did that would be great - as fixing this on the astropy side is easier than on the nuitka side
Nuitka/Nuitka#2313 (comment)
pllim commentedon May 20, 2025
golfinq commentedon May 20, 2025
Ply has a parameter called "module=" in yacc and lex which can be used to supply
p_*andt_*definitions explicitly as opposed to relying on introspection. The most straight forward method would be to utilize this parameter in order to supply the needed definitions to the constructor. This is because compliation makes the call chain introspection unreliable.pllim commentedon May 20, 2025
Having glanced at the updated convo at Nuitka/Nuitka#2313 and dug up some old ply issues we have, I am not that optimistic the patch on astropy side would be simple. First of all, the one we bundle is old, but upgrading is not easy (#13399). And then as you pointed out in Nuitka/Nuitka#2313 (comment) , there are multiple places that would need refactoring. I don't even recall why it was designed this way to begin with... @mhvk , could you add more details if you remember?
Thanks, all!
mhvk commentedon May 21, 2025
The units stuff for ply predates me. I don't think there is any sense in us adjusting our use of the old, vendored ply code; instead, we should update to more recent ply (or another solution for this type of parsing). Bottom line: I think solving this is blocked by #13399.
golfinq commentedon May 21, 2025
As far as replacements are concerned there is lark and pyparsing - would need to look at the token definitions more to see if either of them would work
mhvk commentedon May 26, 2025
From a look around (e.g., https://github.com/goodmami/python-parsing-benchmarks), it seems most other parsing libraries are rather slow, which makes it less clear whether they are good alternatives. Would certainly be wise to have some benchmarks first (and maybe have a look at what the other units packages use).
On the more general issue, I think defining the PLY parser with anything resembling
-OOwill always fail, since the parser is defined in docstrings, and those are stripped in standard optimize mode at least.Build recipe for the standalone pyobs-gui binary