Nuitka won't work with Astropy

Nuitka version, full Python version, flavor, OS, etc. as output by this
command.

python3 -m nuitka --version
1.7
Commercial: None
Python: 3.9.10 (main, Jun 29 2023, 21:55:15)
Flavor: Unknown
Executable: /usr/local/bin/python3
OS: Linux
Arch: x86_64
Distribution: Centos (based on Fedora) 7
Nuitka-Scons:INFO: The provided gcc is too old, switching to its g++ instead.
Version C compiler: /usr/bin/g++ (g++).

How did you install Nuitka and Python

Download from https://www.python.org/

The specific PyPI names and versions

Package Version

astropy 5.3

Many times when you get an error from Nuitka, your setup may be special

I can compile code that does not include Astropy.

Also supply a Short, Self Contained, Correct, Example

Text of minimalTest.py:

import astropy.coordinates #this does not work
#import astropy  #this line appears to work
print('hello world')

Compiling minimal test appears to work. When running the resulting executable, I get errors. Commands to repeat my simple test:

pip3 install jplephem
pip3 install astropy
time python3 -m nuitka --standalone --include-package=astropy.coordinates --include-package=astropy.constants minimalTest.py
cp /usr/local/lib/python3.9/site-packages/astropy/CITATION /generator/tmp/minimalTest.dist/astropy
./minimalTest.dist/minimalTest.bin

The result of running the .bin is an error similar to astropy/astropy#10361.

I think there are 2 issues:
Issue 1: Nuitka does something I don't understand when importing sub-packages.
I don't get an instant runtime fail when minimalTest.py calls "import astropy"
I do get an instant runtime fail when minimalTest.py calls "import astropy.coordinates"
I would expect either import call to work. Since the minimalTest.py does not make use of any astropy functions (it only imports it), I can't confirm that it is fully functional.

Issue 2: Removing docstrings
Astropy uses a 3rd party package Ply that requires docstrings. Compiling with Nuitka seems to have some similar failures as running with python3 -OO minimalTest.py. Does Nuitka remove docstrings? Where can I find documentation related to this?

Provide in your issue the Nuitka options used

--standalone --include-package=astropy.coordinates --include-package=astropy.constants
also tried with --include-package=astropy

Note if this is a regression

Not working in any tested version.

Just a note, you might want to use the EPEL devtools like this:

yum -y install epel-release centos-release-scl;yum repolist;yum -y install devtoolset-9;

The old gcc is supposed to work, but that's going to give nicer results if that gcc is used.

Nuitka does include docstrings unless you ask it to remove it, with --python-flag options.

For data files, there are options and configuration to copy them.

I definitely don't like "error similar to read some other place", please report the actual error.

In the code I found this in:

def get_caller_module_dict(levels):
    f = sys._getframe(levels)

    ldict = f.f_globals.copy()
    if f.f_globals != f.f_locals:
        ldict.update(f.f_locals)
    return ldict

This call to return parsing.yacc(tabmodule="generic_parsetab", package="astropy/units") wants to find lots of unused functions and variables in this way, and instead of passing locals() along, it just does this crapy approach. I believe it merges locals() and globals() too. My Visual Code shows these as unused functions.

I did a bunch of anti-bloat to get rid of IPython and pytest somewhat, to make it more bearable. But it seems there is no easy way to make this compatible. Nuitka populates the frame locals only when an exception occurs, which didn't happen.

To add insult to injury, there is glorious more code.

        with _patch_get_caller_module_dict(yacc):
            parser = yacc.yacc(
                tabmodule=tabmodule,
                outputdir=os.path.dirname(caller_file),
                debug=False,
                optimize=True,
                write_tables=True,
            )

This context generator wraps get_caller_module_dict such that the levels that it expects it up, which is hard coded to 2 is then going to be 3, since it's make a call through another function.

This is clearly not close to good code.

The current state of factory should get you straight there. The exception chain points to yacc and the error is ERROR: No token list is defined. No more options are needed, dependencies and data files were added to Nuitka package configuration there.

Thanks for the quick reply. I've never installed nuitka from factory code, so it may take a bit to test. I plan to try this today.

My testing is still in progress. I stumbled across something interesting. I don't think this is the cause of my issues, but worth noting just in case.

I added the following lines to the top of my minimalTest.py. (It is quite possible that the rest of minimalTest.py is unnecessary, but I didn't test it that way.)

import sys
print(sys.flags)

When I run my script, I get different values of the optimize flag when I run nuitka vs python with the -OO flag.

Running with nuitka:

time python3 -m nuitka --standalone --python-flag=-OO minimalTest.py
./minimalTest.dist/minimalTest.bin

Running with Python
python3 -OO minimalTest.py

Results of optimize flag:
Python Flag - Nuitka - Python
none ------------ 0 -------- 0
-O --------------- 1 -------- 1
-OO ------------- 1 -------- 2
doc+ass* ------- 1 ---------n/a

*full command: --python-flag=no_docstrings --python-flag=no_asserts

That's indeed very interesting. I believe making a patch to upstream, such that get_caller_module_dict is replaced by explicit passing of context should be trival. If you or anybody were to do and share it, that will be great.

For now, the result of said function is an empty dict, which triggers the error we remain seeing.

As for your flags issue, that's indeed a compatiblity bug. I wonder if people check it ever, but rather than wondering, I guess it's better to make it behave compatible.

The last of your cases is -OO spelled out.

                elif part in ("-OO",):
                    _python_flags.add("no_docstrings")
                    _python_flags.add("no_asserts")

Nuitka allows a mode that Python doesn't have there. Gradually we are going to move to allowing a scoping on these things, such that e.g. no_docstrings will become default for stdlib potentially. Or that Yaml package configuration will allow to enforce docstrings for certain modules or functions. Going to get more messy to reduce it to one value.

Just noticing, that the __compiled__ probably deserves to have these flags, and then maybe needs to be made module local where there are differences, or be accompanied by another variable that is not as static.

So for factory, astropy anti-bloat, etc. is complete from my side, should need no options at all anymore. However, an upstream change is needed. I did try to set frame locals value before the call to locals(), like so sys._getframe().f_locals.update(locals()) but it didn't yield the result I was hoping for. This was a new idea I had, but I likely need to check it in a smaller test case.

That's on develop now.

Thanks for the update. I think the next step is for me to contact the astropy team to investigate the issues you mentioned above.

After some thinking - one could fix this by monkey patching the get_caller_module_dict function to examine its caller - then dynamically look up the call chain for the place where that caller is looking for its variables (thus making the "frames" argument meaningless) or the astropy / ply libraries could implement my solution of explicitly passing the local variables to the parser / lexer (This might be able to be monkey patched as well - but would be a lot to keep things organized).

Anyways it took all day, but I got this to work - so hopefully this helps others

@golfinq

Sounds like amazing input @golfinq thanks for sharing it, I will try and look into what to apply for general Nuitka there.

Instead of passing "locals" probably, we could just have it set as a global variable somewhere, maybe reducing the amount of changes that are necessary, just putting this here as it came to my mind, without even looking at much of it.

Yeah, adding it to globals is another idea I had; you would need to keep track of what is added in order to pop it after the parser is done being built.

It is too bad the astropy devs didnt encapsulate the tokens and functions into an explicit module which could then be passed (the code to read this from a given module is literally right there and bypasses the introspection) and instead relied on introspection of the function caller.

Also check licensing stuff to see if def _add_tab_header is legally nessacary, as that saves the copying of source files into the final directory and allows for easily making the final executable one file.

I might have time to do that encapsulation into a module (which is then passed) and make an astropy PR in a few weeks; but if someone else could do it (or if another solution is found) that would be swell. I am sticking with my solution in the meantime 🤷‍♀

Any progress on this one?

Build recipe for the standalone pyobs-gui binary

Thanks for your report. This is fixed on the factory branch, which is a development version that is under rapid development. You can try it out by going here: https://nuitka.net/doc/factory.html

Feedback on whether this is working is very welcome. Please do not share plans to do it; only confirm or deny that it is working.

Part of the current pre-release and will be part of 4.4 release.

Navigation Menu

Repository navigation

Nuitka won't work with Astropy #2313

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Nuitka won't work with Astropy #2313

Description

Activity

kayhayen commented on Jul 6, 2023

kayhayen commented on Jul 6, 2023

MatCat776 commented on Jul 6, 2023

MatCat776 commented on Jul 6, 2023

kayhayen commented on Jul 7, 2023

kayhayen commented on Jul 7, 2023

kayhayen commented on Jul 7, 2023

kayhayen commented on Jul 7, 2023

MatCat776 commented on Jul 7, 2023

17 remaining items

golfinq commented on May 20, 2025

kayhayen commented on May 20, 2025

kayhayen commented on May 20, 2025

golfinq commented on May 20, 2025

golfinq commented on May 20, 2025

kayhayen commented on Sep 25, 2025

kayhayen commented on Oct 30, 2026

kayhayen commented on Nov 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions