Conversation
|
I think we want this to work. @eteq - do you know why this is happening? |
|
Given this and other issues with the configuration, I wonder whether we want to consider trying to refactor it to make it more robust? Otherwise we're just constantly adding workarounds. |
|
Well, I'm certainly interested in specific suggestions for how to make it more robust @astrofrog (the best would probably be to get away from But this part of the code isn't really related to the @ChrisBeaumont - can you try putting some temporary diagnostic |
|
Sure. The changes: diff --git a/astropy/utils/misc.py b/astropy/utils/misc.py
index d8559fa..a949098 100644
--- a/astropy/utils/misc.py
+++ b/astropy/utils/misc.py
@@ -102,11 +102,18 @@ def find_current_module(depth=1, finddiff=False):
# using a patched version of getmodule because the py 3.1 and 3.2 stdlib
# is broken if the list of modules changes during import
+
from .compat import inspect_getmodule
+ from traceback import format_stack
+ print '***********\nBegin Traceback'
+ print ''.join(format_stack())
+ print '************\nEnd Traceback'
frm = currentframe()
+ print frm
for i in range(depth):
frm = frm.f_back
+ print frm
if frm is None:
return NoneWhat it prints normally: And from within py2exe |
|
Yeah I don't think the problem here has to do with the config system directly, and more to do with the fact that several places in Astropy assume that the modules are being imported from files on the file system and not more exotic importers. |
|
I see. So should I not expect Astropy to support py2exe integration? Are there other modules (besides the config system) that also rely on import nuances? |
|
There are. I don't know that this is a closed door though. I've used py2exe before but not in a while so I'm not exactly sure what it's doing here. But I think it's putting the modules in a zip file? I'd have to look into how other projects with the same types of issues integrate with py2exe if at all. |
|
Ok, keep me posted. This is something I'd be willing to work on as well. |
|
I set up a test py2exe project where I was able to reproduce this, so I'll try to poke at it a bit as well. |
|
Thanks. I'm also looking into PyInstaller, which looks like it might be more maintained. |
|
Just wanted to say that it fails the same way as py2exe when using pyinstaller (on my mac), exactly the same backtrace about the configuration. It does seem to work fine when using py2app though. If the problem is to do with the library being in a zip file, isnt this one of the standard methods of packaging python libraries with python 3 now anyway that setuptools/distribute/whatever else have become more standardised with upstream python? |
|
It really works with py2app? Because I had a different report that it fails with py2app, also with the same error. And yes, the problem has to do with the fact that py2exe and friends places all the libraries in a zip file. And no, this is no impact on setuptools. Although it is possible to import setuptools egg packages (which are zip files), Astropy (and actually probably the majority of other packages you'll find that use setuptools) set the zip_safe=False flag which causes them to be extracted to directories instead. eggs are being slowly deprecated anyways in favor of the wheel binary distribution format which doesn't share these issues since they're not designed (at the moment) to be importable. |
|
I have it working fine with py2app here: https://github.com/badders/pyfitsview |
|
I've looked into this a little more, finally. Still haven't tried py2app--I've heard some reports of it working, and some of it not. I think this might actually relate to the Python version...? This ultimately comes down to |
|
Here's the issue (at least so far) in all its gory details: in our Normally this works fine-- That's why I believe it should be easy enough to find a workaround for this, but bear with me... |
|
With the attached changes I've gotten this working so far as That said, with this patch this still does not mean Astropy will work with py2exe. Take for example this example script: It works up to the last line, where it bombs out with this traceback: This is an issue particular to ply, but there are other issues like this that are more particular to Astropy itself (for example I tried a script that runs I'm trying to figure out if there is any way to tell it very forcefully: I want you to just include all files under this package. But I have no idea. Might have to give up on py2exe. While it still seems to have an active user base, it hasn't been updated since 2008. PyInstaller seems to be considered the wave of the future for this sort of thing, and is being actively developed. Though it might have some of the same issues--I don't know. |
|
I discovered looking here that matplotlib provides a |
|
I examined this a little more, and discovered that there's blame to go around. If I modify py2exe slightly to recompile the pyc files so that the code objects serialized therein have source filenames relative to the zip file--like However, the whole thing still breaks down if you run the executable created by py2exe from any directory other than the one that contains it. Then the |
|
I have no specific suggestions, but I just want to be sure to offer @embray some encouragement - it would be great to get this working, but it sounds like it's really in the weeds! |
|
I talked to Marshall about this the other day (he brought up this same issue but in the context of py2app, which is mostly the same as py2exe). He said he doesn't care what utility he uses to bundle his application so long as it works of course. It seems like most activity in Python app bundling is in PyInstaller these days, so I'm going to see if I can have any better luck with that. If so, we might then recommend developers who want to bundle Astropy with their application use PyInstaller, and I would then offer documentation on how best to do that with Astropy. It's probably possible to get things working with py2exe/app too, but I think it's better to focus on PyInstaller first... |
|
With the latest changes importing Astropy from a zipfile mostly works now for most use cases. That's importing only. But that's only the tip of the iceberg. Almost everything that's relying on |
|
Turns out we do already have functions related to getting package data files--I thought I remembered something like that. But not ever package is using them consistently. And we still need to modify them to support loading data from a zip file, but that can be done. |
|
I fixed up the I fixed up a few bits in the code that weren't using |
|
So now astropy can work if, for example, it's imported from an egg archive. You can try this out yourself by running and try things out. We still want to leave |
|
Welp. Except that it doesn't seem to be working on Windows.... |
|
Okay, fixed the Windows thing. |
|
At this point it should be just about possible to bundle Astropy in an application built with, for example PyInstaller. There are a few areas where things are still a little dicey--in particular imports that you need that PyInstaller doesn't pick up automatically. These could be handled by a hook script for PyInstaller which I think we can put together. But I'd rather wait to work with someone on an actual application that uses this before spending much more time on it, as it will be easier to uncover all the issues with a real application. |
|
@embray - is this ready for final review? It should probably be merged soon to avoid rebasing given the extensive changes, right? |
|
No, I still want to put a call out to @ChrisBeaumont and @mperrin (the only people I specifically know of so far who have tried to bundle Astropy in an application) to either test this out, or point me to their application source code so that I can try to keep working on it. This also deserves further documentation on how to bundle Astropy in an application. Currently I would recommend only using PyInstaller over py2exe or py2app, and this should be documented. Currently also Astropy will only work if bundled in it entirely as an .egg archive. It would still be good to write a hook script for PyInstaller to inform it how to correctly pick up all the files that Astropy needs to run. |
|
To amend my last comment: It's probably better if someone give me some code to try to work with this on; I've already gotten some simple programs working with Astropy so I have a basic idea of what's required. I wouldn't want anyone else to waste a lot of time on it until I've been able to document what's required. |
|
Marshall and I got this working once aeons ago for webbpsf. It required a few custom handlers for Astropy plus the tweaks in this branch, but it worked. This branch is way out of date though and would be a tough problem to rebase. |
|
This came up again on StackOverflow. Would like to try to find the time to move it forward again sometime if I can--maybe even add a continuous integration build based on a 'frozen' astropy to keep that working. Need to rebase again, and also need to incorporate the solution from #4531 or something like it. |
|
FWIW, I set up this repo: |
|
I want to know what's the status now? I try to bundle my pipeline on Windows for specific project using windows, and python is easy to distribute. But I tried py2exe pyinstaller and mainly cx_freeze, no luck. I can skip the "six" error by adding astropy.extern.bundle.six manually to the setup script. But, still face the problem of get_config error. I hope this can be solved no matter how. |
|
I did it the same as described above, don't 'pack' it, but make sure astropy gets placed in a directory, I believe it is the |
|
I used cx_freeze and all the modules are in directory, not one package. Still no luck. Even includes astropy manually. The problem is that find_current_module(2) always return None, and I don't know why. |
add astropy to sys.path[0] may cause io and gzip error, so I change to [2], but still get the same error of get_config. |
|
i don't think it's the problem of getabsfile, i found that i can get the right return of getabsfile, but i always get none return of getmodule function. |
|
Are you sure, since what I wrote (I cannot remember all the details):
|
|
Sorry for the carelessness, I didn't check the whole output including "astropy". Under windows, all module list using lower case and getabsfile return .py not .pyc, but cx_freeze only packing .pyc. Also add path to the end to avoid path conflicts. This is ugly but can work. I hope this will be resolved permanently in the future. At least using bundled inspect module? |
|
I think it would be great to include a test in the appveyor build matrix for bundling Astropy with pyinstaller or something, so that we can keep this working in the future. |
|
Hi there @embray 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labelled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃. I see this is an experimental pull request. I'll report back on the checks once the PR discussion in settled. |
|
Hi humans 👋 - this pull request hasn't had any new commits for approximately 1 year, 7 months. I plan to close this in a month if the pull request doesn't have any new commits by then. In lieu of a stalled pull request, please close this and open an issue instead to revisit in the future. Maintainers may also choose to add If you believe I commented on this issue incorrectly, please report this here. |
|
@maartenbreddels @embray @fockez @josePhoenix @badders - I think it would be good to understand what the latest status is here. I think all of you used various tools (py2app, py2exe, pyinstaller). Could you report back as to what currently works/doesn't work with the developer version of astropy? Note that the developer version of Astropy doesn't include bundled versions of six or pytest anymore, which were causing issues for some people before. I think this PR may include some useful things we should incorporate anyway, but it's going to be incredibly difficult to rebase given how old it is and changes in e.g. the testing infrastructure. It might make more sense to just open a new issue and point to this PR in case someone wants to start again (unless, @embray, you want to try rebasing this!). It would also be helpful if someone could point out how we could test this in our continuous integration - what would a minimal project look like, and how would we test that it works? |
|
Recently, I switch my code from py2 to py3, and this cause new trouble. Under PyInstaller, it always report when loading built app: So, when the 3.0 of astropy will be released or at least beta version, totally no need of six package. |
|
@fockez I was able to get this to work using PyInstaller with the following |
|
@fridgerator - ok that sounds pretty straightforward. So no need for the changes in this PR? |
|
@astrofrog Correct, I didn't change anything else or use anything from this PR |
|
⏰ Time's up! ⏰ I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks! If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here. |
|
I'm trying to use the exclude/include trick from above. It got rid of the I'm using a virtualenv named (the produced executable is also >200MB, is that normal?) The full output from |
12 participants
This may be outside the scope of the project, but I'm trying to make a standalone Windows application (with py2exe) that uses astropy. I don't get very far, as trying to import astropy triggers complaints about configuration.
Here's a simple version of the problem:
in main.py:
in setup.py
This is enough to run
python setup.py py2exeand create dist/main.exe on my machine, but running it gives: