convolution cannot be imported

✔ ~/munka/devel/astropy [hacking|✚ 1⚑ 14] $ python -c "from astropy import convolution"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/bsipocz/munka/devel/astropy/astropy/convolution/__init__.py", line 10, in <module>
    from .convolve import convolve, convolve_fft, interpolate_replace_nans, convolve_models
  File "/Users/bsipocz/munka/devel/astropy/astropy/convolution/convolve.py", line 28, in <module>
    lib_path = glob.glob(os.path.join(os.path.dirname(__file__), 'lib_convolve*'))[0]
IndexError: list index out of range

Some more cleanup of #7293 is needed.

@pllim

Well, @pllim pointed out I can't be in the source library to import convolution, at least a bit less cryptic error message would be nice to avoid causing a heart attack.

Building in the source dir also doesn't solve all the issues, now I see another one:

In [2]: >>> from astropy.convolution import convolve
   ...: >>> convolve([1, 4, 5, 6, 5, 7, 8], [0.2, 0.6, 0.2])
   ...: 
   ...: 
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-843c8031be97> in <module>()
----> 1 from astropy.convolution import convolve
      2 convolve([1, 4, 5, 6, 5, 7, 8], [0.2, 0.6, 0.2])

~/munka/devel/astropy/astropy/convolution/__init__.py in <module>()
      8 try:
      9     # Not guaranteed available at setup time
---> 10     from .convolve import convolve, convolve_fft, interpolate_replace_nans, convolve_models
     11 except ImportError:
     12     if not _ASTROPY_SETUP_:

~/munka/devel/astropy/astropy/convolution/convolve.py in <module>()
     23 # Turn the faulthandler ON to help catch any signals, e.g. segfaults
     24 # or asserts. This doesn't, currently, work with Jupyter Notebook.
---> 25 faulthandler.enable()
     26 
     27 # Find and load C convolution library

AttributeError: '_StdoutProxy' object has no attribute 'fileno'

I had none of these on the LTS branch....

This is due to the refactoring of the convolution code. I think we should just remove the call to faulthandler, it's not actually needed for the convolution code. And I agree that we should basically make sure the error message when importing for the source tree is the same as for other extensions. I'll open a PR.

@bsipocz

@bsipocz

Some more cleanup of #7293 is needed.

Regarding the "more", are there other issues that I should be aware of? I just looked but couldn't find any open (or very recently closed).

I also cannot import convolve running MacOS 10.13.6. This is from last night on master, git hash 064dac1

In [1]: import astropy                                                                                                                   

In [2]: astropy.__version__                                                                                                              
Out[2]: '3.2.dev23162'

In [3]: from astropy.convolution import Kernel2D                                                                                         
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-3-0d49cdbc4685> in <module>
----> 1 from astropy.convolution import Kernel2D

~/miniconda3/envs/jwst_dev/lib/python3.7/site-packages/astropy/convolution/__init__.py in <module>
      8 try:
      9     # Not guaranteed available at setup time
---> 10     from .convolve import convolve, convolve_fft, interpolate_replace_nans, convolve_models
     11 except ImportError:
     12     if not _ASTROPY_SETUP_:

~/miniconda3/envs/jwst_dev/lib/python3.7/site-packages/astropy/convolution/convolve.py in <module>
     30     libConvolve = ctypes.windll.LoadLibrary(lib_path)
     31 else:
---> 32     libConvolve = ctypes.cdll.LoadLibrary(lib_path)
     33 
     34 

~/miniconda3/envs/jwst_dev/lib/python3.7/ctypes/__init__.py in LoadLibrary(self, name)
    432 
    433     def LoadLibrary(self, name):
--> 434         return self._dlltype(name)
    435 
    436 cdll = LibraryLoader(CDLL)

~/miniconda3/envs/jwst_dev/lib/python3.7/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    354 
    355         if handle is None:
--> 356             self._handle = _dlopen(self._name, mode)
    357         else:
    358             self._handle = handle

OSError: dlopen(/Users/jdavies/miniconda3/envs/jwst_dev/lib/python3.7/site-packages/astropy/convolution/lib_convolve.py, 6): no suitable image found.  Did find:
	/Users/jdavies/miniconda3/envs/jwst_dev/lib/python3.7/site-packages/astropy/convolution/lib_convolve.py: file too short
	/Users/jdavies/miniconda3/envs/jwst_dev/lib/python3.7/site-packages/astropy/convolution/lib_convolve.py: file too short

@jdavies-st

Just to be clear #8035 does not fix the error @jdavies-st reported.

@jamienoss

@jamienoss - any ideas about @jdavies-st's issue?

I'll walk over and find out...

Regarding the "more", are there other issues that I should be aware of? I just looked but couldn't find any open (or very recently closed).

more, in addition to the last commit done in that PR. My first (wrong) thought was that this is due to some remnants of openMP, but apparently ctypes messes with my pure python life just as much.

@bsipocz

@bsipocz ok cool, just wanted to double check I wasn't missing something.

@nden

@nden just walked me through their findings and I'm just in the process of trying to reproduce on my box... done.

In gist, I never tested the build via ./setup install, I only ever used ./setup build and then imported explicitly from the build dir. I only tested the install with conda-build.

So the issue is that the install generates the extralib_convolve.py. The grep for lib_convolve* picks that up instead of the .so or .dll. The grep could be made more explicit, it could loop through the returned list and exclude the *py or be explicit conditional on OS, e.g. lib_convolve*.so.

The content of lib_convolve.py is:

def __bootstrap__():
    global __bootstrap__, __loader__, __file__
    import sys, pkg_resources, imp
    __file__ = pkg_resources.resource_filename(__name__, 'lib_convolve.cpython-36m-darwin.so')
    __loader__ = None; del __bootstrap__, __loader__
    imp.load_dynamic(__name__,__file__)
__bootstrap__()

The other approach is to determine why it is even there. I will also sanity check the conda-build method again.

To be clear ./setup build doesn't produce lib_convolve.py this is generated as part of the install stage. Perhaps it's an egg thing?

conda build and install works great for me, if you ignore the numpy 1.11 -> 1.15 issue and the failing erfa test:

Traceback (most recent call last):
  File "//anaconda/conda-bld/astropy_1540921946927/test_tmp/run_test.py", line 6, in <module>
    import astropy._erfa._core
ModuleNotFoundError: No module named 'astropy._erfa._core'

conda-build doesn't generate the superfluous file, lib_conovlve.py. Which is interesting since the build script only executes $PYTHON setup.py install --offline --old-and-unmanageable.

Looks like the trick is --old-and-unmanageable thus preventing the egg build. Guess it is an egg thing?

Should we split this issue out into two?

lib_convolve.py is also produced on windows via ./setup install.

Navigation Menu

Repository navigation

convolution cannot be imported #8032

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

convolution cannot be imported #8032

Description

Activity

bsipocz commented on Oct 30, 2018

bsipocz commented on Oct 30, 2018

astrofrog commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

jdavies-st commented on Oct 30, 2018

nden commented on Oct 30, 2018

astrofrog commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

bsipocz commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

jamienoss commented on Oct 30, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions