Exception catching failure with Python 3.7a3

While testing Astropy with Python 3.7a3 and Numpy 1.14pre I get failures with lead me to this piece of Cython code:

        for key, val in fill_values.items():
            try:
                self.fill_values[int(key)] = val
                self.fill_values[float(key)] = val
            except (ValueError, np.ma.MaskError):
                pass

where key can be a numpy.ma.core.MaskedConstant subclass (with a __hash__ value). And I get this traceback:

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    t.write('test.csv', overwrite=True)
  File "/home/simon/dev/astropy/astropy/table/table.py", line 2536, in write
    io_registry.write(self, *args, **kwargs)
  File "/home/simon/dev/astropy/astropy/io/registry.py", line 564, in write
    writer(data, *args, **kwargs)
  File "/home/simon/dev/astropy/astropy/io/ascii/connect.py", line 86, in write_csv
    return write(table, filename, **kwargs)
  File "/home/simon/dev/astropy/astropy/io/ascii/ui.py", line 880, in write
    writer.write(table, output)
  File "/home/simon/dev/astropy/astropy/io/ascii/fastbasic.py", line 194, in write
    self._write(table, output, {'fill_values': [(core.masked, '')]})
  File "/home/simon/dev/astropy/astropy/io/ascii/fastbasic.py", line 170, in _write
    writer = cparser.FastWriter(table, **write_kwargs)
SystemError: <class 'astropy.io.ascii.cparser.FastWriter'> returned NULL without setting an error

I found a number of workarounds that avoid the crash, like printing the fill_values dict before the loop, or printing a random number or str in the except clause (before pass), or maybe more interesting getting a local reference for MaskError before the loop and using except (ValueError, MaskError).

Is there a known reason for this error, which happen only with Python 3.7 ?

Using a Python recompiled in debug mode, I managed to get a crash (it is not systematic, especially when running inside gdb):

(gdb) bt
#0  0x00007ffff70e3860 in raise () from /usr/lib/libc.so.6
#1  0x00007ffff70e4ec9 in abort () from /usr/lib/libc.so.6
#2  0x00007ffff70dc0bc in __assert_fail_base () from /usr/lib/libc.so.6
#3  0x00007ffff70dc133 in __assert_fail () from /usr/lib/libc.so.6
#4  0x00005555556980e5 in _PyType_Lookup (type=0x555555b1f1c0 <PyModule_Type>, 
    name=0x7ffff0625d60) at Objects/typeobject.c:3037
#5  0x00005555556781da in _PyObject_GenericGetAttrWithDict (
    obj=0x7ffff06202d8, name=0x7ffff0625d60, dict=0x0) at Objects/object.c:1127
#6  0x0000555555678667 in PyObject_GenericGetAttr (obj=0x7ffff06202d8, 
    name=0x7ffff0625d60) at Objects/object.c:1195
#7  0x0000555555674438 in module_getattro (m=0x7ffff06202d8, 
    name=0x7ffff0625d60) at Objects/moduleobject.c:683
#8  0x00007fffed2b237c in __Pyx_PyObject_GetAttrStr (obj=0x7ffff06202d8, 
    attr_name=0x7ffff0625d60) at astropy/io/ascii/cparser.c:1302
#9  0x00007fffed30604e in __pyx_pf_7astropy_2io_5ascii_7cparser_10FastWriter___cinit__ (__pyx_v_self=0x7fffec70b948, __pyx_v_table=0x7fffed55f670, 
    __pyx_v_delimiter=0x7ffff6dd0178, 
    __pyx_v_comment=0x555555b1f820 <_Py_NoneStruct>, 
    __pyx_v_quotechar=0x7ffff66b0040, __pyx_v_expchar=0x7ffff7e5c318, 
    __pyx_v_formats=0x555555b1f820 <_Py_NoneStruct>, 
    __pyx_v_strip_whitespace=0x555555b08c80 <_Py_TrueStruct>, 
    __pyx_v_names=0x555555b1f820 <_Py_NoneStruct>, 
    __pyx_v_include_names=0x555555b1f820 <_Py_NoneStruct>, 
    =0x555555b1f820 <_Py_NoneStruct>, __pyx_v_fill_values=0x7fffec48a3a0, __pyx_v_fill_include_names=0x555555b1f820 <_Py_NoneStruct>, 
    __pyx_v_fill_exclude_names=0x555555b1f820 <_Py_NoneStruct>, __pyx_v_fast_writer=0x7fffec47d850) at astropy/io/ascii/cparser.c:17865
#10 0x00007fffed301af3 in __pyx_pw_7astropy_2io_5ascii_7cparser_10FastWriter_1__cinit__ (__pyx_v_self=0x7fffec70b948, __pyx_args=0x7ffff7e04260, __pyx_kwds=0x7fffec47d238)
    at astropy/io/ascii/cparser.c:17228
#11 0x00007fffed3297d9 in __pyx_tp_new_7astropy_2io_5ascii_7cparser_FastWriter (t=0x7fffed54cda0 <__pyx_type_7astropy_2io_5ascii_7cparser_FastWriter>, a=0x7ffff7e04260, k=0x7fffec47d238)
    at astropy/io/ascii/cparser.c:24891

The #9 (cparser.c:17865) matches the except (..., np.ma.MaskError) line, and more specifically getting MaskError from np.ma (crashes on the last line from the excerpt below):

      __pyx_t_5 = __Pyx_GetModuleGlobalName(__pyx_n_s_np); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 943, __pyx_L15_except_error)
      __Pyx_GOTREF(__pyx_t_5);
      __pyx_t_3 = __Pyx_PyObject_GetAttrStr(__pyx_t_5, __pyx_n_s_ma); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 943, __pyx_L15_except_error)
      __Pyx_GOTREF(__pyx_t_3);
      __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;
      __pyx_t_5 = __Pyx_PyObject_GetAttrStr(__pyx_t_3, __pyx_n_s_MaskError); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 943, __pyx_L15_except_error)

The generated code doesn't seem to be correctly handling exceptions raised while processing the except clauses.
Here's a snippet which causes this error, and also the opposite (value returned but an error set):

import threading  # Module with an exception class.
cdef exc(): raise SyntaxError()  # cdef, so directly called. 

def null_without_error():
    try:
        raise Exception
    except threading.ThreadError:
        pass

def value_with_error():
    try:
        raise Exception
    except exc():
        print('Crash')

In the first original error, I suspect something in the process of the module getattr is catching and resetting the error value, then returning the right value. The version-specific crash probably relates to how exceptions are now stored in thread storage.

The second crashes in 3.6 as well as 3.7.

In both cases, I think the caught exception needs to be saved off into local variables, not read directly from Python's global state (in addition to exceptions 'running' from outside the try block). Otherwise any external code could overwrite it. ceval.c stores it in the stack, so it doesn't have this problem.

Already fixed in latest master.

Great, thanks!

Hello. Exactly which commit to master fixed this? I am curious on why this error still happens on Astropy for Cython 0.28.5. Thanks!

We are currently preparing the release of Cython 0.29, which is recommended for use with Python 3.7. Just wait another week or two.

Navigation Menu

Repository navigation

Exception catching failure with Python 3.7a3 #2043

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Exception catching failure with Python 3.7a3 #2043

Description

Activity

saimn commented on Dec 14, 2017

TeamSpen210 commented on Dec 14, 2017

scoder commented on Dec 15, 2017

saimn commented on Dec 15, 2017

pllim commented on Sep 10, 2018

scoder commented on Sep 22, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions