soft_unicode() function: SystemError when DeprecationWarning warnings are not ignored

The "tests/test_core_tags.py::TestSet::test_block_escaping_filtered" test does fail with a Python built in debug mode if DeprecationWarning warnings are displayed using Python -X dev command line option:
https://docs.python.org/dev/library/devmode.html

The problem seems to come from the soft_unicode() function which returns a result (Markup instance) whereas there is a current exception: DeprecationWarning.

I tested Jinja 2.11.2 with Python 3.10.0a0 on Fedora 32 (x86-64).

Steps to reproduce with Python 3.8 with a regular Python binary:

wget https://files.pythonhosted.org/packages/64/a7/45e11eebf2f15bf987c3bc11d37dcc838d9dc81250e67e4c5968f6008b6c/Jinja2-2.11.2.tar.gz
tar -xf Jinja2-2.11.2.tar.gz 
cd Jinja2-2.11.2/
python3.8 -m venv env
env/bin/python setup.py install
env/bin/python -m pip install pytest
env/bin/python -W default -m pytest -v tests/test_core_tags.py

Result:

(...)
                                                                                                                                 [ 88%]
tests/test_core_tags.py::TestSet::test_namespace_loop PASSED                                                                                                                                                                                               [ 90%]
tests/test_core_tags.py::TestSet::test_namespace_macro PASSED                                                                                                                                                                                              [ 91%]
tests/test_core_tags.py::TestSet::test_block_escaping_filtered FAILED                                                                                                                                                                                      [ 93%]
tests/test_core_tags.py::TestSet::test_block_filtered FAILED                                                                                                                                                                                               [ 95%]
tests/test_core_tags.py::TestSet::test_block_filtered_set FAILED                                                                                                                                                                                           [ 96%]
tests/test_core_tags.py::TestWith::test_with PASSED                                                                                                                                                                                                        [ 98%]
tests/test_core_tags.py::TestWith::test_with_argument_scoping PASSED                                                                                                                                                                                       [100%]

============================================================================================================================ FAILURES ============================================================================================================================
______________________________________________________________________________________________________________ TestSet.test_block_escaping_filtered ______________________________________________________________________________________________________________
DeprecationWarning: 'soft_unicode' has been renamed to 'soft_str'. The old name will be removed in version 2.1.

The above exception was the direct cause of the following exception:

self = <test_core_tags.TestSet object at 0x7f9da56ed250>

    def test_block_escaping_filtered(self):
        env = Environment(autoescape=True)
        tmpl = env.from_string(
            "{% set foo | trim %}<em>{{ test }}</em>    {% endset %}foo: {{ foo }}"
        )
>       assert tmpl.render(test="<unsafe>") == "foo: <em>&lt;unsafe&gt;</em>"

tests/test_core_tags.py:553: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:1090: in render
    self.environment.handle_exception()
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:832: in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/_compat.py:28: in reraise
    raise value.with_traceback(tb)
<template>:1: in top-level template code
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = Markup('<em>&lt;unsafe&gt;</em>    '), chars = None

    def do_trim(value, chars=None):
        """Strip leading and trailing characters, by default whitespace."""
>       return soft_unicode(value).strip(chars)
E       SystemError: <built-in function soft_unicode> returned a result with an error set

env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/filters.py:828: SystemError
__________________________________________________________________________________________________________________ TestSet.test_block_filtered ___________________________________________________________________________________________________________________
DeprecationWarning: 'soft_unicode' has been renamed to 'soft_str'. The old name will be removed in version 2.1.

The above exception was the direct cause of the following exception:

self = <test_core_tags.TestSet object at 0x7f9da5f974c0>, env_trim = <jinja2.environment.Environment object at 0x7f9da5f975b0>

    def test_block_filtered(self, env_trim):
        tmpl = env_trim.from_string(
            "{% set foo | trim | length | string %} 42    {% endset %}{{ foo }}"
        )
>       assert tmpl.render() == "2"

tests/test_core_tags.py:559: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:1090: in render
    self.environment.handle_exception()
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:832: in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/_compat.py:28: in reraise
    raise value.with_traceback(tb)
<template>:1: in top-level template code
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = ' 42    ', chars = None

    def do_trim(value, chars=None):
        """Strip leading and trailing characters, by default whitespace."""
>       return soft_unicode(value).strip(chars)
E       SystemError: <built-in function soft_unicode> returned a result with an error set

env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/filters.py:828: SystemError
________________________________________________________________________________________________________________ TestSet.test_block_filtered_set _________________________________________________________________________________________________________________
DeprecationWarning: 'soft_unicode' has been renamed to 'soft_str'. The old name will be removed in version 2.1.

The above exception was the direct cause of the following exception:

self = <test_core_tags.TestSet object at 0x7f9da56e3790>, env_trim = <jinja2.environment.Environment object at 0x7f9da56e3be0>

    def test_block_filtered_set(self, env_trim):
        def _myfilter(val, arg):
            assert arg == " xxx "
            return val
    
        env_trim.filters["myfilter"] = _myfilter
        tmpl = env_trim.from_string(
            '{% set a = " xxx " %}'
            "{% set foo | myfilter(a) | trim | length | string %}"
            ' {% set b = " yy " %} 42 {{ a }}{{ b }}   '
            "{% endset %}"
            "{{ foo }}"
        )
>       assert tmpl.render() == "11"

tests/test_core_tags.py:575: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:1090: in render
    self.environment.handle_exception()
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:832: in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/_compat.py:28: in reraise
    raise value.with_traceback(tb)
<template>:1: in top-level template code
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = '  42  xxx  yy    ', chars = None

    def do_trim(value, chars=None):
        """Strip leading and trailing characters, by default whitespace."""
>       return soft_unicode(value).strip(chars)
E       SystemError: <built-in function soft_unicode> returned a result with an error set

env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/filters.py:828: SystemError

Python traceback:

(gdb) p _Py_DumpTraceback(3, tstate)
Stack (most recent call first):
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/Jinja2-2.11.2-py3.10.egg/jinja2/filters.py", line 828 in do_trim
  File "<template>", line 21 in root
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/Jinja2-2.11.2-py3.10.egg/jinja2/environment.py", line 1088 in render
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/Jinja2-2.11.2-py3.10.egg/jinja2/asyncsupport.py", line 71 in render
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/Jinja2-2.11.2/tests/test_core_tags.py", line 553 in test_block_escaping_filtered
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/_pytest/python.py", line 170 in pytest_pyfunc_call
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/pluggy/callers.py", line 187 in _multicall
(...)

Raw debug logs:

vstinner@apu$ gdb -args /home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/bin/python -X dev -m pytest --tb=short -v 
(...)
(gdb) b _Py_FatalErrorFunc 
Breakpoint 1 at 0x56ca7c: file Python/pylifecycle.c, line 2280.
(gdb) b fatal_error
Breakpoint 2 at 0x56c894: file Python/pylifecycle.c, line 2192.

(gdb) run
(...)
tests/test_core_tags.py::TestSet::test_block_escaping_filtered 
Breakpoint 1, _Py_FatalErrorFunc (func=0x70b250 <__func__.19> "_Py_CheckFunctionResult", msg=0x70aec8 "a function returned a result with an error set") at Python/pylifecycle.c:2280
2280	    fatal_error(stderr, 1, func, msg, -1);
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.8-2.fc32.x86_64 expat-2.2.8-2.fc32.x86_64 libuuid-2.35.2-1.fc32.x86_64 libxcrypt-4.4.16-3.fc32.x86_64 openssl-libs-1.1.1g-1.fc32.x86_64 xz-libs-5.2.5-1.fc32.x86_64 zlib-1.2.11-21.fc32.x86_64
(gdb) up
#1  0x00000000004301f0 in _Py_CheckFunctionResult (tstate=0x89a780, callable=0x7fffe9d63b90, result=0x7fffe8b11270, where=0x0) at Objects/call.c:61
61	            Py_FatalError("a function returned a result with an error set");
(gdb) p tstate
$1 = (PyThreadState *) 0x89a780
(gdb) p tstate->curexc_type 
$2 = (PyObject *) 0x846c20 <_PyExc_DeprecationWarning>

Thanks, looks like the expected result when turning deprecation warnings into errors. It seems that you're using a pre-release of MarkupSafe, 2.0.0a1; the current release does not yet raise this warning. We should of course update the code here to use the preferred name going forward.

It seems that you're use a pre-release of MarkupSafe, 2.0.0a1

Right. But I didn't pick this version on purpose, it's what I get by default when running python setup.py install. setup.py contains:

install_requires=["MarkupSafe>=0.23"],

If you follow my wget, tar, etc. instructions, you get a venv with these dependencies:

$ env/bin/python -m pip freeze
attrs==19.3.0
iniconfig==1.0.1
Jinja2==2.11.2
MarkupSafe==2.0.0a1
more-itertools==8.4.0
packaging==20.4
pluggy==0.13.1
py==1.9.0
pyparsing==2.4.7
pytest==6.0.1
six==1.15.0
toml==0.10.1

When I install MarkupSafe manually with pip, I get MarkupSafe 1.1.1.

$ python3.8 -m venv env
$ env/bin/python -m pip install MarkupSafe
(...)
Successfully installed MarkupSafe-1.1.1

$ env/bin/python -m pip freeze
MarkupSafe==1.1.1

setup.py install is deprecated, it does not support most modern features expected in packaging, such as python_requires or not picking pre-releases by default. Use pip install to install packages.

Navigation Menu

Repository navigation

soft_unicode() function: SystemError when DeprecationWarning warnings are not ignored #1276

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

soft_unicode() function: SystemError when DeprecationWarning warnings are not ignored #1276

Description

Activity

davidism commented on Aug 6, 2020

vstinner commented on Aug 6, 2020

davidism commented on Aug 6, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions