Skip to content

Commit a162c9c

Browse files
committed
Report template location for compile time warnings
Warnings raised by Python while a template is compiled, in practice SyntaxWarning, were reported against the generated module, giving a line number that has no relation to the template source; the same warning was additionally reported twice, once from parsing the individual expression, where the location is the meaningless "<unknown>", line 1, and again from compiling the module as a whole. The warnings display hook is now replaced while the template is compiled. Warnings raised for the generated module are shown against the template's filename, with the line translated through the same line map that RichTraceback uses to report the location of an exception. This is applied both to templates compiled in memory and to those compiled to a module file, where the compilation is performed by the import system and the warning would otherwise be reported against the generated module file. Warnings from the parsing of individual expressions are dropped, as the same warning is always raised again when the module is compiled, where a usable location is available; the hook that does so is in place only while the expressions are parsed, so that a warning raised by a module imported from a <%! %> block is shown unchanged even if it carries the same filename. Replacing the display hook is preferred to recording the warnings and raising them a second time, as the latter passes each warning through the filters twice. A filter with a stateful action, notably "once", suppresses the second occurrence, so the warning would be lost altogether. For the same reason the occurrence that is dropped, which the filters see first, has its entry removed from the "once" registry, else it would suppress the occurrence that can be translated. The module level code of a <%! %> block is executed within the same block, as the import system compiles and executes a module file as one step and the two paths would otherwise disagree on where a warning raised by such code is reported. Warnings raised while a template renders are unchanged, and continue to report a location within the generated module. The warnings module takes the location from the executing frame, so translating these would require replacing the display hook for the life of the process, which is not something a library should do on import. A module file that carries no usable metadata, having been written by some other means or damaged, would raise from the attempt to read its line map, from inside the display hook and therefore out of an unrelated compile() call; such a warning is now shown untranslated instead. A line that cannot be translated likewise retains its own filename, rather than being given the template's filename with the module's line. Fixes: #430 Change-Id: Ie7e942d5b0a753788de1ea012d83193be03dbe2a
1 parent 67df138 commit a162c9c

4 files changed

Lines changed: 450 additions & 20 deletions

File tree

‎doc/build/unreleased/430.rst‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
.. change::
2+
:tags: bug, exceptions
3+
:tickets: 430
4+
5+
Warnings raised while a template is compiled, which in practice means
6+
``SyntaxWarning``, now report the filename and line number of the
7+
template rather than a line within the generated module, and are no
8+
longer reported twice. This applies equally to templates compiled from a
9+
string, from a file, and to a module file in a
10+
:paramref:`.Template.module_directory`, as does a warning raised while
11+
the module level code of a ``<%! %>`` block runs. Warnings raised while
12+
a template renders continue to report the location within the generated
13+
module.
14+
15+
To do this, Mako replaces ``warnings.showwarning`` while a template is
16+
compiled. As that name is global to the process, an unrelated warning
17+
displayed by another thread during that window may also be passed through
18+
Mako's hook, which shows any warning it does not recognize unchanged.

‎mako/pyparser.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,17 @@
2020
# the "id" attribute on a function node
2121
arg_id = operator.attrgetter("arg")
2222

23+
# filename given to the compiler when an individual expression is parsed.
24+
# warnings raised against this name have no location that can be related
25+
# back to the template
26+
EXPRESSION_FILENAME = "<unknown>"
27+
2328

2429
def parse(code, mode="exec", **exception_kwargs):
2530
"""Parse an expression into AST"""
2631

2732
try:
28-
return _ast_util.parse(code, "<unknown>", mode)
33+
return _ast_util.parse(code, EXPRESSION_FILENAME, mode)
2934
except Exception as e:
3035
raise exceptions.SyntaxException(
3136
"(%s) %s (%r)"

‎mako/template.py‎

Lines changed: 141 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
"""Provides the Template class, a facade for parsing, generating and executing
88
template strings, as well as template runtime operations."""
99

10+
import contextlib
1011
from importlib import abc
1112
from importlib import machinery
1213
import json
@@ -16,12 +17,14 @@
1617
import stat
1718
import tempfile
1819
import types
20+
import warnings
1921
import weakref
2022

2123
from mako import cache
2224
from mako import codegen
2325
from mako import compat
2426
from mako import exceptions
27+
from mako import pyparser
2528
from mako import runtime
2629
from mako import util
2730
from mako.lexer import Lexer
@@ -365,21 +368,27 @@ def _compile_from_file(self, path, filename):
365368
if path is not None:
366369
util.verify_directory(os.path.dirname(path))
367370
filemtime = os.stat(filename)[stat.ST_MTIME]
368-
if (
369-
not os.path.exists(path)
370-
or os.stat(path)[stat.ST_MTIME] < filemtime
371+
with _translate_module_warnings(
372+
lambda: util.read_python_file(path), path, filename
371373
):
372-
data = util.read_file(filename)
373-
_compile_module_file(
374-
self, data, filename, path, self.module_writer
375-
)
376-
module = compat.load_module(self.module_id, path)
377-
if module._magic_number != codegen.MAGIC_NUMBER:
378-
data = util.read_file(filename)
379-
_compile_module_file(
380-
self, data, filename, path, self.module_writer
381-
)
374+
if (
375+
not os.path.exists(path)
376+
or os.stat(path)[stat.ST_MTIME] < filemtime
377+
):
378+
data = util.read_file(filename)
379+
with _drop_expression_warnings():
380+
_compile_module_file(
381+
self, data, filename, path, self.module_writer
382+
)
382383
module = compat.load_module(self.module_id, path)
384+
if module._magic_number != codegen.MAGIC_NUMBER:
385+
data = util.read_file(filename)
386+
with _drop_expression_warnings():
387+
_compile_module_file(
388+
self, data, filename, path, self.module_writer
389+
)
390+
module = compat.load_module(self.module_id, path)
391+
383392
ModuleInfo(module, path, self, filename, None, None, None)
384393
else:
385394
# template filename and no module directory, compile code
@@ -688,21 +697,134 @@ def is_package(self, name):
688697
return False
689698

690699

700+
@contextlib.contextmanager
701+
def _show_warnings_as(locate):
702+
"""Replace the warnings display hook for the duration of the block.
703+
704+
``locate`` is passed the message, category, filename and line number of
705+
each warning, and returns the filename and line number it should be
706+
shown as, or ``None`` for a warning that should not be shown at all.
707+
708+
The display hook is replaced, rather than the warnings being recorded
709+
and raised a second time, so that each warning passes through the
710+
warnings filters exactly once. A filter with a stateful action such as
711+
"once" would otherwise suppress the second occurrence, and the warning
712+
would be lost entirely.
713+
714+
"""
715+
716+
show_warning = warnings.showwarning
717+
718+
def _show(message, category, filename, lineno, file=None, line=None):
719+
location = locate(message, category, filename, lineno)
720+
if location is None:
721+
return
722+
723+
if location != (filename, lineno):
724+
# the source line, if given, is that of the original location;
725+
# allow it to be looked up again for the new one
726+
line = None
727+
filename, lineno = location
728+
729+
show_warning(message, category, filename, lineno, file, line)
730+
731+
warnings.showwarning = _show
732+
try:
733+
yield
734+
finally:
735+
warnings.showwarning = show_warning
736+
737+
738+
def _drop_expression_warnings():
739+
"""Drop warnings raised while individual expressions are parsed.
740+
741+
These carry no location that can be related back to the template, as the
742+
expression is parsed on its own, and the same warning is raised again
743+
when the module as a whole is compiled, where the location can be
744+
translated.
745+
746+
"""
747+
748+
def _locate(message, category, filename, lineno):
749+
if filename != pyparser.EXPRESSION_FILENAME:
750+
return filename, lineno
751+
752+
# a "once" filter records the warning globally as it is passed
753+
# over, so remove that record, else the occurrence that can be
754+
# translated would be suppressed
755+
warnings.onceregistry.pop((str(message), category), None)
756+
return None
757+
758+
return _show_warnings_as(_locate)
759+
760+
761+
def _translate_module_warnings(get_source, module_id, filename):
762+
"""Report warnings raised for a generated module against the template
763+
it was generated from, translating the line number through the module's
764+
line map.
765+
766+
Any other warning, such as one raised by a module imported from a
767+
``<%! %>`` block as the template module executes, is shown unchanged, as
768+
is one whose line cannot be translated.
769+
770+
"""
771+
772+
line_map = None
773+
774+
def _locate(message, category, warning_filename, lineno):
775+
nonlocal line_map
776+
777+
if warning_filename != module_id:
778+
return warning_filename, lineno
779+
780+
if line_map is None:
781+
try:
782+
line_map = ModuleInfo.get_module_source_metadata(
783+
get_source(), full_line_map=True
784+
)["full_line_map"]
785+
except Exception:
786+
# a module file that carries no usable metadata, having
787+
# been written by some other means or damaged. the warning
788+
# is worth more than the translation is
789+
line_map = []
790+
791+
try:
792+
translated = line_map[lineno - 1]
793+
except IndexError:
794+
return warning_filename, lineno
795+
else:
796+
return filename, translated
797+
798+
return _show_warnings_as(_locate)
799+
800+
691801
def _compile_text(template, text, filename):
692802
identifier = template.module_id
693-
source, lexer = _compile(
694-
template, text, filename, generate_magic_comment=False
695-
)
696803

697804
cid = identifier
698805
module = types.ModuleType(cid)
806+
807+
with _drop_expression_warnings():
808+
source, lexer = _compile(
809+
template, text, filename, generate_magic_comment=False
810+
)
811+
699812
loader = _ModuleSourceLoader(cid, source)
700813
module.__loader__ = loader
701814
module.__spec__ = machinery.ModuleSpec(cid, loader, origin=cid)
702-
code = compile(source, cid, "exec")
703815

704-
# this exec() works for 2.4->3.3.
705-
exec(code, module.__dict__, module.__dict__)
816+
with _translate_module_warnings(
817+
lambda: source, cid, filename or template.uri
818+
):
819+
code = compile(source, cid, "exec")
820+
821+
# the module body, which is the code of any <%! %> blocks, is
822+
# executed within the same block, so that a warning it raises is
823+
# reported the same way here as it is when the template is loaded
824+
# from a module file, where the import system compiles and executes
825+
# the module as one step
826+
exec(code, module.__dict__, module.__dict__)
827+
706828
return (source, module)
707829

708830

0 commit comments

Comments
 (0)