Hy doesn't lex unicode literals #1126
Description
(Original issue 1422 created by vodik on 2018-03-04T23:26:20.628206+00:00)
I've updated the Hy documentation to use hylang in its code blocks, and we've found a few places where it fails to lex valid Hy code. Most of the problems revolve around REPL output, which is understandable (trying to figure out what's the best practice here), but there are a few snippets that fail.
Consider the following snippet:
=> (deftag ↻ [code]
... (setv op (last code) params (list (butlast code)))
... `(~op ~@params))
=> #↻(1 2 3 +)
6When pushing this through pymentize, I hit this:
*****************************************************************
An unhandled exception occurred while highlighting.
Please report the whole traceback to the issue tracker at
<https://bitbucket.org/birkenfeld/pygments-main/issues>.
*****************************************************************
=> (deftag Traceback (most recent call last):
File "/home/simon/.virtualenvs/hy-eeIqtnUk/bin/pygmentize", line 11, in <module>
load_entry_point('Pygments', 'console_scripts', 'pygmentize')()
File "/home/simon/src/pygments-main/pygments/cmdline.py", line 548, in main
return main_inner(popts, args, usage)
File "/home/simon/src/pygments-main/pygments/cmdline.py", line 512, in main_inner
highlight(code, lexer, fmter, outfile)
File "/home/simon/src/pygments-main/pygments/__init__.py", line 85, in highlight
return format(lex(code, lexer), formatter, outfile)
File "/home/simon/src/pygments-main/pygments/__init__.py", line 67, in format
formatter.format(tokens, outfile)
File "/home/simon/src/pygments-main/pygments/formatters/terminal.py", line 101, in format
return Formatter.format(self, tokensource, outfile)
File "/home/simon/src/pygments-main/pygments/formatter.py", line 95, in format
return self.format_unencoded(tokensource, outfile)
File "/home/simon/src/pygments-main/pygments/formatters/terminal.py", line 121, in format_unencoded
for ttype, value in tokensource:
File "/home/simon/src/pygments-main/pygments/filter.py", line 20, in _apply
for token in filter_.filter(lexer, stream):
File "/home/simon/src/pygments-main/pygments/filters/__init__.py", line 196, in filter
raise self.exception(value)
pygments.filters.ErrorToken: ↻
Link to Hy issue: hylang/hy#1522