Repository navigation
Description
Per the guidance in doc/docs/security.rst ("When reporting a bug where you
suspect super-linear execution times, please make sure to attach an input to
reproduce it") — here is a reproducing input and the measured scaling.
Affected: SparqlLexer, TurtleLexer, ShExCLexer (pygments/lexers/rdf.py)
Tested against: Pygments 2.20.0 (current PyPI release)
Reproduction
```python
import time
from pygments.lexers.rdf import SparqlLexer, TurtleLexer, ShExCLexer
for LexerCls in [SparqlLexer, TurtleLexer, ShExCLexer]:
lexer = LexerCls()
print(f"--- {LexerCls.name} ---")
prev = None
for n in [1000, 2000, 4000, 8000, 16000]:
payload = "a" * n
t0 = time.perf_counter()
list(lexer.get_tokens(payload))
dt = time.perf_counter() - t0
ratio = f" (x{dt/prev:.2f})" if prev else ""
print(f" n={n:6d} time={dt:8.4f}s{ratio}")
prev = dt
```
Measured on my machine, pygments 2.20.0:
| input length (chars) | SparqlLexer | TurtleLexer | ShExCLexer |
|---|---|---|---|
| 1,000 | 0.020 s | 0.018 s | 0.019 s |
| 2,000 | 0.070 s | 0.065 s | 0.065 s |
| 4,000 | 0.258 s | 0.244 s | 0.253 s |
| 8,000 | 0.971 s | 0.973 s | 0.965 s |
| 16,000 | 3.896 s | — | — |
| 24,000 | 8.723 s | — | — |
Time roughly quadruples each time input length doubles. A log-log regression
of the SparqlLexer series above gives a fitted exponent of 1.92 (i.e.
close to O(n^2), not linear, and not exponential either — the fitted exponent
is the actual measurement, not an assumed worst case).
The input is a plain run of ASCII letters with no terminating :, so it never
completes a valid "prefixed name" token — no unusual characters, no crafted
Unicode, just an ordinary-looking string that happens to never resolve.
Root cause (as far as I can tell)
SparqlLexer, TurtleLexer, and ShExCLexer each build their "prefixed
name" pattern from the same two byte-for-byte-identical regex fragments:
```python
PN_PREFIX = PN_CHARS_BASE + '(?:[' + PN_CHARS_GRP + '.]' + PN_CHARS + ')?'
PN_LOCAL = ('(?:[' + PN_CHARS_U_GRP + ':0-9' + ']|' + PLX + ')' +
'(?:(?:[' + PN_CHARS_GRP + '.:]|' + PLX + ')(?:[' +
PN_CHARS_GRP + ':]|' + PLX + '))?')
```
(current locations: SparqlLexer — lines 76 and 87; TurtleLexer — lines
216 and 232; ShExCLexer — lines 383 and 391.)
Both have the shape (?:X*Y)?, where X and Y match overlapping character
classes. When the surrounding rule never completes a match, the engine has
O(n) ways to divide a matching run of length n between the starred group and
the final group, and RegexLexer's driver retries the rule set at each of
the n starting offsets as it advances — the combination is what produces the
measured quadratic. I haven't attempted a fix; a length cap before the full
pattern runs, or an atomic/possessive-group rewrite that removes the
backtracking ambiguity, both seem like plausible directions, but the right
fix needs to preserve the grammar exactly and that's better judged here than
guessed at by me.
How I found this
This came out of an automated study that has several LLM-based code-review
passes read library source and report suspected defects, each only counted
once reproduced end-to-end against the real source — the script above is the
actual reproduction, not a description written after the fact. Happy to
answer questions about the methodology, but the finding should be judged on
whether the numbers above reproduce on your machine, not on how it was found.
Activity
frenzymadness commentedon Aug 17, 2026
I've got an AI-powered report of similar issues in
CommonLispLexerandMapleLexer.Reproducer for
CommonLispLexer(there is a similar issue inEmacsLispLexertoo):Produces the following:
The problem is here:
pygments/pygments/lexers/lisp.py
Lines 311 to 312 in 19c5817
The
*+-/fragment creates a character range that unintentionally includes.. This allows.to match through two overlapping paths inside(?:{constituent})*: vianonmacroand via the explicit#.:branch. When the lexer evaluates ther'*' + symbol + r'*'rule on an input such as".....\n"with no closing ``, Python's backtracking regex engine explores many equivalent paths before failing the terminating*, leading to excessive CPU consumption. The generic `symbol` rule follows, but the expensive failed match occurs first.And for
MapleLexer:Result:
The practical problem is the quote rule:
r"('+)((.|\n)*?)\1"combines a backreference with a broad wildcard across potentially large input, which leads to pathological runtime on crafted quote-heavy strings.