Skip to content

Super-linear (quadratic) execution time in SparqlLexer/TurtleLexer/ShExCLexer on adversarial input #3226

Open

Description

@pontusj101

Per the guidance in doc/docs/security.rst ("When reporting a bug where you
suspect super-linear execution times, please make sure to attach an input to
reproduce it") — here is a reproducing input and the measured scaling.

Affected: SparqlLexer, TurtleLexer, ShExCLexer (pygments/lexers/rdf.py)
Tested against: Pygments 2.20.0 (current PyPI release)

Reproduction

```python
import time
from pygments.lexers.rdf import SparqlLexer, TurtleLexer, ShExCLexer

for LexerCls in [SparqlLexer, TurtleLexer, ShExCLexer]:
lexer = LexerCls()
print(f"--- {LexerCls.name} ---")
prev = None
for n in [1000, 2000, 4000, 8000, 16000]:
payload = "a" * n
t0 = time.perf_counter()
list(lexer.get_tokens(payload))
dt = time.perf_counter() - t0
ratio = f" (x{dt/prev:.2f})" if prev else ""
print(f" n={n:6d} time={dt:8.4f}s{ratio}")
prev = dt
```

Measured on my machine, pygments 2.20.0:

input length (chars) SparqlLexer TurtleLexer ShExCLexer
1,000 0.020 s 0.018 s 0.019 s
2,000 0.070 s 0.065 s 0.065 s
4,000 0.258 s 0.244 s 0.253 s
8,000 0.971 s 0.973 s 0.965 s
16,000 3.896 s — —
24,000 8.723 s — —

Time roughly quadruples each time input length doubles. A log-log regression
of the SparqlLexer series above gives a fitted exponent of 1.92 (i.e.
close to O(n^2), not linear, and not exponential either — the fitted exponent
is the actual measurement, not an assumed worst case).

The input is a plain run of ASCII letters with no terminating :, so it never
completes a valid "prefixed name" token — no unusual characters, no crafted
Unicode, just an ordinary-looking string that happens to never resolve.

Root cause (as far as I can tell)

SparqlLexer, TurtleLexer, and ShExCLexer each build their "prefixed
name" pattern from the same two byte-for-byte-identical regex fragments:

```python
PN_PREFIX = PN_CHARS_BASE + '(?:[' + PN_CHARS_GRP + '.]' + PN_CHARS + ')?'
PN_LOCAL = ('(?:[' + PN_CHARS_U_GRP + ':0-9' + ']|' + PLX + ')' +
'(?:(?:[' + PN_CHARS_GRP + '.:]|' + PLX + ')
(?:[' +
PN_CHARS_GRP + ':]|' + PLX + '))?')
```

(current locations: SparqlLexer — lines 76 and 87; TurtleLexer — lines
216 and 232; ShExCLexer — lines 383 and 391.)

Both have the shape (?:X*Y)?, where X and Y match overlapping character
classes. When the surrounding rule never completes a match, the engine has
O(n) ways to divide a matching run of length n between the starred group and
the final group, and RegexLexer's driver retries the rule set at each of
the n starting offsets as it advances — the combination is what produces the
measured quadratic. I haven't attempted a fix; a length cap before the full
pattern runs, or an atomic/possessive-group rewrite that removes the
backtracking ambiguity, both seem like plausible directions, but the right
fix needs to preserve the grammar exactly and that's better judged here than
guessed at by me.

How I found this

This came out of an automated study that has several LLM-based code-review
passes read library source and report suspected defects, each only counted
once reproduced end-to-end against the real source — the script above is the
actual reproduction, not a description written after the fact. Happy to
answer questions about the methodology, but the finding should be judged on
whether the numbers above reproduce on your machine, not on how it was found.

Activity

  1. frenzymadness commented on Aug 17, 2026

    @frenzymadness

    I've got an AI-powered report of similar issues in CommonLispLexer and MapleLexer.

    Reproducer for CommonLispLexer (there is a similar issue in EmacsLispLexer too):

    import sys, time
    from pygments import lex
    from pygments.lexers.lisp import CommonLispLexer
    
    for n in 10, 20, 25, 30:
        s = "*" + "." * n + "\n"   # no closing '*'
        t0 = time.perf_counter()
        list(lex(s, CommonLispLexer()))
        print(n, round(time.perf_counter() - t0, 3), "s")

    Produces the following:

    10 0.004 s
    20 0.057 s
    25 1.553 s
    30 55.299 s
    

    The problem is here:

    nonmacro = r'\\.|[\w!$%&*+-/<=>?@\[\]^{}~]'
    constituent = nonmacro + '|[#.:]'

    The *+-/ fragment creates a character range that unintentionally includes .. This allows . to match through two overlapping paths inside (?:{constituent})*: via nonmacro and via the explicit #.: branch. When the lexer evaluates the r'*' + symbol + r'*' rule on an input such as ".....\n" with no closing ``, Python's backtracking regex engine explores many equivalent paths before failing the terminating *, leading to excessive CPU consumption. The generic `symbol` rule follows, but the expensive failed match occurs first.

    And for MapleLexer:

    import time
    from pygments.lexers.maple import MapleLexer
    
    lx = MapleLexer()
    
    for n in [1000, 2000, 4000, 8000]:
        s = "'" * n + "x"
        t0 = time.perf_counter()
        list(lx.get_tokens_unprocessed(s))
        print(n, round(time.perf_counter() - t0, 3), "s")

    Result:

    1000 0.013 s
    2000 0.094 s
    4000 0.787 s
    8000 5.207 s
    

    The practical problem is the quote rule: r"('+)((.|\n)*?)\1" combines a backreference with a broad wildcard across potentially large input, which leads to pathological runtime on crafted quote-heavy strings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions