Repository navigation
Conversation
Skip impossible matcher retries inside colonless prefix runs while preserving exact tokenization. Fixes pygments#3226 Signed-off-by: Konstantin Baltsat <[email protected]> Assisted-by: OpenAI Codex
Author
|
Hi — this remains clean against the current base. The patch avoids repeated full-string rescans in the RDF prefixed-name path and keeps the existing tokenization behavior. Could a maintainer take a look when convenient? I can add a focused benchmark if that would help the review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3226.
Root cause
The RDF prefixed-name regex was retried at every position in a colonless prefix-character run. Each failed attempt rescanned the remaining suffix, producing quadratic work in the SPARQL, Turtle, and ShExC lexers.
Change
The shared RDF lexer now remembers the end of a failed colonless prefix run and skips only prefixed-name attempts that are already known to be impossible. All other lexer rules and the existing token stream remain unchanged.
The regression measures matcher work rather than wall-clock time. On the unpatched base, the three lexers process 8,255–8,256 candidate units for a 128-character input; after the fix, each processes 128. The test also preserves the exact pre-existing
Error/Keywordtoken stream.Validation
scripts/check_crlf.py pygments externalscripts/detect_missing_analyse_text.py --skip-no-aliasesruff check --ignore UP031 .scripts/check_sources.pyscripts/count_token_references.pyregexlint pygments.lexersgit diff --checkLicensing and assistance
I agree that this contribution is licensed under the repository's BSD-2-Clause license.
OpenAI Codex assisted with investigation, implementation, testing, and review. I am accountable for the submitted change.