[NOJIRA]: Optimize isLineContinuation for requirements.txt#180
Merged
Conversation
…ring scan isLineContinuation ran a backtracking regexp on every line of every requirements*.txt file. CPU-profiling a full scan of the dd-source monorepo attributed ~16% of total scan time (3.9s of 24.2s) to this single function, almost entirely regexp backtracking (regexp.tryBacktrack / MatchRunePos / bitState). The check is simply "does the line end in an odd number of backslashes?", which is trivial string logic. Counting trailing backslashes is behavior-identical to the old pattern (the existing pip/line-continuation.txt fixture test passes unchanged) and removes the regexp machinery entirely. Results: - micro-benchmark (representative requirements lines): 11,865 ns/op -> 5.9 ns/op - dd-source scan: CPU 24.2s -> 19.1s (-21%), wall 26.1s -> 20.5s; all regexp.* frames disappear from the profile. Co-Authored-By: Claude Opus 4.8 <[email protected]> Signed-off-by: Ryan Coulter <[email protected]>
|
🎯 Code Coverage (details) 🔗 Commit SHA: eb03395 | Docs | Datadog PR Page | Give us feedback! |
isLineContinuation for requirements.txt
dastrong
approved these changes
Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
isLineContinuation(in therequirements.txtparser) ran a backtracking regexp on every line of everyrequirements*.txtfile:The check is just "does the line end in an odd number of backslashes?". This PR replaces the regexp with a trailing-backslash count done iteratively without regexes.
Why
I CPU-profiled a full scan of the
dd-sourcemonorepo (large, Python-heavy).pprof -peekshowedregexp.(*Regexp).MatchStringwas driven 100% bypython.isLineContinuation, and it accounted for ~3.9s cumulative — ~16% of total scan CPU, almost entirely regexp backtracking.Before — top self-time,
dd-sourcescan (24.2s CPU total):After — same scan (19.1s CPU total): every
regexp.*frame is gone.Results
dd-sourcescan — CPU samplesdd-sourcescan — wall timedd-sourcescan cpu profile beforedd-sourcescan cpu profile afterCorrectness
The trailing-backslash count is behavior-identical to
([^\\]|^)(\\{2})*\\$across all inputs (empty string, leading backslash, and odd/even runs of trailing backslashes). The existingpkg/extractor/python/fixtures/pip/line-continuation.txtfixture test passes unchanged;go vetand the fullpkg/extractor/pythonsuite are green.