Skip to content

[NOJIRA]: Optimize isLineContinuation for requirements.txt#180

Merged
rjcoulter22 merged 1 commit into
mainfrom
ryan.coulter/requirements-optimization
Jun 29, 2026
Merged

[NOJIRA]: Optimize isLineContinuation for requirements.txt#180
rjcoulter22 merged 1 commit into
mainfrom
ryan.coulter/requirements-optimization

Conversation

@rjcoulter22

@rjcoulter22 rjcoulter22 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

What

isLineContinuation (in the requirements.txt parser) ran a backtracking regexp on every line of every requirements*.txt file:

re := cachedregexp.MustCompile(`([^\\]|^)(\\{2})*\\$`)
return re.MatchString(line)

The check is just "does the line end in an odd number of backslashes?". This PR replaces the regexp with a trailing-backslash count done iteratively without regexes.

trailingBackslashes := 0
for i := len(line) - 1; i >= 0 && line[i] == '\\'; i-- {
    trailingBackslashes++
}
return trailingBackslashes%2 == 1

Why

I CPU-profiled a full scan of the dd-source monorepo (large, Python-heavy). pprof -peek showed regexp.(*Regexp).MatchString was driven 100% by python.isLineContinuation, and it accounted for ~3.9s cumulative — ~16% of total scan CPU, almost entirely regexp backtracking.

Before — top self-time, dd-source scan (24.2s CPU total):

   flat  flat%        cum   cum%
 15.93s 65.77%     15.93s 65.77%  syscall.rawsyscalln
  1.93s  7.97%      2.01s  8.30%  path/filepath.matchChunk
  1.77s  7.31%      3.82s 15.77%  regexp.(*Regexp).tryBacktrack      <- isLineContinuation
  0.82s  3.39%      0.82s  3.39%  regexp/syntax.(*Inst).MatchRunePos <- isLineContinuation
  0.57s  2.35%      0.57s  2.35%  regexp.(*bitState).shouldVisit     <- isLineContinuation
  0.31s  1.28%      0.36s  1.49%  regexp.(*inputString).step         <- isLineContinuation
  0.16s  0.66%      0.17s  0.70%  regexp.(*bitState).push            <- isLineContinuation

After — same scan (19.1s CPU total): every regexp.* frame is gone.

   flat  flat%        cum   cum%
 14.94s 78.06%     14.94s 78.06%  syscall.rawsyscalln
  2.25s 11.76%      2.32s 12.12%  path/filepath.matchChunk
  0.44s  2.30%      0.44s  2.30%  path/filepath.scanChunk
  0.22s  1.15%      2.98s 15.57%  path/filepath.Match
  0.03s  0.16%      2.71s 14.16%  go-git/.../gitignore.(*pattern).simpleNameMatch

Results

Metric Before After Δ
dd-source scan — CPU samples 24.2s 19.1s −21%
dd-source scan — wall time 26.1s 20.5s −21%
What graph
dd-source scan cpu profile before pprof_before_top
dd-source scan cpu profile after pprof_after_top

Correctness

The trailing-backslash count is behavior-identical to ([^\\]|^)(\\{2})*\\$ across all inputs (empty string, leading backslash, and odd/even runs of trailing backslashes). The existing pkg/extractor/python/fixtures/pip/line-continuation.txt fixture test passes unchanged; go vet and the full pkg/extractor/python suite are green.

…ring scan

isLineContinuation ran a backtracking regexp on every line of every
requirements*.txt file. CPU-profiling a full scan of the dd-source monorepo
attributed ~16% of total scan time (3.9s of 24.2s) to this single function,
almost entirely regexp backtracking (regexp.tryBacktrack / MatchRunePos /
bitState).

The check is simply "does the line end in an odd number of backslashes?",
which is trivial string logic. Counting trailing backslashes is
behavior-identical to the old pattern (the existing pip/line-continuation.txt
fixture test passes unchanged) and removes the regexp machinery entirely.

Results:
- micro-benchmark (representative requirements lines): 11,865 ns/op -> 5.9 ns/op
- dd-source scan: CPU 24.2s -> 19.1s (-21%), wall 26.1s -> 20.5s; all
  regexp.* frames disappear from the profile.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Ryan Coulter <[email protected]>
@rjcoulter22
rjcoulter22 requested a review from a team as a code owner June 26, 2026 19:09
@datadog-official

datadog-official Bot commented Jun 26, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 84.57% (+0.40%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: eb03395 | Docs | Datadog PR Page | Give us feedback!

@rjcoulter22 rjcoulter22 changed the title perf(python): replace per-line regexp in isLineContinuation with a st… [NOJIRA]: Optimize isLineContinuation Jun 26, 2026
@rjcoulter22 rjcoulter22 changed the title [NOJIRA]: Optimize isLineContinuation [NOJIRA]: Optimize isLineContinuation for requirements.txt Jun 26, 2026
@rjcoulter22
rjcoulter22 merged commit f7a84ce into main Jun 29, 2026
11 checks passed
@rjcoulter22
rjcoulter22 deleted the ryan.coulter/requirements-optimization branch June 29, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants