Repository navigation
fix: find call-like text in one pass and split each segment once - #21
Merged
Merged
Conversation
The call-looking pattern in `_scan_code` backtracked from every start position to the end of the string, so a long command with no parenthesis in it cost O(n^2): 100,000 characters took 39-44 s. It now scans whole word/dot runs once and asks the same question of each run. Each segment was also split by `shlex` twice, once for the local-install check and once for the destinations; the words are now computed once and passed to both. Where a segment has no quote or backslash, splitting on the shell's whitespace set gives the same words at C speed. Decisions are unchanged: `scamgate bench --strict` reports the same 24/24 table before and after, and the old pattern is kept in the tests as the reference for every sample and every line of the repository's own text.
This was referenced Oct 25, 2026
odaiameera
pushed a commit
that referenced
this pull request
Oct 25, 2026
A segment with a quote or a backslash in it still went to shlex, which builds a long token one character at a time: a 2,000,000-character command took 274 s here. _shell_words() reads the same words in one pass and raises the same errors, so every decision is unchanged. The harness compared 1,000,000 random strings word for word against shlex.split(posix=True) with no difference, and bench --strict stays 24/24, identical line for line to the branch of #21.
This was referenced Oct 26, 2026
simin-yuan
pushed a commit
to simin-yuan/scamgate
that referenced
this pull request
Oct 30, 2026
harden._script_tokens was the last shlex.split on the hook path. odaiameera#21 and odaiameera#22 made extract() linear, but the full hook call still handed every segment to shlex: 2.6 s at 400k characters, and over 400 s at 2,000,000 (the input cap is 2 MiB). It now uses hook._words, which returns exactly shlex's words (fuzzed in odaiameera#22) in one pass. hook.run on a 2,000,000-character quoted command: >200 s -> ~5 s. New tests/test_hook_perf_full.py times hook.run (not just extract) in a subprocess, and checks harden's tokens against a shlex reference on 21 samples and 3,000 random strings.
simin-yuan
pushed a commit
to simin-yuan/scamgate
that referenced
this pull request
Oct 30, 2026
Covers odaiameera#21, odaiameera#22, odaiameera#23 and odaiameera#24: banner and badge row, a note that the hook stays linear on long commands, the plugin is in the Hermes catalog, the current test count, and thanks to @simin-yuan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #20
hook.extract()spends time out of proportion to the input on a long single line: a100,000-character command took 39-44 s here, and the hook accepts up to
MAX_INPUT_BYTES(2 MiB).Two things caused it, and nothing about the decisions changes.
Changes
_code_like()instead of_CODE_LIKE = re.compile(r"[A-Za-z_][\w.]*\("). At every startposition the old pattern walked the rest of the string to find the
(, so a long run ofletters with no parenthesis cost O(n^2). The new code walks the maximal runs of
\w/.onceand asks the same question of each run: does it end on
(and does it contain a name start.That is the same test, in one pass.
_words()splits a segment once._local_installand_shell_destinationseach calledshlex.spliton the same segment. They now take the words as an argument.shlexis purePython and assembles tokens one character at a time, so on a large line it is the expensive
step; where the segment has no quote and no backslash there is nothing for the shell to remove,
and splitting on the shell's own whitespace set is the same answer at C speed. Segments with
quotes or escapes still go through
shlex.The cheap answers still come first.
_LOCAL_ONLYis checked before the split, so a localinstall is answered without reading the words at all, as before. Reading them first would hand
a 400,000-character quoted
cargo fetchtoshlexand turn an instant decision into seconds;there is a test for that ordering.
Timing
Windows 11, CPython 3.14.9,
time.perf_counter()aroundhook.extract, three runs per cell; thevalue shown is the range. Command is
"curl example.com " + body.Four times the input now costs about four times the time. A quoted local install of 400,000
characters takes 0.5 s, the cost of the other linear scans over the segment, and 1.34 s at 1 MB.
No change in decisions
uv run scamgate bench --strict: 24/24 exact decision match, and the two outputs are identicalline by line (34 lines each) before and after the patch.
hook.py,engine.pyandREADME.md, must give the new function the same answer asre.searchon the old pattern._words()is compared againstshlex.spliton hand-picked strings and on 5,000 random onesdrawn from shell-relevant characters (spaces, tabs,
\r,\n,\x0b,\x0c,\u3000,#,CJK,
|,;,$, quotes, backslash). A standalone run of the same comparison over 100,012strings found no difference.
hook.extract, no new state.Tests
tests/test_hook_perf.pyruns the timing in a subprocess with a hard timeout, because a regex stuckin the C engine cannot be interrupted from inside the same process.
within the 60 s timeout. The local-install case also passes on 0.3.1; it guards the order of the
checks in this patch rather than a speed-up, so it is not one of the six.
400k stays under 8x.
uv run pytest -q: 785 passed, 2 skipped, 1 failed. The failure istests/test_hermes_holes.py::test_other_profiles_config_is_protected, which fails the same way onunmodified 0.3.1 and is untouched here. The same counts on Python 3.11, the lowest version
requires-pythonallows.Not fixed here
A command that contains a quote or a backslash and approaches the 2 MiB cap still goes through
shlexand is still slow: 2 MB took 274 s in a single run here. A follow-up pull request willreplace
shlexwith a reader that returns the same words in one linear pass, checked againstshlexon a large differential corpus, so that case becomes linear too. It is separate so thispatch stays small and can be read quickly. The alternative, refusing to read very long segments and
handing them to review, changes a decision, so it is not in either patch; the issue has the detail
and leaves that choice to you.