Repository navigation
Commit 9dc3c20
committed
perf(security): seek token gaps in C and stop scanning confusables as a regex class
Three whole-text fast paths in the security scan, all behaviour-preserving.
1. Token-gap scanning seeks the next candidate with a compiled regex instead of
stepping through the text one character at a time in Python. Every token-gap
character lies outside printable ASCII and tab/newline/carriage-return, so the
seek class is a superset and each hit is still confirmed by the exact
predicate. Verified: no code point in Unicode is a gap character that the
seek class fails to match.
2. `_ASCII_CONFUSABLE_PATTERN` is replaced by a frozenset membership test. The
question asked of it is only ever "does this text contain any of these", and
the class holds 1,515 code points spanning 528 disjoint ranges, so as a regex
it costs a bounded scan per character. On 180 KB of ASCII prose:
regex character class 164.30 ms
range-compressed class 44.97 ms
frozenset.isdisjoint 0.70 ms
The class contains no ASCII code point at all, so ordinary text answers with
a single disjointness check. This lifts `_requires_normalized_security_view`
from 166.31 ms to 2.78 ms on that text, a 60x improvement, and the regex
accounted for 164 of those 166 ms.
3. `normalized_security_view` and the letter-spacing span scan are memoized, in
the manner of #570. The view is a frozen dataclass whose `source_offsets`
array is only ever read -- sliced, or copied into a fresh array -- so callers
can share one instance. A caller passing `check_runtime` bypasses the cached
path so runtime budgets are still enforced.
Measured on 901 real skills, two alternating runs per arm on an idle 8-core
host, against upstream main: p95 -8.8%, p99 -12.1%, total scan time -4.1%.
Findings are byte-identical (8,152 on both arms), as are coverage-ledger
outcomes (176) and error counts (0).
The corpus-level gain is much smaller than the microbenchmarks because the
predicate that improves 60x is already memoized by #570, so it runs about 23
times per scan rather than 780. It is still the single most expensive thing left
in that predicate, and the regex class would cost proportionally more on larger
files.
Tests assert the seek class covers every token-gap code point in Unicode, that
gap spans are unchanged across 1,500 randomized texts, that confusable
membership matches the original character class across 1,500 more, that the
memoized spans equal the uncached scan, and that a runtime budget is still
honoured. Full suite: 5808 passed.
Signed-off-by: Steven Moy <[email protected]>1 parent c3bb813 commit 9dc3c20
2 files changed
Lines changed: 207 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
285 | | - | |
286 | | - | |
287 | | - | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
288 | 292 | | |
289 | 293 | | |
290 | 294 | | |
| |||
464 | 468 | | |
465 | 469 | | |
466 | 470 | | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
467 | 483 | | |
468 | 484 | | |
469 | 485 | | |
470 | 486 | | |
471 | 487 | | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
472 | 505 | | |
473 | 506 | | |
474 | 507 | | |
| |||
1524 | 1557 | | |
1525 | 1558 | | |
1526 | 1559 | | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
1527 | 1567 | | |
1528 | 1568 | | |
1529 | 1569 | | |
| |||
1550 | 1590 | | |
1551 | 1591 | | |
1552 | 1592 | | |
| 1593 | + | |
| 1594 | + | |
| 1595 | + | |
| 1596 | + | |
1553 | 1597 | | |
1554 | 1598 | | |
1555 | 1599 | | |
| |||
1703 | 1747 | | |
1704 | 1748 | | |
1705 | 1749 | | |
| 1750 | + | |
1706 | 1751 | | |
1707 | | - | |
| 1752 | + | |
| 1753 | + | |
| 1754 | + | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
1708 | 1758 | | |
1709 | 1759 | | |
1710 | 1760 | | |
| |||
1987 | 2037 | | |
1988 | 2038 | | |
1989 | 2039 | | |
1990 | | - | |
| 2040 | + | |
1991 | 2041 | | |
1992 | 2042 | | |
1993 | 2043 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
0 commit comments