Repository navigation
Commit 7eb9528
fix(core): reject submatch offsets that split a multi-byte character
Decoding `raw.subarray(0, offset)` in isolation is not the same as taking a
prefix of the full decode when the offset lands inside a VALID multi-byte
sequence. `Buffer.from("éneedle")` sliced at byte 1 decodes to U+FFFD, so the
offset rebased to 3 — a plausible value pointing at the wrong character in a
line that decoded cleanly there. A byte-mode pattern can match at such a
position, so this was reachable rather than theoretical.
The prefix must now actually prefix the decoded line; an offset landing
mid-character is unaddressable and marks the record corrupt, so it is skipped
and counted like any other unusable record. Costs nothing asymptotically —
decoding the prefix was already O(offset).
A cheaper O(1) test (rejecting when the byte at the offset is a UTF-8
continuation byte) was measured against the exact check over 1.3M fuzzed
offsets and rejected: it disagrees on ~243k of them, always by over-rejecting
lines that begin with a stray continuation byte, which would drop valid matches
on binary files.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>1 parent 32cfa33 commit 7eb9528
3 files changed
Lines changed: 42 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
147 | | - | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
148 | 159 | | |
149 | 160 | | |
150 | 161 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
330 | 330 | | |
331 | 331 | | |
332 | 332 | | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
333 | 353 | | |
334 | 354 | | |
335 | 355 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
150 | | - | |
| 150 | + | |
151 | 151 | | |
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
155 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
156 | 164 | | |
157 | 165 | | |
158 | 166 | | |
| |||
0 commit comments