Skip to content

Commit 7eb9528

Browse files
sahrizviclaude
andcommitted
fix(core): reject submatch offsets that split a multi-byte character
Decoding `raw.subarray(0, offset)` in isolation is not the same as taking a prefix of the full decode when the offset lands inside a VALID multi-byte sequence. `Buffer.from("éneedle")` sliced at byte 1 decodes to U+FFFD, so the offset rebased to 3 — a plausible value pointing at the wrong character in a line that decoded cleanly there. A byte-mode pattern can match at such a position, so this was reachable rather than theoretical. The prefix must now actually prefix the decoded line; an offset landing mid-character is unaddressable and marks the record corrupt, so it is skipped and counted like any other unusable record. Costs nothing asymptotically — decoding the prefix was already O(offset). A cheaper O(1) test (rejecting when the byte at the offset is a UTF-8 continuation byte) was measured against the exact check over 1.3M fuzzed offsets and rejected: it disagrees on ~243k of them, always by over-rejecting lines that begin with a stray continuation byte, which would drop valid matches on binary files. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 32cfa33 commit 7eb9528

3 files changed

Lines changed: 42 additions & 3 deletions

File tree

‎packages/core/src/ripgrep.ts‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,18 @@ const normalizeMatch = (json: object): unknown => {
144144
corrupt = true
145145
return offset
146146
}
147-
return Buffer.byteLength(raw.subarray(0, offset).toString("utf8"), "utf8")
147+
// Decoding the prefix in isolation is not the same as taking a prefix of the full decode when
148+
// the offset splits a VALID multi-byte sequence: `Buffer.from("éneedle")` sliced at byte 1
149+
// decodes to U+FFFD, so the offset would rebase to 3 and point at the wrong character in a line
150+
// that decoded cleanly at that spot. Requiring the prefix to actually prefix the decoded line
151+
// catches that; an offset that lands mid-character is not addressable and marks the record
152+
// corrupt. This costs nothing asymptotically — decoding the prefix is already O(offset).
153+
const prefix = raw.subarray(0, offset).toString("utf8")
154+
if (!lines.text.startsWith(prefix)) {
155+
corrupt = true
156+
return offset
157+
}
158+
return Buffer.byteLength(prefix, "utf8")
148159
}
149160
const submatches = readProp(data, "submatches")
150161
const normalized = {

‎packages/core/test/ripgrep.test.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,26 @@ describe("Ripgrep", () => {
330330
expect(lastSkip()?.skipped).toBe(1)
331331
})
332332

333+
// A byte-mode pattern can match inside a valid multi-byte character. Decoding the prefix alone
334+
// then yields U+FFFD where the full decode has "é", so the offset would rebase to a plausible but
335+
// wrong position (3, landing mid-word) instead of being recognised as unaddressable.
336+
stubTest("skips a record whose submatch offset splits a multi-byte character", async () => {
337+
const raw = Buffer.concat([Buffer.from("é"), Buffer.from("needle")])
338+
const matches = await Effect.runPromise(
339+
grepWithStubbedRecords([
340+
matchRecord("a.txt"),
341+
matchRecord("b.txt", {
342+
lines: { bytes: raw.toString("base64") },
343+
submatches: [{ match: { text: "needle" }, start: 1, end: 3 }],
344+
}),
345+
matchRecord("c.txt"),
346+
]),
347+
)
348+
349+
expect(matches.map((item) => item.entry.path)).toEqual([RelativePath.make("a.txt"), RelativePath.make("c.txt")])
350+
expect(lastSkip()?.skipped).toBe(1)
351+
})
352+
333353
stubTest("skips a record whose bytes field is empty rather than emitting an empty match", async () => {
334354
const matches = await Effect.runPromise(
335355
grepWithStubbedRecords([matchRecord("a.txt"), matchRecord("b.txt", { lines: { bytes: "" } })]),

‎packages/opencode/src/file/ripgrep.ts‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,12 +147,20 @@ export namespace Ripgrep {
147147
// An unaddressable offset marks the record corrupt so it is skipped and counted instead.
148148
let corrupt = false
149149
const rebase = (offset: unknown): unknown => {
150-
if (!raw) return offset
150+
if (!raw || !lines) return offset
151151
if (typeof offset !== "number" || !Number.isInteger(offset) || offset < 0 || offset > raw.length) {
152152
corrupt = true
153153
return offset
154154
}
155-
return Buffer.byteLength(raw.subarray(0, offset).toString("utf8"), "utf8")
155+
// Decoding the prefix alone differs from a prefix of the full decode when the offset splits a
156+
// VALID multi-byte sequence, so require it to actually prefix the decoded line. See
157+
// packages/core/src/ripgrep.ts for the worked example.
158+
const prefix = raw.subarray(0, offset).toString("utf8")
159+
if (!lines.text.startsWith(prefix)) {
160+
corrupt = true
161+
return offset
162+
}
163+
return Buffer.byteLength(prefix, "utf8")
156164
}
157165
const submatches = read(data, "submatches")
158166
// Only rewrite keys the record actually carries — `begin`/`end`/`summary` records reach here too

0 commit comments

Comments
 (0)