Skip to content

Commit 8cb32e7

Browse files
sahrizviclaude
andcommitted
fix(core): drop unusable submatches instead of records; bound submatch text
Third review round — one finding from cubic, two from the codex reviewer Ralph triggered on the PR. All three validated before acting. - The `startsWith` guard on rebased offsets had an aliasing blind spot. When an invalid byte precedes a LITERAL U+FFFD, an offset inside that character still produces a decoded prefix that prefixes the line, because the replacement characters are indistinguishable — `[ff ef bf bd]` accepts offset 2 and rebases it to 6. Replaced with byte-boundary validation, which has no such blind spot. Measured over 3.4M fuzzed offsets against the definition (decoding both halves must reconstruct the whole): zero unsafe offsets accepted. It errs only conservatively, on lines beginning mid-sequence. - An offset that cannot be rebased now drops ITS SUBMATCH rather than the whole record. The file, line and text are still correct and useful, and losing a highlight range beats losing the match — which is what this branch exists to stop. Applies to out-of-range and fractional offsets too. - Bound the retained submatch text. A broad pattern such as `x.*` makes ripgrep repeat nearly the whole line in `submatches[].match.text`, so capping only `lines.text` left the retained-memory bound defeated by the submatches instead. - `parseRecords` is namespace-private again. packages/opencode/AGENTS.md requires namespace-private helpers to be non-exported top-level declarations, and it had been exported solely so tests could reach it. The legacy tests now drive the public `search()` against a stub `rg` on PATH, which covers the same skip branches without widening the public API. Tests: 21 core, 8 legacy. The four core cases and one legacy case covering the new behaviour were confirmed to fail against the previous commit. Full core suite diffed against a freshly built origin/main worktree: 26 failures on both, none introduced. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 7eb9528 commit 8cb32e7

4 files changed

Lines changed: 175 additions & 160 deletions

File tree

‎packages/core/src/ripgrep.ts‎

Lines changed: 34 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -130,32 +130,28 @@ const normalizeMatch = (json: object): unknown => {
130130
const lines = decodeField(readProp(data, "lines"))
131131
if (!lines) return json
132132
const raw = lines.raw
133-
// Rebasing is a claim about coordinates, so it is only made for an offset actually addressable in
134-
// the raw line. `Buffer.subarray` clamps an out-of-range end and truncates a fractional one rather
135-
// than throwing, so an unchecked rebase would quietly repair a corrupt offset into a
136-
// plausible-looking one — and the schema would not catch it, since `NonNegativeInt` happily
137-
// accepts a number past the end of the line. An unaddressable offset therefore marks the whole
138-
// record corrupt (undefined) so it is skipped and counted. Offsets on the `{text}` arm are left
139-
// alone: no rebasing happens there, so no claim is made and ripgrep's values stand as before.
140-
let corrupt = false
141-
const rebase = (offset: unknown): unknown => {
142-
if (!raw) return offset
143-
if (typeof offset !== "number" || !Number.isInteger(offset) || offset < 0 || offset > raw.length) {
144-
corrupt = true
145-
return offset
146-
}
147-
// Decoding the prefix in isolation is not the same as taking a prefix of the full decode when
148-
// the offset splits a VALID multi-byte sequence: `Buffer.from("éneedle")` sliced at byte 1
149-
// decodes to U+FFFD, so the offset would rebase to 3 and point at the wrong character in a line
150-
// that decoded cleanly at that spot. Requiring the prefix to actually prefix the decoded line
151-
// catches that; an offset that lands mid-character is not addressable and marks the record
152-
// corrupt. This costs nothing asymptotically — decoding the prefix is already O(offset).
153-
const prefix = raw.subarray(0, offset).toString("utf8")
154-
if (!lines.text.startsWith(prefix)) {
155-
corrupt = true
156-
return offset
157-
}
158-
return Buffer.byteLength(prefix, "utf8")
133+
// Submatch `start`/`end` are BYTE offsets into the RAW line, so a lossy decode invalidates them:
134+
// every undecodable byte widens to a 3-byte U+FFFD. They are rebased onto the decoded text's own
135+
// UTF-8 encoding, which keeps the established byte-offset contract instead of switching these
136+
// records to a different unit.
137+
//
138+
// A rebase is only sound when the split point is a character boundary of the whole-buffer decode.
139+
// Testing that by decoded-string comparison is NOT enough: for a raw line where an invalid byte
140+
// precedes a literal U+FFFD, an offset inside that literal character still produces a decoded
141+
// prefix that prefixes the line, because the replacement characters alias. Byte-boundary
142+
// validation has no such blind spot — a continuation byte (0b10xxxxxx) at the offset means the
143+
// split lands inside a sequence. Measured over 3.4M fuzzed offsets against the definition
144+
// (decoding both halves must reconstruct the whole) this never accepts an unsafe offset; it only
145+
// errs conservatively, on lines that begin mid-sequence.
146+
//
147+
// An offset that cannot be rebased drops ITS SUBMATCH, not the record: the file, line and text
148+
// are still correct and useful, and this whole change exists to stop losing matches. Offsets on
149+
// the `{text}` arm are untouched — no rebasing happens there, so no claim is made.
150+
const isContinuationByte = (byte: number | undefined) => byte !== undefined && (byte & 0xc0) === 0x80
151+
const rebase = (offset: unknown): number | undefined => {
152+
if (typeof offset !== "number" || !Number.isInteger(offset) || offset < 0 || offset > raw!.length) return undefined
153+
if (offset !== 0 && offset !== raw!.length && isContinuationByte(raw![offset])) return undefined
154+
return Buffer.byteLength(raw!.subarray(0, offset).toString("utf8"), "utf8")
159155
}
160156
const submatches = readProp(data, "submatches")
161157
const normalized = {
@@ -167,21 +163,23 @@ const normalizeMatch = (json: object): unknown => {
167163
lines: { text: capLineText(lines.text) },
168164
// Sliced BEFORE decoding so a pathological submatch count is not decoded only to be dropped.
169165
submatches: Array.isArray(submatches)
170-
? submatches.slice(0, MAX_SUBMATCHES).map((submatch) => {
171-
if (!submatch || typeof submatch !== "object") return submatch
166+
? submatches.slice(0, MAX_SUBMATCHES).flatMap((submatch) => {
167+
if (!submatch || typeof submatch !== "object") return [submatch]
172168
const match = decodeField(readProp(submatch, "match"))
173-
if (!match) return submatch
174-
return {
175-
...submatch,
176-
match: { text: match.text },
177-
start: rebase(readProp(submatch, "start")),
178-
end: rebase(readProp(submatch, "end")),
179-
}
169+
if (!match) return [submatch]
170+
// A broad pattern such as `x.*` makes ripgrep repeat almost the whole line here, so the
171+
// matched text needs the same bound as the line or the retained-memory cap is defeated
172+
// by the submatches instead.
173+
const decoded = { ...submatch, match: { text: capLineText(match.text) } }
174+
if (!raw) return [decoded]
175+
const start = rebase(readProp(submatch, "start"))
176+
const end = rebase(readProp(submatch, "end"))
177+
return start === undefined || end === undefined ? [] : [{ ...decoded, start, end }]
180178
})
181179
: submatches,
182180
},
183181
}
184-
return corrupt ? undefined : normalized
182+
return normalized
185183
}
186184
// altimate_change end
187185

‎packages/core/test/ripgrep.test.ts‎

Lines changed: 44 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -312,42 +312,73 @@ describe("Ripgrep", () => {
312312

313313
// `Buffer.subarray` clamps an out-of-range end and truncates a fractional one instead of throwing,
314314
// so rebasing without a range check would turn a corrupt offset into a plausible-looking one.
315-
stubTest("skips a record whose submatch offset is not addressable in the line", async () => {
315+
// The MATCH still survives — only the coordinate we cannot express is dropped.
316+
stubTest("drops a submatch whose offset is not addressable, keeping the match", async () => {
316317
const raw = Buffer.concat([Buffer.from([0xff]), Buffer.from("needle tail\n")])
317318
const matches = await Effect.runPromise(
318319
grepWithStubbedRecords([
319-
matchRecord("a.txt"),
320320
matchRecord("b.txt", {
321321
lines: { bytes: raw.toString("base64") },
322322
// Well past the end of the raw line — nonsense that must not be silently clamped.
323323
submatches: [{ match: { text: "needle" }, start: 1, end: 9_999 }],
324324
}),
325-
matchRecord("c.txt"),
326325
]),
327326
)
328327

329-
expect(matches.map((item) => item.entry.path)).toEqual([RelativePath.make("a.txt"), RelativePath.make("c.txt")])
330-
expect(lastSkip()?.skipped).toBe(1)
328+
expect(matches.map((item) => item.entry.path)).toEqual([RelativePath.make("b.txt")])
329+
expect(matches[0].submatches).toEqual([])
330+
expect(matches[0].text).toContain("needle")
331331
})
332332

333-
// A byte-mode pattern can match inside a valid multi-byte character. Decoding the prefix alone
334-
// then yields U+FFFD where the full decode has "é", so the offset would rebase to a plausible but
335-
// wrong position (3, landing mid-word) instead of being recognised as unaddressable.
336-
stubTest("skips a record whose submatch offset splits a multi-byte character", async () => {
333+
// A byte-mode pattern can match inside a valid multi-byte character, where the offset is simply not
334+
// expressible in the decoded line. Byte-boundary validation catches it; a decoded-string
335+
// comparison does not, because an invalid byte followed by a LITERAL U+FFFD aliases — both
336+
// prefixes decode to the same replacement text.
337+
stubTest("drops a submatch whose offset splits a multi-byte character", async () => {
337338
const raw = Buffer.concat([Buffer.from("é"), Buffer.from("needle")])
338339
const matches = await Effect.runPromise(
339340
grepWithStubbedRecords([
340-
matchRecord("a.txt"),
341341
matchRecord("b.txt", {
342342
lines: { bytes: raw.toString("base64") },
343343
submatches: [{ match: { text: "needle" }, start: 1, end: 3 }],
344344
}),
345-
matchRecord("c.txt"),
346345
]),
347346
)
348347

349-
expect(matches.map((item) => item.entry.path)).toEqual([RelativePath.make("a.txt"), RelativePath.make("c.txt")])
350-
expect(lastSkip()?.skipped).toBe(1)
348+
expect(matches[0].submatches).toEqual([])
349+
})
350+
351+
stubTest("drops a submatch whose offset sits inside a literal replacement character", async () => {
352+
// Invalid byte, then the three bytes that spell U+FFFD. Offset 2 is inside that literal
353+
// character, but both decoded prefixes read as the same replacement text.
354+
const raw = Buffer.from([0xff, 0xef, 0xbf, 0xbd])
355+
const matches = await Effect.runPromise(
356+
grepWithStubbedRecords([
357+
matchRecord("b.txt", {
358+
lines: { bytes: raw.toString("base64") },
359+
submatches: [{ match: { text: "x" }, start: 2, end: 4 }],
360+
}),
361+
]),
362+
)
363+
364+
expect(matches[0].submatches).toEqual([])
365+
})
366+
367+
// A broad pattern such as `x.*` makes ripgrep repeat nearly the whole line in the submatch text,
368+
// which would defeat the retained-memory bound that capping `lines.text` establishes.
369+
stubTest("caps the retained submatch text, not just the line", async () => {
370+
const huge = "n".repeat(50_000)
371+
const matches = await Effect.runPromise(
372+
grepWithStubbedRecords([
373+
matchRecord("a.txt", {
374+
lines: { text: huge },
375+
submatches: [{ match: { text: huge }, start: 0, end: huge.length }],
376+
}),
377+
]),
378+
)
379+
380+
expect(matches[0].submatches[0].text).toHaveLength(2_003)
381+
expect(matches[0].submatches[0].text.endsWith("...")).toBe(true)
351382
})
352383

353384
stubTest("skips a record whose bytes field is empty rather than emitting an empty match", async () => {

‎packages/opencode/src/file/ripgrep.ts‎

Lines changed: 23 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -141,26 +141,18 @@ export namespace Ripgrep {
141141
// `/find` route, so unrebased offsets would be newly wrong output rather than a skipped record.
142142
// Mirrors packages/core/src/ripgrep.ts.
143143
const raw = lines?.raw
144-
// Only rebase an offset addressable in the raw line: `Buffer.subarray` clamps an out-of-range
145-
// end and truncates a fractional one rather than throwing, so an unchecked rebase would quietly
146-
// repair a corrupt offset, and the schema would not catch it (`z.number()` accepts any number).
147-
// An unaddressable offset marks the record corrupt so it is skipped and counted instead.
148-
let corrupt = false
149-
const rebase = (offset: unknown): unknown => {
150-
if (!raw || !lines) return offset
151-
if (typeof offset !== "number" || !Number.isInteger(offset) || offset < 0 || offset > raw.length) {
152-
corrupt = true
153-
return offset
154-
}
155-
// Decoding the prefix alone differs from a prefix of the full decode when the offset splits a
156-
// VALID multi-byte sequence, so require it to actually prefix the decoded line. See
157-
// packages/core/src/ripgrep.ts for the worked example.
158-
const prefix = raw.subarray(0, offset).toString("utf8")
159-
if (!lines.text.startsWith(prefix)) {
160-
corrupt = true
161-
return offset
162-
}
163-
return Buffer.byteLength(prefix, "utf8")
144+
// Byte-boundary validation, matching packages/core/src/ripgrep.ts. A decoded-string comparison
145+
// is not sufficient: when an invalid byte precedes a LITERAL U+FFFD, an offset inside that
146+
// character still yields a prefix that prefixes the line, because the replacement characters
147+
// alias. A continuation byte (0b10xxxxxx) at the offset means the split lands inside a sequence.
148+
// An offset that cannot be rebased drops ITS SUBMATCH, not the record — the file, line and text
149+
// stay correct, and losing a highlight range beats losing the match.
150+
const isContinuationByte = (byte: number | undefined) => byte !== undefined && (byte & 0xc0) === 0x80
151+
const rebase = (offset: unknown): number | undefined => {
152+
if (!raw) return typeof offset === "number" ? offset : undefined
153+
if (typeof offset !== "number" || !Number.isInteger(offset) || offset < 0 || offset > raw.length) return undefined
154+
if (offset !== 0 && offset !== raw.length && isContinuationByte(raw[offset])) return undefined
155+
return Buffer.byteLength(raw.subarray(0, offset).toString("utf8"), "utf8")
164156
}
165157
const submatches = read(data, "submatches")
166158
// Only rewrite keys the record actually carries — `begin`/`end`/`summary` records reach here too
@@ -175,22 +167,20 @@ export namespace Ripgrep {
175167
...(lines ? { lines: { text: lines.text } } : {}),
176168
...(Array.isArray(submatches)
177169
? {
178-
submatches: submatches.map((submatch) => {
179-
if (!submatch || typeof submatch !== "object") return submatch
170+
submatches: submatches.flatMap((submatch) => {
171+
if (!submatch || typeof submatch !== "object") return [submatch]
180172
const match = decode(read(submatch, "match"))
181-
if (!match) return submatch
182-
return {
183-
...submatch,
184-
match: { text: match.text },
185-
start: rebase(read(submatch, "start")),
186-
end: rebase(read(submatch, "end")),
187-
}
173+
if (!match) return [submatch]
174+
const start = rebase(read(submatch, "start"))
175+
const end = rebase(read(submatch, "end"))
176+
if (start === undefined || end === undefined) return []
177+
return [{ ...submatch, match: { text: match.text }, start, end }]
188178
}),
189179
}
190180
: {}),
191181
},
192182
}
193-
return corrupt ? undefined : normalized
183+
return normalized
194184
}
195185

196186
/**
@@ -199,9 +189,10 @@ export namespace Ripgrep {
199189
* `JSON.parse` + a strict `Result.parse` on every line meant one unusable record threw out of
200190
* `search()` and discarded every match already collected from unrelated files — the same defect
201191
* fixed in packages/core/src/ripgrep.ts. Records are independent, so a bad one is dropped and
202-
* counted. Exported so the skip paths are testable without a stub ripgrep binary.
192+
* counted. Namespace-private per packages/opencode/AGENTS.md: the skip behaviour is covered
193+
* through the public `search()` boundary instead of exporting an implementation detail.
203194
*/
204-
export function parseRecords(lines: string[]): Match["data"][] {
195+
function parseRecords(lines: string[]): Match["data"][] {
205196
const matches: Match["data"][] = []
206197
let skipped = 0
207198
for (const line of lines) {

0 commit comments

Comments
 (0)