Skip to content

Commit e983649

Browse files
sahrizviclaude
andcommitted
fix(core): reject text-arm offsets that split a character
Sixth review round — cubic and Kilo on the previous commit, both pointing at the `{text}`-arm validation that commit had just added. - That validation checked the range but not the character boundary, so a byte offset landing inside a multi-byte sequence was accepted: for the line `éa` (3 bytes) a corrupt `start: 1` splits `é`, and a consumer slicing the line's UTF-8 encoding there gets invalid bytes. The `{bytes}` arm already rejected exactly that shape. Both arms now apply the same continuation-byte check. ripgrep never emits a mid-boundary offset for a valid-UTF-8 line, so this drops nothing legitimate — a boundary-aligned offset on the same line is kept, which the tests assert alongside the rejection. The line is encoded once per record and shared by every submatch. That replaces the `Buffer.byteLength` walk the previous commit added rather than stacking on top of it; the extra cost is the allocation. - Fixed a comment stale as of the previous commit: it still said `{text}`-arm offsets are untouched and that no claim is made about them, which the validation added in that same commit contradicts. - Gave the >16 MiB record test an explicit 30s timeout. It materialises the record, so it is slow enough to trip the default when the suite runs under load; it failed once that way locally while another suite ran concurrently, then passed 3/3 in isolation. Tests: 24 core, 14 legacy. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent c95f234 commit e983649

4 files changed

Lines changed: 72 additions & 22 deletions

File tree

‎packages/core/src/ripgrep.ts‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -145,15 +145,26 @@ const normalizeMatch = (json: object): unknown => {
145145
// errs conservatively, on lines that begin mid-sequence.
146146
//
147147
// An offset that cannot be rebased drops ITS SUBMATCH, not the record: the file, line and text
148-
// are still correct and useful, and this whole change exists to stop losing matches. Offsets on
149-
// the `{text}` arm are untouched — no rebasing happens there, so no claim is made.
148+
// are still correct and useful, and this whole change exists to stop losing matches. `{text}`-arm
149+
// offsets are not rebased — there is nothing to rebase onto — but they are validated the same
150+
// way, because returning a coordinate pair that indexes nothing, or that splits a character, is
151+
// itself a claim.
150152
const isContinuationByte = (byte: number | undefined) => byte !== undefined && (byte & 0xc0) === 0x80
151-
const lineBytes = Buffer.byteLength(lines.text, "utf8")
153+
// Encoded once per record and shared by every submatch. This also supplies the byte length, so it
154+
// replaces a `Buffer.byteLength` walk rather than adding one; the extra cost is the allocation.
155+
const textBytes = raw ?? Buffer.from(lines.text, "utf8")
152156
const rebase = (offset: unknown): number | undefined => {
153-
// `{text}` arm: nothing is rebased, but the offset must still be addressable in the line it
154-
// indexes, or the record carries a coordinate pair that points at nothing.
157+
// `{text}` arm: nothing is rebased, but the offset must still be addressable AND land on a
158+
// character boundary. A byte offset can fall inside a multi-byte sequence — for `éa` the byte
159+
// offset 1 splits `é` — and a consumer slicing the line's UTF-8 encoding there gets invalid
160+
// bytes. ripgrep never emits such an offset for a valid-UTF-8 line, so rejecting it drops
161+
// nothing legitimate. Same rule the `{bytes}` arm applies below.
155162
if (!raw)
156-
return typeof offset === "number" && Number.isInteger(offset) && offset >= 0 && offset <= lineBytes
163+
return typeof offset === "number" &&
164+
Number.isInteger(offset) &&
165+
offset >= 0 &&
166+
offset <= textBytes.length &&
167+
!(offset !== 0 && offset !== textBytes.length && isContinuationByte(textBytes[offset]))
157168
? offset
158169
: undefined
159170
if (typeof offset !== "number" || !Number.isInteger(offset) || offset < 0 || offset > raw.length) return undefined

‎packages/core/test/ripgrep.test.ts‎

Lines changed: 30 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -248,17 +248,23 @@ describe("Ripgrep", () => {
248248

249249
// The size ceiling is asserted on a record built to exceed it, rather than inferred from a large
250250
// file — that keeps the case independent of whether a given ripgrep build emits the match at all.
251-
stubTest("skips an oversized record and keeps parsing the records after it", async () => {
252-
const matches = await Effect.runPromise(
253-
grepWithStubbedRecords([
254-
matchRecord("a.txt"),
255-
matchRecord("b-huge.txt", { lines: { text: "needle" + "x".repeat(17 * 1024 * 1024) } }),
256-
matchRecord("c.txt"),
257-
]),
258-
)
259-
260-
expect(matches.map((item) => item.entry.path)).toEqual([RelativePath.make("a.txt"), RelativePath.make("c.txt")])
261-
})
251+
stubTest(
252+
"skips an oversized record and keeps parsing the records after it",
253+
async () => {
254+
const matches = await Effect.runPromise(
255+
grepWithStubbedRecords([
256+
matchRecord("a.txt"),
257+
matchRecord("b-huge.txt", { lines: { text: "needle" + "x".repeat(17 * 1024 * 1024) } }),
258+
matchRecord("c.txt"),
259+
]),
260+
)
261+
262+
expect(matches.map((item) => item.entry.path)).toEqual([RelativePath.make("a.txt"), RelativePath.make("c.txt")])
263+
// Explicit timeout: this materialises a >16 MiB record, so it is slow enough to trip the default
264+
// under load even though the parser rejects the record before parsing it.
265+
},
266+
30_000,
267+
)
262268

263269
// A path is an identifier the caller reopens, so it must never be lossily decoded. Such a record
264270
// is skipped rather than reported under a U+FFFD-mangled path that names no real file.
@@ -362,6 +368,19 @@ describe("Ripgrep", () => {
362368
expect(matches[0].submatches).toEqual([])
363369
})
364370

371+
stubTest("drops a text-arm submatch whose offset splits a multi-byte character", async () => {
372+
const matches = await Effect.runPromise(
373+
grepWithStubbedRecords([
374+
matchRecord("a.txt", { lines: { text: "éa" }, submatches: [{ match: { text: "a" }, start: 1, end: 3 }] }),
375+
matchRecord("b.txt", { lines: { text: "éa" }, submatches: [{ match: { text: "a" }, start: 2, end: 3 }] }),
376+
]),
377+
)
378+
379+
// Offset 1 splits `é`; offset 2 is the character boundary, so that submatch is kept.
380+
expect(matches[0].submatches).toEqual([])
381+
expect(matches[1].submatches).toHaveLength(1)
382+
})
383+
365384
// Endpoints are rebased independently, so an inverted range can survive both endpoint checks.
366385
stubTest("drops a submatch whose range is inverted", async () => {
367386
const matches = await Effect.runPromise(

‎packages/opencode/src/file/ripgrep-records.ts‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -157,13 +157,19 @@ const normalizeRecord = (line: string): unknown => {
157157
// An offset that cannot be rebased drops ITS SUBMATCH, not the record — the file, line and text
158158
// stay correct, and losing a highlight range beats losing the match.
159159
const isContinuationByte = (byte: number | undefined) => byte !== undefined && (byte & 0xc0) === 0x80
160-
const lineBytes = lines ? Buffer.byteLength(lines.text, "utf8") : 0
160+
// Encoded once per record and shared by every submatch; also supplies the byte length.
161+
const textBytes = raw ?? (lines ? Buffer.from(lines.text, "utf8") : Buffer.alloc(0))
161162
const rebase = (offset: unknown): number | undefined => {
162-
// `{text}` arm: nothing is rebased, but the offset must still be addressable in the line it
163-
// indexes. `z.number()` accepts negatives, fractions and values past the end, so without this a
164-
// corrupt record reaches the `/find` response with coordinates that index nothing.
163+
// `{text}` arm: nothing is rebased, but the offset must still be addressable AND land on a
164+
// character boundary — a byte offset can fall inside a multi-byte sequence (`éa`, offset 1
165+
// splits `é`). `z.number()` rejects none of that, so a corrupt record would otherwise reach the
166+
// `/find` response with coordinates that index nothing or half a character.
165167
if (!raw)
166-
return typeof offset === "number" && Number.isInteger(offset) && offset >= 0 && offset <= lineBytes
168+
return typeof offset === "number" &&
169+
Number.isInteger(offset) &&
170+
offset >= 0 &&
171+
offset <= textBytes.length &&
172+
!(offset !== 0 && offset !== textBytes.length && isContinuationByte(textBytes[offset]))
167173
? offset
168174
: undefined
169175
if (typeof offset !== "number" || !Number.isInteger(offset) || offset < 0 || offset > raw.length) return undefined

‎packages/opencode/test/file/ripgrep-records.test.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,20 @@ describe("RipgrepRecords.parseRecords", () => {
132132
expect(fractional[0].submatches).toEqual([])
133133
})
134134

135+
// A byte offset can fall inside a multi-byte character even on a perfectly valid UTF-8 line.
136+
test("drops a text-arm submatch whose offset splits a multi-byte character", () => {
137+
const parsed = RipgrepRecords.parseRecords([
138+
record("a.txt", { lines: { text: "éa" }, submatches: [{ match: { text: "a" }, start: 1, end: 3 }] }),
139+
])
140+
expect(parsed[0].submatches).toEqual([])
141+
142+
// The same line with boundary-aligned offsets is kept.
143+
const ok = RipgrepRecords.parseRecords([
144+
record("a.txt", { lines: { text: "éa" }, submatches: [{ match: { text: "a" }, start: 2, end: 3 }] }),
145+
])
146+
expect(ok[0].submatches).toHaveLength(1)
147+
})
148+
135149
// Each submatch costs a rebase per endpoint, and a rebase allocates a string up to the line
136150
// length, so an unbounded array turns one in-ceiling record into O(count x line) work.
137151
test("bounds the submatch count like the core parser", () => {

0 commit comments

Comments
 (0)