Skip to content

Commit 3b9c563

Browse files
committed
One way to publish a pair, and one bound that carries its own sentence
Integrates the two actionable comments of the first-green review on PR 997. Every caller of `publishPair` passed an empty prefix once the upgrade moved to `copyEntryRecords`, so its body, origin, append branch and the paragraph on why bytes rather than events are copied described a path nothing took. It is now `establishPair` followed by `acknowledgePair`, that rationale sits on `copyEntryRecords`, which is what copies bytes, and the three call sites lose an argument. One way to create a pair, and the copy is the only thing that copies. `copyEntryRecords` passed an infinite bound, which made its `exceeded` sentence unreachable: the file it reads is one `readRecords` has already opened under the supported limit. A bound now arrives as the units together with the refusal that names them, and a reader with no bound passes none and carries no sentence for a refusal that cannot happen. The upgrade creates the pair before it reads the older file, so a refused or cancelled copy leaves an interrupted publication that opening refuses, with the older file still holding everything. `upgradeExecution` now says so, and names the one refusal a history this process has already opened can reach: the copy decodes strictly where `readRecords` decodes leniently. A row covers it — a legacy history with a byte that begins no UTF-8 sequence opens as one replacement character, refuses its upgrade by name, keeps every byte, and reopens as an interrupted publication. Lenient opening is unchanged: a history that is not UTF-8 still opens, which is the behavior the review's other option would have changed.
1 parent 4657c4d commit 3b9c563

6 files changed

Lines changed: 100 additions & 55 deletions

File tree

‎packages/cli/src/repl/journal.ts‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -226,19 +226,21 @@ export function* readRecords(
226226
yield* forEachLine(
227227
path,
228228
{
229-
limit,
230-
refusals: {
231-
unterminated: () =>
232-
new ReplJournalError(
233-
"this execution's history ends in a record that was never finished writing.",
234-
),
229+
limit: {
230+
units: limit,
235231
exceeded: () =>
236232
new ReplJournalError(
237233
`this execution's history exceeds the supported limit of ` +
238234
`${limit.toLocaleString("en-US")} UTF-16 code units. Start a new execution with ` +
239235
"`xmd repl` for further work; this history was left unchanged.",
240236
),
241237
},
238+
refusals: {
239+
unterminated: () =>
240+
new ReplJournalError(
241+
"this execution's history ends in a record that was never finished writing.",
242+
),
243+
},
242244
},
243245
function* (line): Operation<void> {
244246
const parsed = parseDurableEvent(line);

‎packages/cli/src/repl/record-file.ts‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -87,22 +87,33 @@ function ignore(): undefined {
8787
export interface LineRefusals {
8888
/** The file's last line has no newline, so it was never finished writing. */
8989
unterminated(): Error;
90-
/** The file holds more text than `limit` allows. */
91-
exceeded(): Error;
9290
/** A line is not UTF-8, when decoding is strict. */
9391
undecodable?(): Error;
9492
}
9593

96-
export interface LineOptions {
94+
/**
95+
* How much text a read may decode, and the refusal that names the bound.
96+
*
97+
* The two together, because a bound nobody can state is a bound nobody can be
98+
* told about: a reader with no bound to state passes none, and then carries no
99+
* sentence for a refusal that cannot happen.
100+
*/
101+
export interface LineBound {
97102
/**
98103
* The most UTF-16 code units the file may decode to, newlines included.
99104
*
100105
* Counted as the file is read and checked before each piece of a line joins
101-
* the rest, so a file past the limit is refused without building a string
106+
* the rest, so a file past the bound is refused without building a string
102107
* past it. Code units rather than bytes, because the ceiling this guards is a
103108
* string length and a byte count is a different number for the same text.
104109
*/
105-
readonly limit: number;
110+
readonly units: number;
111+
exceeded(): Error;
112+
}
113+
114+
export interface LineOptions {
115+
/** What the decoded text may not exceed, where the caller has a bound. */
116+
readonly limit?: LineBound;
106117
/** Refuse bytes that are not UTF-8, rather than replacing them. */
107118
readonly strict?: boolean;
108119
readonly refusals: LineRefusals;
@@ -132,6 +143,7 @@ export function forEachLine(
132143
fatal: options.strict === true,
133144
ignoreBOM: true,
134145
});
146+
const bound = options.limit;
135147
let pieces: string[] = [];
136148
let units = 0;
137149
let lines = 0;
@@ -145,8 +157,8 @@ export function forEachLine(
145157
throw options.refusals.undecodable?.() ?? error;
146158
}
147159
units += text.length;
148-
if (units > options.limit) {
149-
throw options.refusals.exceeded();
160+
if (bound !== undefined && units > bound.units) {
161+
throw bound.exceeded();
150162
}
151163
if (text.length > 0) {
152164
pieces.push(text);
@@ -161,8 +173,8 @@ export function forEachLine(
161173
// settled here rather than carried into the next line.
162174
decode(chunk.subarray(start, end), false);
163175
units += 1;
164-
if (units > options.limit) {
165-
throw options.refusals.exceeded();
176+
if (bound !== undefined && units > bound.units) {
177+
throw bound.exceeded();
166178
}
167179
const line = pieces.join("");
168180
pieces = [];

‎packages/cli/src/repl/sidekick/history.ts‎

Lines changed: 29 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1305,44 +1305,19 @@ export interface Publication {
13051305
}
13061306

13071307
/**
1308-
* Publish a pair for an execution that has none, copying an exact prefix.
1308+
* Publish an empty pair, for an execution whose history starts here.
13091309
*
1310-
* The order is the contract. Both files are created exclusively, the copied
1311-
* prefix is appended record by record so the new entry file holds exactly the
1312-
* records the old one held, the owner naming that exact origin is written, and
1313-
* `ready` is written last — so a crash anywhere before `ready` leaves a
1314-
* directory that refuses rather than a pair that looks usable, and leaves the
1315-
* original recoverable and untouched.
1310+
* The order is the contract: both files are created exclusively, the owner is
1311+
* written, and `ready` is written last — so a crash anywhere before `ready`
1312+
* leaves a directory that refuses rather than a pair that looks usable.
13161313
*
1317-
* Nothing here executes a copied record. They are a person's history, not a
1318-
* migration script.
1314+
* An execution that grew out of an older single file is published by
1315+
* {@link upgradeExecution} instead, which copies that file's bytes between
1316+
* these same two steps and names what it copied in the owner record.
13191317
*/
1320-
export function* publishPair(
1321-
root: string,
1322-
execution: string,
1323-
prefix: {
1324-
/**
1325-
* The acknowledged bytes to copy, exactly as the older file holds them.
1326-
*
1327-
* The bytes rather than the events, because re-encoding parsed events is
1328-
* not copying them: `parseDurableEvent` rebuilds an envelope in its own
1329-
* member order and `serializeDurableEvent` preserves whatever order it is
1330-
* handed, so a round trip produces *a* valid encoding of the same history
1331-
* and not the encoding the origin's digest describes. The events beside it
1332-
* are that same snapshot parsed, for the retained stream to answer reads.
1333-
*/
1334-
readonly body: string;
1335-
readonly events: readonly DurableEvent[];
1336-
readonly origin?: RetainedOrigin;
1337-
},
1338-
): Operation<Publication> {
1318+
export function* publishPair(root: string, execution: string): Operation<Publication> {
13391319
const location = yield* establishPair(root, execution);
1340-
if (prefix.body.length > 0) {
1341-
// One append of the whole snapshot. Every record it holds is already
1342-
// terminated, so the copy is byte-for-byte what the origin's digest covers.
1343-
yield* ReplHost.operations.appendRecord(location.entries, prefix.body);
1344-
}
1345-
return yield* acknowledgePair(location, execution, prefix.events, prefix.origin);
1320+
return yield* acknowledgePair(location, execution, [], undefined);
13461321
}
13471322

13481323
/** Create both files of a new pair, exclusively, and nothing else yet. */
@@ -1399,6 +1374,16 @@ const COPY_BATCH = 1 << 22;
13991374
* covers; a file that is not UTF-8 is refused rather than copied as something
14001375
* else. No parsed record is kept — the retained stream already holds this
14011376
* history's events — and the count read must be the count it holds.
1377+
*
1378+
* The bytes rather than the events, because re-encoding parsed events is not
1379+
* copying them: `parseDurableEvent` rebuilds an envelope in its own member
1380+
* order and `serializeDurableEvent` preserves whatever order it is handed, so
1381+
* a round trip produces *a* valid encoding of the same history and not the
1382+
* encoding the origin's digest describes.
1383+
*
1384+
* No bound is passed. The file this reads is one `readRecords` has already
1385+
* opened under the supported limit, so a bound here would be a sentence
1386+
* nothing could reach.
14021387
*/
14031388
function* copyEntryRecords(from: string, to: string, expected: number): Operation<RetainedOrigin> {
14041389
const hash = createHash("sha256");
@@ -1409,7 +1394,6 @@ function* copyEntryRecords(from: string, to: string, expected: number): Operatio
14091394
const records = yield* forEachLine(
14101395
from,
14111396
{
1412-
limit: Number.POSITIVE_INFINITY,
14131397
strict: true,
14141398
observe: (chunk) => {
14151399
hash.update(chunk);
@@ -1420,8 +1404,6 @@ function* copyEntryRecords(from: string, to: string, expected: number): Operatio
14201404
new SidekickHistoryError(
14211405
"this execution's history ends in a record that was never finished writing.",
14221406
),
1423-
exceeded: () =>
1424-
new SidekickHistoryError("this execution's history is too long to publish as a pair."),
14251407
undecodable: () =>
14261408
new SidekickHistoryError(
14271409
"this execution's entry history holds bytes that are not text, so it cannot be " +
@@ -1526,6 +1508,16 @@ export function* openPair(
15261508
* whole reason `redirect` is synchronous: between acknowledging publication and
15271509
* selecting the new destination there is no point at which another coroutine
15281510
* could append to the path whose prefix was just declared final.
1511+
*
1512+
* What a refusal leaves behind is the pair's two empty files and no `owner`,
1513+
* because the copy is what reads the older history and the pair exists before
1514+
* it starts. That directory is an interrupted publication from then on: opening
1515+
* the execution refuses it and never falls back to the older file, which is
1516+
* still there and still holds everything. Cancellation leaves the same thing.
1517+
* The one refusal a history this process has already opened can reach is the
1518+
* copy's strict decoding, because {@link readRecords} decodes leniently — so a
1519+
* history with a byte that is not UTF-8 opens, and upgrading it stops at a
1520+
* refusal rather than publishing a pair whose bytes are not the bytes it read.
15291521
*/
15301522
export function* upgradeExecution(
15311523
root: string,

‎packages/cli/src/repl/sidekick/repository.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ export function pairedReplRepository(root: string): PairedReplRepository {
101101
// two processes racing for the same name is a refusal rather than a merge
102102
// — the same guarantee the single-file format took from exclusive create.
103103
const execution = yield* ReplHost.operations.identify();
104-
const published = yield* publishPair(root, execution, { body: "", events: [] });
104+
const published = yield* publishPair(root, execution);
105105
return {
106106
id: execution,
107107
stream: movableWriter(published.entries),

‎packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -758,7 +758,7 @@ function* phases(): Operation<unknown> {
758758
// the only part of these cases that is a fixture: section 3 opens no
759759
// conversation, and a Submit outside one is not a state this grammar has.
760760
yield* useHost(dataRoot);
761-
const published = yield* publishPair(repository, execution, { body: "", events: [] });
761+
const published = yield* publishPair(repository, execution);
762762
for (const record of conversationWithReply()) {
763763
yield* published.sidekick.append(record);
764764
}
@@ -919,7 +919,7 @@ function* phases(): Operation<unknown> {
919919
// the production owner in the phases below, not seeded here: cases 2, 5 and
920920
// 6 are about what that owner leaves behind.
921921
yield* useHost(dataRoot);
922-
yield* publishPair(repository, execution, { body: "", events: [] });
922+
yield* publishPair(repository, execution);
923923
return { seeded: true, records: (yield* readSidekickRecords(sidekick)).map((one) => one.kind) };
924924
}
925925

‎packages/cli/tests/repl-retained-history.test.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -535,6 +535,45 @@ describe("#992 — Sidekick prefixes and the upgrade use the file's own bytes",
535535
// The original is retained, untouched.
536536
expect(yield* until(readFile(legacy, "utf8"))).toBe(text);
537537
});
538+
539+
it("H4: a history that is not UTF-8 opens, refuses its upgrade, and keeps its bytes", function* () {
540+
const root = yield* useTempDirectory("xmd-upgrade-");
541+
yield* useHost("not-text");
542+
// One byte that begins no UTF-8 sequence, inside a record that is otherwise
543+
// a record: it stands where a bound value's text goes. Opening decodes
544+
// leniently, so the value reads as one replacement character and the record
545+
// parses; the copy decodes strictly, because what it appends has to encode
546+
// back to the bytes it read.
547+
const legacy = join(root, "not-text.jsonl");
548+
const [before, after] = line(record("one", "MARK")).split("MARK");
549+
const encoder = new TextEncoder();
550+
const damaged = new Uint8Array([
551+
...encoder.encode(before ?? ""),
552+
0xff,
553+
...encoder.encode(after ?? ""),
554+
]);
555+
yield* until(writeFile(legacy, damaged));
556+
const repository = pairedReplRepository(root);
557+
558+
const opened = yield* repository.open("not-text");
559+
const admitted = yield* opened.stream.readAll();
560+
expect(admitted).toHaveLength(1);
561+
expect(admitted[0]?.result).toEqual({ status: "ok", value: "\uFFFD" });
562+
563+
expect(yield* refusal(() => repository.upgrade(opened))).toBe(
564+
"this execution's entry history holds bytes that are not text, so it cannot be copied " +
565+
"exactly.",
566+
);
567+
// The older history still holds every byte it held.
568+
expect(new Uint8Array(yield* until(readFile(legacy)))).toEqual(damaged);
569+
// And what the refusal left is an interrupted publication: the pair's two
570+
// files exist and hold no publication, so reopening the execution refuses
571+
// them and never falls back to the older file beside them.
572+
expect(yield* refusal(() => pairedReplRepository(root).open("not-text"))).toBe(
573+
"this execution's chat history does not begin with the record that publishes it, so this " +
574+
"is an interrupted publication rather than a history to continue.",
575+
);
576+
});
538577
});
539578

540579
describe("#992 — every saved history reads as it did", () => {

0 commit comments

Comments
 (0)