Skip to content

Commit 331fc71

Browse files
t3dotggclaude
andcommitted
Harden dev:seed against state it cannot settle
Review of the seeding path found five ways a seeded database could describe something the copy cannot back up: - Nothing stopped `dev:seed` from running against a live server. SQLite locking lets the seed take the write lock between the server's own writes and commit cleanly; what it cannot do is tell that server its projections and event log were replaced underneath it. Refuse instead, via a pid probe over both state directories. - A thread copied mid-turn kept `state = 'running'`. The session override did not cover it — the turn is read independently, and an unsettled one leaves the thread spinning and its timeline unfoldable forever. Settle those to "interrupted", matching what the server does when a session leaves "running". Streaming messages are finished for the same reason. - Attachment metadata was copied without the files behind it, so a seeded thread rendered an image whose request 404s. Report the referenced ids and copy the matching files alongside the rows. - Orphaned `provider_session_runtime` rows survived the seed naming threads it had just deleted, which the session reaper then tried to stop against a provider instance the worktree may not define. - `--threads` had no upper bound, so a large enough value died with "too many SQL variables" after the target was already emptied. Also corrects the projector-cursor comment: `orchestration_events.sequence` is AUTOINCREMENT, so emptying the log does not reset its high-water mark and sequence does not restart at 1. Zero remains the right cursor, for the reason now stated. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 1d02536 commit 331fc71

9 files changed

Lines changed: 500 additions & 24 deletions

File tree

‎.agents/skills/test-t3-app/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ It finds the live server state under either the `dev` or `userdata` state direct
6262
Read [references/sqlite-fixtures.md](references/sqlite-fixtures.md) before changing the database.
6363

6464
- Use `node apps/server/scripts/t3-sqlite-state.ts query` for schema discovery and read-only checks.
65-
- To populate realistic list and thread data, stop the server and run `bun run dev:seed`, then restart it. This copies recent projections from the shared home while refusing to write to it.
65+
- To populate realistic list and thread data, stop the server and run `bun run dev:seed`, then restart it. This copies recent projections from the shared home while refusing to write to it. It also refuses to run while a server is still using the target directory, so stop the server first rather than expecting the seed to wait.
6666
- Stop the dev server before using `node apps/server/scripts/t3-sqlite-state.ts exec`, then restart it with the same base directory.
6767
- Seed projection tables only for disposable UI fixtures. Use application commands and APIs when testing business behavior or projection correctness.
6868
- Use the auth CLI, not direct `auth_*` table edits, for pairing and sessions.

‎.agents/skills/test-t3-app/references/sqlite-fixtures.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ The web UI primarily reads these projection tables:
4949
- `projection_pending_approvals`
5050
- `projection_thread_proposed_plans`
5151

52+
`projection_state` holds one cursor row per projector and is not read by the UI, but a fixture that omits it still breaks: `computeSnapshotSequence` returns 0 unless every projector has a row, which makes every shell snapshot advertise sequence 0. Write all of them (`PROJECTOR_NAMES` in `scripts/lib/projection-tables.ts`), at a sequence no higher than the event log actually reaches.
53+
5254
Inspect `PRAGMA table_info(<table>)` and the current migrations under `apps/server/src/persistence/Migrations/` before constructing inserts. Keep identifiers unique, timestamps as ISO strings, JSON columns valid, and related project/thread/turn IDs consistent.
5355

5456
For a substantial current example, inspect `seedDatabase` in `scripts/mobile-showcase-environment.ts`. Adapt its column set to the target database instead of assuming copied SQL remains current.

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
- In a linked git worktree, dev state defaults to that worktree's gitignored `.t3`. This deliberately outranks an ambient `T3CODE_HOME`, which could otherwise select the installed app's live `~/.t3/userdata` database. An explicit `--home-dir` still wins.
1919
- Start the web stack with `bun run dev`. Use `bun run dev:share` when someone needs to open it from another device on the tailnet.
2020
- Use `bun run dev:pair` to issue a fresh URL for the running server without reconstructing its port or origin.
21-
- Use `bun run dev:seed` while the server is stopped to populate an isolated database with recent projects and threads from the shared home.
21+
- Use `bun run dev:seed` while the server is stopped to populate an isolated database with recent projects and threads from the shared home. It refuses to run against a directory a live server is using.
2222
- Browser dev is single-origin: Vite proxies `/api`, `/ws`, `/oauth`, and `/.well-known` to the backend. Do not set `VITE_HTTP_URL` or `VITE_WS_URL` for `dev`/`dev:web`.
2323
- Worktree paths supply stable preferred port offsets. Read the actual server and web ports from the `[dev-runner]` line because occupied ports can still shift them.
2424

‎docs/reference/scripts.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@
4747

4848
Worktrees derive a preferred port offset from their path. The runner shifts both ports together when either is occupied, so treat the `[dev-runner]` output as authoritative.
4949

50-
An isolated dev database starts empty. `vp run dev:seed` copies projection data only, clears the target event log, and resets every projector cursor to zero so later events are not skipped or replayed over unrelated projections. Copied sessions are stopped and pending interaction counts are cleared because no live agent accompanies the copy. The copy tolerates source/target migration drift and remains local because it includes real message bodies and host paths.
50+
An isolated dev database starts empty. `vp run dev:seed` copies projection data only, clears the target event log, and resets every projector cursor to zero so later events are not skipped or replayed over unrelated projections. Because no live agent accompanies the copy, it also settles anything that would otherwise wait on one: sessions are stopped, running turns are interrupted, streaming messages are finished, pending interaction counts are cleared, and orphaned provider bindings are removed. Files backing copied image attachments are copied alongside the rows. The copy tolerates source/target migration drift and remains local because it includes real message bodies and host paths.
51+
52+
`dev:seed` refuses to run while a server is using the target directory: it replaces the projections and event log that server is holding open, and it has no way to tell the running process its read model was swapped underneath it. Stop the server, seed, then start it again.
5153

5254
## Running multiple dev instances
5355

‎scripts/dev-seed.ts‎

Lines changed: 92 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,14 +18,17 @@ import { Command, Flag } from "effect/unstable/cli";
1818
import { resolveWorktreeT3Home } from "@t3tools/shared/devHome";
1919

2020
import { DevSeedError, seedDevDatabase } from "./lib/dev-seed.ts";
21+
import { findRunningServerPid } from "./lib/server-runtime-probe.ts";
2122

2223
const DEFAULT_THREAD_LIMIT = 25;
2324
const DEFAULT_ACTIVITY_LIMIT = 200;
25+
/** SQLite's default SQLITE_MAX_VARIABLE_NUMBER; one bound parameter per thread. */
26+
const MAX_THREAD_LIMIT = 32_766;
2427

2528
class DevSeedTargetError extends Schema.TaggedErrorClass<DevSeedTargetError>()(
2629
"DevSeedTargetError",
2730
{
28-
reason: Schema.Literals(["shared-home", "missing-target", "not-a-worktree"]),
31+
reason: Schema.Literals(["shared-home", "missing-target", "not-a-worktree", "server-running"]),
2932
detail: Schema.String,
3033
},
3134
) {
@@ -46,13 +49,68 @@ class DevSeedTargetError extends Schema.TaggedErrorClass<DevSeedTargetError>()(
4649
"Not inside a git worktree, so there is no worktree-local data directory to seed.",
4750
"Pass --to <base-dir> to choose one explicitly.",
4851
].join("\n");
52+
case "server-running":
53+
return [
54+
`A T3 Code server (pid ${this.detail}) is running against this data directory.`,
55+
"Seeding replaces its projections and empties its event log while it holds them open,",
56+
"which leaves the running server's in-memory read model describing rows that no longer exist.",
57+
"Stop it, run the seed, then start it again.",
58+
].join("\n");
4959
}
5060
}
5161
}
5262

5363
const stateDbPath = (path: Path.Path, baseDir: string) =>
5464
path.join(baseDir, "userdata", "state.sqlite");
5565

66+
/**
67+
* Copies the files behind the copied messages' attachment references.
68+
*
69+
* The rows carry only metadata; the bytes sit in a flat directory beside the
70+
* database as `<attachmentId><ext>` (apps/server/src/attachmentStore.ts). Copy
71+
* the rows alone and a seeded thread renders an image whose request 404s.
72+
*
73+
* Matched by directory listing rather than by reconstructing the extension: the
74+
* server infers it from mime type and file name at upload, and duplicating that
75+
* inference here would rot independently of it.
76+
*/
77+
const copyAttachments = Effect.fn("devSeed.copyAttachments")(function* (input: {
78+
readonly sourceDir: string;
79+
readonly targetDir: string;
80+
readonly attachmentIds: ReadonlyArray<string>;
81+
}) {
82+
const fileSystem = yield* FileSystem.FileSystem;
83+
const path = yield* Path.Path;
84+
85+
if (input.attachmentIds.length === 0) {
86+
return 0;
87+
}
88+
89+
const entries = yield* fileSystem
90+
.readDirectory(input.sourceDir)
91+
.pipe(Effect.orElseSucceed(() => [] as Array<string>));
92+
if (entries.length === 0) {
93+
return 0;
94+
}
95+
96+
const wanted = new Set(input.attachmentIds);
97+
const matches = entries.filter((entry) => wanted.has(path.parse(entry).name));
98+
if (matches.length === 0) {
99+
return 0;
100+
}
101+
102+
yield* fileSystem.makeDirectory(input.targetDir, { recursive: true });
103+
// Attachments are decoration on a dev fixture: one unreadable file should
104+
// leave the rest of the seed intact rather than fail a completed copy.
105+
const copied = yield* Effect.forEach(matches, (entry) =>
106+
fileSystem.copyFile(path.join(input.sourceDir, entry), path.join(input.targetDir, entry)).pipe(
107+
Effect.as(1),
108+
Effect.orElseSucceed(() => 0),
109+
),
110+
);
111+
return copied.reduce((total, one) => total + one, 0);
112+
});
113+
56114
const devSeedCli = Command.make("dev-seed", {
57115
from: Flag.string("from").pipe(
58116
Flag.withDescription(
@@ -68,12 +126,17 @@ const devSeedCli = Command.make("dev-seed", {
68126
Flag.optional,
69127
Flag.map(Option.getOrUndefined),
70128
),
71-
// Bounded: SQLite reads a negative LIMIT as "no limit", which would quietly
72-
// turn a capped copy into a full-table one.
129+
// Bounded below: SQLite reads a negative LIMIT as "no limit", which would
130+
// quietly turn a capped copy into a full-table one. Bounded above because
131+
// every selected thread becomes one bound parameter in the copy's `IN (...)`,
132+
// and SQLite's compiled-in ceiling is 32766 — past it the seed dies with
133+
// "too many SQL variables" after the target has already been emptied.
73134
threads: Flag.integer("threads").pipe(
74-
Flag.withSchema(Schema.Int.check(Schema.isGreaterThan(0))),
135+
Flag.withSchema(
136+
Schema.Int.check(Schema.isGreaterThan(0), Schema.isLessThanOrEqualTo(MAX_THREAD_LIMIT)),
137+
),
75138
Flag.withDescription(
76-
`How many recent threads to copy (default ${String(DEFAULT_THREAD_LIMIT)}).`,
139+
`How many recent threads to copy (default ${String(DEFAULT_THREAD_LIMIT)}, max ${String(MAX_THREAD_LIMIT)}).`,
77140
),
78141
Flag.withDefault(DEFAULT_THREAD_LIMIT),
79142
),
@@ -117,6 +180,19 @@ const devSeedCli = Command.make("dev-seed", {
117180
return yield* new DevSeedTargetError({ reason: "missing-target", detail: targetDbPath });
118181
}
119182

183+
// SQLite locking is not enough on its own: the seed's transaction can take
184+
// the write lock between a running server's own writes and commit cleanly.
185+
// What it cannot do is tell that server the projections it is holding in
186+
// memory, and the event log it is appending to, were both replaced under
187+
// it. Its next write then lands on top of a read model it never built.
188+
const runningPid = yield* findRunningServerPid(targetBaseDir);
189+
if (Option.isSome(runningPid)) {
190+
return yield* new DevSeedTargetError({
191+
reason: "server-running",
192+
detail: String(runningPid.value),
193+
});
194+
}
195+
120196
const seededAt = yield* Effect.map(DateTime.now, DateTime.formatIso);
121197
const summary = yield* Effect.try({
122198
try: () =>
@@ -130,6 +206,12 @@ const devSeedCli = Command.make("dev-seed", {
130206
catch: (cause) => cause as DevSeedError,
131207
});
132208

209+
const copiedAttachments = yield* copyAttachments({
210+
sourceDir: path.join(sourceBaseDir, "userdata", "attachments"),
211+
targetDir: path.join(targetBaseDir, "userdata", "attachments"),
212+
attachmentIds: summary.attachmentIds,
213+
});
214+
133215
yield* Console.log(
134216
[
135217
`Seeded ${targetDbPath}`,
@@ -139,6 +221,11 @@ const devSeedCli = Command.make("dev-seed", {
139221
` messages ${String(summary.messages)}`,
140222
` activity ${String(summary.activities)}`,
141223
` turns ${String(summary.turns)} sessions ${String(summary.sessions)}`,
224+
...(summary.attachmentIds.length > 0
225+
? [
226+
` files ${String(copiedAttachments)}/${String(summary.attachmentIds.length)} attachment(s)`,
227+
]
228+
: []),
142229
...(summary.skippedColumns.length > 0
143230
? [
144231
` note: skipped ${String(summary.skippedColumns.length)} column(s) absent from the target schema`,

‎scripts/lib/dev-seed.test.ts‎

Lines changed: 114 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ function createSchema(
3131
${options.withMonitor ? ", monitor_json TEXT" : ""})`);
3232
database.exec(`CREATE TABLE projection_thread_messages (
3333
message_id TEXT PRIMARY KEY, thread_id TEXT NOT NULL, role TEXT NOT NULL,
34-
text TEXT NOT NULL, is_streaming INTEGER NOT NULL,
34+
text TEXT NOT NULL, is_streaming INTEGER NOT NULL, attachments_json TEXT,
3535
created_at TEXT NOT NULL, updated_at TEXT NOT NULL)`);
3636
database.exec(`CREATE TABLE projection_thread_activities (
3737
activity_id TEXT PRIMARY KEY, thread_id TEXT NOT NULL, tone TEXT NOT NULL,
@@ -42,7 +42,8 @@ function createSchema(
4242
active_turn_id TEXT, last_error TEXT, updated_at TEXT NOT NULL)`);
4343
database.exec(`CREATE TABLE projection_turns (
4444
row_id INTEGER PRIMARY KEY AUTOINCREMENT, thread_id TEXT NOT NULL, turn_id TEXT,
45-
state TEXT NOT NULL, requested_at TEXT NOT NULL, checkpoint_files_json TEXT NOT NULL,
45+
state TEXT NOT NULL, requested_at TEXT NOT NULL, completed_at TEXT,
46+
checkpoint_files_json TEXT NOT NULL,
4647
UNIQUE (thread_id, turn_id))`);
4748
database.exec(`CREATE TABLE projection_thread_proposed_plans (
4849
plan_id TEXT PRIMARY KEY, thread_id TEXT NOT NULL, plan_markdown TEXT NOT NULL,
@@ -55,6 +56,9 @@ function createSchema(
5556
database.exec(`CREATE TABLE orchestration_events (
5657
sequence INTEGER PRIMARY KEY AUTOINCREMENT, event_id TEXT NOT NULL UNIQUE,
5758
stream_id TEXT NOT NULL, event_type TEXT NOT NULL, payload_json TEXT NOT NULL)`);
59+
database.exec(`CREATE TABLE provider_session_runtime (
60+
thread_id TEXT PRIMARY KEY, provider_name TEXT NOT NULL, adapter_key TEXT NOT NULL,
61+
status TEXT NOT NULL, last_seen_at TEXT NOT NULL)`);
5862
}
5963

6064
/** Source DB with `threadCount` threads, oldest first so recency ordering is testable. */
@@ -95,18 +99,35 @@ function makeSource(
9599
)
96100
.run(threadId, "p1", `Thread ${String(index)}`, `turn-${threadId}`, at, at, at, 4, 2, "{}");
97101
}
102+
// The newest thread is left mid-turn, as a copy taken while an agent is
103+
// working would be: a running turn with no completion, and a message still
104+
// streaming into it.
105+
const midFlight = index === threadCount - 1;
98106
database
99107
.prepare(
100-
`INSERT INTO projection_turns (thread_id, turn_id, state, requested_at, checkpoint_files_json)
101-
VALUES (?,?,?,?,'[]')`,
108+
`INSERT INTO projection_turns (thread_id, turn_id, state, requested_at, completed_at, checkpoint_files_json)
109+
VALUES (?,?,?,?,?,'[]')`,
102110
)
103-
.run(threadId, `turn-${threadId}`, "completed", at);
111+
.run(
112+
threadId,
113+
`turn-${threadId}`,
114+
midFlight ? "running" : "completed",
115+
at,
116+
midFlight ? null : at,
117+
);
104118
database
105119
.prepare(`INSERT INTO projection_thread_sessions VALUES (?,'running','claude',?,'boom',?)`)
106120
.run(threadId, `turn-${threadId}`, at);
107121
database
108-
.prepare(`INSERT INTO projection_thread_messages VALUES (?,?, 'user','hello',0,?,?)`)
109-
.run(`m-${threadId}`, threadId, at, at);
122+
.prepare(`INSERT INTO projection_thread_messages VALUES (?,?, 'user','hello',?,?,?,?)`)
123+
.run(
124+
`m-${threadId}`,
125+
threadId,
126+
midFlight ? 1 : 0,
127+
JSON.stringify([{ type: "image", id: `${threadId}-image`, name: "shot.png" }]),
128+
at,
129+
at,
130+
);
110131
for (let a = 0; a < activitiesPerThread; a += 1) {
111132
database
112133
.prepare(`INSERT INTO projection_thread_activities VALUES (?,?,'neutral','tool','ran',?,?)`)
@@ -130,6 +151,11 @@ function makeTarget(path: string, options?: { readonly dropProposedPlans?: boole
130151
.prepare(`INSERT INTO orchestration_events (event_id, stream_id, event_type, payload_json)
131152
VALUES ('old-event','stale-thread','thread.created','{}')`)
132153
.run();
154+
// A binding for a thread the seed is about to delete.
155+
database
156+
.prepare(`INSERT INTO provider_session_runtime
157+
VALUES ('stale-thread','claude','claude-code','running','${SEEDED_AT}')`)
158+
.run();
133159
if (options?.dropProposedPlans) {
134160
// A target far enough behind that a whole table is missing (migration 013).
135161
database.exec("DROP TABLE projection_thread_proposed_plans");
@@ -271,6 +297,87 @@ describe("seedDevDatabase", () => {
271297
});
272298
});
273299

300+
// A copied running turn has no agent left to finish it. The session override
301+
// alone does not cover this: the turn is read independently, and an unsettled
302+
// one keeps the thread spinning and its timeline unfoldable forever.
303+
it("settles turns copied mid-flight", () => {
304+
withDatabases(({ source, target }) => {
305+
seedDevDatabase({
306+
sourceDbPath: source,
307+
targetDbPath: target,
308+
threadLimit: 3,
309+
activityLimit: 10,
310+
seededAt: SEEDED_AT,
311+
});
312+
313+
const turns = query<{ state: string; completed_at: string | null }>(
314+
target,
315+
"SELECT state, completed_at FROM projection_turns",
316+
);
317+
assert.isAbove(turns.length, 0);
318+
for (const turn of turns) {
319+
assert.notEqual(turn.state, "running");
320+
assert.isNotNull(turn.completed_at);
321+
}
322+
});
323+
});
324+
325+
// Nothing will ever stream into a copied message, so a carried-over flag
326+
// renders a caret that never resolves.
327+
it("clears the streaming flag on copied messages", () => {
328+
withDatabases(({ source, target }) => {
329+
seedDevDatabase({
330+
sourceDbPath: source,
331+
targetDbPath: target,
332+
threadLimit: 3,
333+
activityLimit: 10,
334+
seededAt: SEEDED_AT,
335+
});
336+
337+
const streaming = query<{ count: number }>(
338+
target,
339+
"SELECT COUNT(*) count FROM projection_thread_messages WHERE is_streaming <> 0",
340+
);
341+
assert.equal(streaming[0]?.count, 0);
342+
});
343+
});
344+
345+
// The rows carry attachment metadata but the bytes live on disk, so the
346+
// caller has to be told which files to bring along.
347+
it("reports the attachment ids the copied messages reference", () => {
348+
withDatabases(({ source, target }) => {
349+
const summary = seedDevDatabase({
350+
sourceDbPath: source,
351+
targetDbPath: target,
352+
threadLimit: 2,
353+
activityLimit: 10,
354+
seededAt: SEEDED_AT,
355+
});
356+
357+
assert.deepStrictEqual([...summary.attachmentIds].sort(), ["t3-image", "t4-image"]);
358+
});
359+
});
360+
361+
// Bindings are keyed by thread id and are not copied; left behind they name
362+
// threads the seed just deleted, which the session reaper then tries to stop.
363+
it("clears provider runtime bindings the seed orphans", () => {
364+
withDatabases(({ source, target }) => {
365+
seedDevDatabase({
366+
sourceDbPath: source,
367+
targetDbPath: target,
368+
threadLimit: 1,
369+
activityLimit: 10,
370+
seededAt: SEEDED_AT,
371+
});
372+
373+
const [bindings] = query<{ count: number }>(
374+
target,
375+
"SELECT COUNT(*) count FROM provider_session_runtime",
376+
);
377+
assert.equal(bindings?.count, 0);
378+
});
379+
});
380+
274381
it("caps activities per thread, keeping the newest", () => {
275382
withDatabases(
276383
({ source, target }) => {

0 commit comments

Comments
 (0)