Skip to content

Commit 3ee6cba

Browse files
donjor-agent[bot]rsi-pr-agent
authored andcommitted
fix(server): purge imported v1 rows the v2 runtime never reads (#1723)
## Problem The V2 runtime never reads a thread's V1 events after its transcript import, yet nothing deletes them. On `ugrindtime` that dead data is most of a 12 GB database: | Rows | Count | Payload | | --- | --- | --- | | V1 events | 2.58M | 3.18 GB, plus indexes | | `projection_thread_activities` | 1.81M | 2.86 GB, no reader | Every page of it adds disk IO to checkpoints, backups, and cold reads. Upstream's `ProjectionMaintenance.compactEventStore` (`pingdotgg#2829`) drops the same V1 events, but nothing calls it in production. Its thread-state supersession would also delete the fork's legacy issue markers, which are V2 `thread.metadata-updated` events. Losing a marker re-runs the issue import and undoes V2 unlinks. ## Change ### Code - **New file:** `LegacyV1Purge.fork.ts` - **Hook:** two lines in the V1 importer - **When:** after the background transcript import ### Rows - **Events:** V1 rows of imported threads - **Activities:** whole table - **Kept:** V2, project, and pending-thread events ### Pacing - **Pages:** 200 rows, 50 ms pause between - **Failure:** logged; the next startup resumes ### Measured on a full copy of the live database - **Live data:** 12.09 GB to 3.08 GB - **Page hold:** p50 8 ms, p99 171 ms - **Expected first run:** about 25 min, background - **Later runs:** no-op ## Reclaiming disk The purge frees pages for reuse, so the file stops growing but does not shrink. To shrink it once, after the purge has logged `Purged imported legacy v1 rows`: 1. Stop `t3code.service` 2. Back up `~/.t3/userdata/statev2.sqlite` 3. Run `sqlite3 ~/.t3/userdata/statev2.sqlite VACUUM` 4. Start the service ## Follow-ups - **Growth:** `turn-item.updated` adds ~260 MB/day - **Markers:** store outside events first ## Scope and approval Refs #1720. The maintainer chose this narrow purge over wiring upstream's compaction. ## Verification - **New test:** 1001-row purge keeps live rows - **Mutation check:** fails without import filter - **Legacy tests:** 10 of 10 pass - **Typecheck:** `apps/server`, exit 0 - **Fork CI:** `vp run fork:ci`, exit 0 - **Not checked:** a live server on this build Claude Code, claude-opus-5-5. ## Fork trailers Squashes: - 7595062 fix(server): purge imported v1 rows the v2 runtime never reads (#1723) (#1723) - d1395ce fixup! fix(server): purge imported v1 rows the v2 runtime never reads (#1723) (#1723) Fork-Domain: upstream-fixes Fork-Tier: bugfix Fork-Upstreamable: yes
1 parent cb27f62 commit 3ee6cba

3 files changed

Lines changed: 134 additions & 0 deletions

File tree

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
import { assert, it } from "@effect/vitest";
2+
import * as Effect from "effect/Effect";
3+
import * as SqlClient from "effect/sql/SqlClient";
4+
5+
import * as SqlitePersistence from "../../persistence/Sqlite.ts";
6+
import { purgeImportedLegacyV1Fork } from "./LegacyV1Purge.fork.ts";
7+
8+
const now = "2026-01-01T00:00:00.000Z";
9+
const imported = "thread:purge-imported";
10+
const pending = "thread:purge-pending";
11+
12+
// The purge paces pages with real sleeps.
13+
it.layer(SqlitePersistence.layerMemory, { excludeTestServices: true })("legacy v1 purge", (it) => {
14+
it.effect("drops V1 rows of imported threads in pages and keeps everything V2 reads", () =>
15+
Effect.gen(function* () {
16+
const sql = yield* SqlClient.SqlClient;
17+
const insertEvents = (
18+
prefix: string,
19+
aggregateKind: string,
20+
streamId: string,
21+
version: number,
22+
count: number,
23+
firstVersion = 1,
24+
) => sql`
25+
WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ${count})
26+
INSERT INTO orchestration_events (
27+
event_id, aggregate_kind, stream_id, stream_version, event_type,
28+
occurred_at, actor_kind, payload_json, metadata_json, application_event_version
29+
)
30+
SELECT ${prefix} || i, ${aggregateKind}, ${streamId}, ${firstVersion - 1} + i,
31+
'thread.activity-appended', ${now}, 'server', '{}', '{}', ${version}
32+
FROM n
33+
`;
34+
// More than one page of V1 events for the imported thread.
35+
yield* insertEvents("v1:", "thread", imported, 1, 1001);
36+
yield* insertEvents("v1-pending:", "thread", pending, 1, 2);
37+
yield* insertEvents("v1-project:", "project", "project:purge", 1, 1);
38+
// Fork issue markers are V2 thread events and must survive.
39+
yield* insertEvents("v2:", "thread", imported, 2, 1, 1002);
40+
yield* sql`
41+
INSERT INTO orchestration_v2_legacy_imports (
42+
thread_id, source_updated_at, shell_imported_at, transcript_imported_at,
43+
imported_message_count, last_error
44+
)
45+
VALUES (${imported}, ${now}, ${now}, ${now}, 0, NULL),
46+
(${pending}, ${now}, ${now}, NULL, 0, NULL)
47+
`;
48+
yield* sql`
49+
WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 1500)
50+
INSERT INTO projection_thread_activities (
51+
activity_id, thread_id, tone, kind, summary, payload_json, created_at
52+
)
53+
SELECT 'activity:' || i, ${imported}, 'info', 'tool.completed', '', '{}', ${now} FROM n
54+
`;
55+
56+
assert.deepEqual(yield* purgeImportedLegacyV1Fork(sql), { events: 1001, activities: 1500 });
57+
58+
const remaining = yield* sql<{ readonly event_id: string }>`
59+
SELECT event_id FROM orchestration_events ORDER BY event_id
60+
`;
61+
assert.deepEqual(
62+
remaining.map((row) => row.event_id),
63+
["v1-pending:1", "v1-pending:2", "v1-project:1", "v2:1"],
64+
);
65+
assert.deepEqual(yield* purgeImportedLegacyV1Fork(sql), { events: 0, activities: 0 });
66+
}),
67+
);
68+
});
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
// Fork-owned purge of V1 rows the V2 runtime never reads. Every V2 reader selects
2+
// only `application_event_version = 2` thread events, so a thread's V1 events are
3+
// dead once its transcript import finishes, and `projection_thread_activities`
4+
// has no reader at all. Upstream's `ProjectionMaintenance.compactEventStore` also
5+
// drops these events but has no production caller, and its thread-state
6+
// supersession would delete the fork's legacy issue markers. Retire this when
7+
// upstream schedules its own compaction and the markers no longer depend on events.
8+
import * as Effect from "effect/Effect";
9+
import type * as SqlClient from "effect/sql/SqlClient";
10+
import type * as SqlError from "effect/sql/SqlError";
11+
12+
// Measured on a 12 GB database: 200-row pages hold the connection for a p99 of
13+
// 171 ms, and the pause keeps the purge from saturating the disk.
14+
const PAGE_SIZE = 200;
15+
const PAGE_PAUSE = "50 millis";
16+
17+
/** Deletes one page at a time; each page commits alone so requests interleave. */
18+
const drainPages = Effect.fn("LegacyV1Purge.drainPages")(function* (
19+
deletePage: Effect.Effect<ReadonlyArray<unknown>, SqlError.SqlError>,
20+
) {
21+
let deleted = 0;
22+
while (true) {
23+
const rows = yield* deletePage;
24+
deleted += rows.length;
25+
if (rows.length < PAGE_SIZE) return deleted;
26+
yield* Effect.sleep(PAGE_PAUSE);
27+
}
28+
});
29+
30+
/** Runs after the background transcript import; a no-op once the V1 rows are gone. */
31+
export const purgeImportedLegacyV1Fork = Effect.fn("purgeImportedLegacyV1Fork")(
32+
function* (sql: SqlClient.SqlClient) {
33+
const events = yield* drainPages(sql`
34+
DELETE FROM orchestration_events
35+
WHERE sequence IN (
36+
SELECT sequence
37+
FROM orchestration_events INDEXED BY idx_orchestration_events_application_sequence
38+
WHERE application_event_version = 1
39+
AND aggregate_kind = 'thread'
40+
AND stream_id IN (
41+
SELECT thread_id FROM orchestration_v2_legacy_imports
42+
WHERE transcript_imported_at IS NOT NULL
43+
)
44+
LIMIT ${PAGE_SIZE}
45+
)
46+
RETURNING sequence
47+
`);
48+
const activities = yield* drainPages(sql`
49+
DELETE FROM projection_thread_activities
50+
WHERE rowid IN (SELECT rowid FROM projection_thread_activities LIMIT ${PAGE_SIZE})
51+
RETURNING rowid
52+
`);
53+
if (events + activities > 0) {
54+
yield* Effect.logInfo("Purged imported legacy v1 rows", { events, activities });
55+
}
56+
return { events, activities };
57+
},
58+
// A failed purge leaves the rest for the next startup.
59+
Effect.catchCause((cause) =>
60+
Effect.logWarning("Legacy v1 purge stopped", { cause }).pipe(
61+
Effect.as({ events: 0, activities: 0 }),
62+
),
63+
),
64+
);

‎apps/server/src/orchestration-v2/legacy/LegacyV1ThreadImporter.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ import * as SqlClient from "effect/sql/SqlClient";
3535
import * as EventSink from "../EventSink.ts";
3636
import { randomUuidV4 } from "@t3tools/provider-core/server/randomUuid";
3737
import { importLegacyThreadIssuesFork } from "./LegacyV1ThreadIssues.fork.ts"; // fork-hook: github-issues/legacy-issues-import
38+
import { purgeImportedLegacyV1Fork } from "./LegacyV1Purge.fork.ts"; // fork-hook: upstream-fixes/legacy-v1-purge-import
3839

3940
const IMPORT_EVENT_PREFIX = "migration:v1";
4041
const TRANSCRIPT_EVENT_BATCH_SIZE = 100;
@@ -804,6 +805,7 @@ const make = Effect.gen(function* () {
804805
importedMessageCount += result.importedMessageCount;
805806
yield* Effect.yieldNow;
806807
}
808+
yield* purgeImportedLegacyV1Fork(sql); // fork-hook: upstream-fixes/legacy-v1-purge
807809
return { importedThreadCount, importedMessageCount };
808810
}).pipe(
809811
Effect.catchCause((cause) =>

0 commit comments

Comments
 (0)