Skip to content

Commit 9a2a02e

Browse files
committed
console: add data for the redesigned cluster page's hydration tab
First of two PRs adding a Hydration tab to the redesigned cluster page, on top of the configuration drawer. This one adds the data the tab reads. The next one adds the tab. - History, read with react-query since the history tables have no index: a cluster's hydration episodes from mz_replica_hydration_history (26.43 and later, one row per episode with the highest peaks across its processes), the cluster's replicas from mz_cluster_replica_history so dropped ones keep their name and size, an episode's objects from mz_object_hydration_history, and the highest CPU sampled during an episode. - Live state, in SUBSCRIBE-fed TanStack DB collections: whether each user object on each replica is hydrated (a lookup on mz_compute_hydration_times_ind), and each object's queued, working or hydrated state on one replica, from that replica's own introspection (26.39 and later) in a session pinned to it. - A model that counts an episode as a full hydration when it starts within two minutes of its replica booting, finds replicas that went offline before they finished hydrating, such as out-of-memory restarts, and splits an episode's objects into time waiting for a hydration slot and time working. - Collection subscribes keep their session variables across connection attempts. Each attempt sent the defaults instead, so a cluster given to a collection subscribe never reached the server. No existing caller passed one. - types/materialize.d.ts regenerated with yarn gen:types.
1 parent 966a242 commit 9a2a02e

14 files changed

Lines changed: 3249 additions & 374 deletions

‎console/src/api/materialize/cluster/hydrationHistory.test.sql.ts‎

Lines changed: 507 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
// Copyright Materialize, Inc. and contributors. All rights reserved.
2+
//
3+
// Use of this software is governed by the Business Source License
4+
// included in the LICENSE file.
5+
//
6+
// As of the Change Date specified in that file, in accordance with
7+
// the Business Source License, use of this software will be governed
8+
// by the Apache License, Version 2.0.
9+
10+
import { QueryKey } from "@tanstack/react-query";
11+
import { sql } from "kysely";
12+
13+
import { executeSqlV2, queryBuilder } from "~/api/materialize";
14+
import { buildSubscribeQuery } from "~/api/materialize/buildSubscribeQuery";
15+
16+
const timestampLit = (date: string) => sql<Date>`${sql.lit(date)}::timestamptz`;
17+
18+
/**
19+
* A cluster's hydration episodes that finished at or after `startDate`, one
20+
* row per episode. Requires mz >= 26.43, which records one row per process.
21+
* Episodes with no objects stay in: on a replica with nothing to hydrate, such
22+
* as one running only sources, they're the record that a boot finished.
23+
*
24+
* Peaks are the highest any of the episode's processes reached since it
25+
* started, read when the episode was recorded, so they can predate it.
26+
* `peakHeapBytes` is the highest memory plus disk peak of a single process.
27+
*
28+
* NOTE: the history has no index, so this reads storage. It is sampled one
29+
* replica at a time, so an episode shows up a while after it finishes, and
30+
* only episodes that finish are recorded.
31+
*/
32+
export function buildClusterHydrationEpisodesQuery({
33+
clusterId,
34+
startDate,
35+
}: {
36+
clusterId: string;
37+
startDate: string;
38+
}) {
39+
return (
40+
queryBuilder
41+
.selectFrom("mz_replica_hydration_history")
42+
.select([
43+
"replica_id as replicaId",
44+
"started_at as startedAt",
45+
// Timing and object count repeat on each process's row.
46+
sql<Date>`max(finished_at)`.as("finishedAt"),
47+
sql<number>`max(object_count)::float8`.as("objectCount"),
48+
sql<number | null>`max(peak_memory_bytes)::float8`.as(
49+
"peakMemoryBytes",
50+
),
51+
sql<number | null>`max(peak_disk_bytes)::float8`.as("peakDiskBytes"),
52+
sql<
53+
number | null
54+
>`max(peak_memory_bytes + coalesce(peak_disk_bytes, 0))::float8`.as(
55+
"peakHeapBytes",
56+
),
57+
])
58+
.where("cluster_id", "=", clusterId)
59+
// Rows recorded before 26.43 have no process id, peaks taken across
60+
// processes, and counts that include introspection dataflows.
61+
.where("process_id", "is not", null)
62+
.where("finished_at", ">=", timestampLit(startDate))
63+
.groupBy(["replica_id", "started_at"])
64+
);
65+
}
66+
67+
export async function fetchClusterHydrationEpisodes({
68+
params,
69+
queryKey,
70+
requestOptions,
71+
}: {
72+
params: Parameters<typeof buildClusterHydrationEpisodesQuery>[0];
73+
queryKey: QueryKey;
74+
requestOptions?: RequestInit;
75+
}) {
76+
const res = await executeSqlV2({
77+
queries: buildClusterHydrationEpisodesQuery(params).compile(),
78+
queryKey,
79+
requestOptions,
80+
sessionVariables: { transaction_isolation: "serializable" },
81+
});
82+
return res.rows;
83+
}
84+
85+
/**
86+
* A cluster's replicas that existed at or after `startDate`, including
87+
* dropped ones, with each replica's name and size when it was created. System
88+
* clusters have none.
89+
*/
90+
export function buildClusterReplicaHistoryQuery({
91+
clusterId,
92+
startDate,
93+
}: {
94+
clusterId: string;
95+
startDate: string;
96+
}) {
97+
return queryBuilder
98+
.selectFrom("mz_cluster_replica_history")
99+
.select([
100+
"replica_id as replicaId",
101+
"replica_name as name",
102+
"size",
103+
"created_at as createdAt",
104+
"dropped_at as droppedAt",
105+
])
106+
.where("cluster_id", "=", clusterId)
107+
.where((eb) =>
108+
eb.or([
109+
eb("dropped_at", "is", null),
110+
eb("dropped_at", ">=", timestampLit(startDate)),
111+
]),
112+
);
113+
}
114+
115+
export async function fetchClusterReplicaHistory({
116+
params,
117+
queryKey,
118+
requestOptions,
119+
}: {
120+
params: Parameters<typeof buildClusterReplicaHistoryQuery>[0];
121+
queryKey: QueryKey;
122+
requestOptions?: RequestInit;
123+
}) {
124+
const res = await executeSqlV2({
125+
queries: buildClusterReplicaHistoryQuery(params).compile(),
126+
queryKey,
127+
requestOptions,
128+
sessionVariables: { transaction_isolation: "serializable" },
129+
});
130+
return res.rows;
131+
}
132+
133+
export interface ReplicaObjectHydrationState {
134+
replicaId: string;
135+
objectId: string;
136+
isHydrated: boolean;
137+
}
138+
139+
export const replicaObjectHydrationStateKey = ({
140+
replicaId,
141+
objectId,
142+
}: ReplicaObjectHydrationState) => `${replicaId}|${objectId}`;
143+
144+
/**
145+
* Whether each user object on the given replicas is hydrated. The literal id
146+
* list makes this a lookup on `mz_compute_hydration_times_ind`.
147+
*
148+
* NOTE: for a few seconds after a replica restarts, this can still report the
149+
* previous process's hydrated objects.
150+
*/
151+
export function buildReplicaHydrationStatesSubscribe({
152+
replicaIds,
153+
}: {
154+
replicaIds: string[];
155+
}) {
156+
return buildSubscribeQuery<ReplicaObjectHydrationState>(
157+
queryBuilder
158+
.selectFrom("mz_compute_hydration_times")
159+
.select([
160+
"replica_id as replicaId",
161+
"object_id as objectId",
162+
sql<boolean>`time_ns IS NOT NULL`.as("isHydrated"),
163+
])
164+
.where("replica_id", "in", replicaIds)
165+
.where("object_id", "like", "u%"),
166+
{ upsertKey: ["replicaId", "objectId"] },
167+
);
168+
}
169+
170+
/** One object's hydration on a replica, across the replica's workers. */
171+
export interface LiveObjectHydration {
172+
globalId: string;
173+
/** The catalog id, null for a dropped object's dataflow. */
174+
objectId: string | null;
175+
installedAt: Date;
176+
/** When the first worker got a hydration slot, null while queued. */
177+
startedAt: Date | null;
178+
/** When the last worker finished, null until every worker has. */
179+
hydratedAt: Date | null;
180+
}
181+
182+
/**
183+
* Each compute object's hydration on one replica, as it happens. Reads the
184+
* replica's introspection, so it must run in a session pinned to the replica
185+
* (`cluster` and `cluster_replica`), and runs on that replica rather than on
186+
* `mz_catalog_server`. Only user objects, which leaves out this subscribe's
187+
* own transient dataflow. Requires mz >= 26.39.
188+
*/
189+
export function buildLiveObjectHydrationsSubscribe() {
190+
return buildSubscribeQuery<LiveObjectHydration>(
191+
queryBuilder
192+
.selectFrom("mz_compute_hydration_times_per_worker as h")
193+
.leftJoin("mz_object_global_ids as g", "g.global_id", "h.export_id")
194+
.select([
195+
"h.export_id as globalId",
196+
"g.id as objectId",
197+
sql<Date>`min(h.installed_at)`.as("installedAt"),
198+
sql<Date | null>`min(h.started_at)`.as("startedAt"),
199+
sql<Date | null>`CASE WHEN count(h.hydrated_at) = count(*) THEN max(h.hydrated_at) END`.as(
200+
"hydratedAt",
201+
),
202+
])
203+
.where("h.export_id", "like", "u%")
204+
.groupBy(["h.export_id", "g.id"]),
205+
{ upsertKey: "globalId" },
206+
);
207+
}
208+
209+
/**
210+
* Each object hydration on `replicaId` whose dataflow was installed between
211+
* `startDate` and `endDate`, the window of one episode. `objectId` is the
212+
* catalog id, null once the object is dropped. Requires mz >= 26.40.
213+
*
214+
* NOTE: the history has no index, so this reads storage.
215+
*/
216+
export function buildEpisodeObjectHydrationsQuery({
217+
replicaId,
218+
startDate,
219+
endDate,
220+
}: {
221+
replicaId: string;
222+
startDate: string;
223+
endDate: string;
224+
}) {
225+
return queryBuilder
226+
.selectFrom("mz_object_hydration_history as h")
227+
.leftJoin("mz_object_global_ids as g", "g.global_id", "h.object_id")
228+
.select([
229+
"h.object_id as globalId",
230+
"g.id as objectId",
231+
"h.installed_at as installedAt",
232+
"h.started_at as startedAt",
233+
"h.hydrated_at as hydratedAt",
234+
])
235+
.where("h.replica_id", "=", replicaId)
236+
.where("h.installed_at", ">=", timestampLit(startDate))
237+
.where("h.installed_at", "<=", timestampLit(endDate));
238+
}
239+
240+
export async function fetchEpisodeObjectHydrations({
241+
params,
242+
queryKey,
243+
requestOptions,
244+
}: {
245+
params: Parameters<typeof buildEpisodeObjectHydrationsQuery>[0];
246+
queryKey: QueryKey;
247+
requestOptions?: RequestInit;
248+
}) {
249+
const res = await executeSqlV2({
250+
queries: buildEpisodeObjectHydrationsQuery(params).compile(),
251+
queryKey,
252+
requestOptions,
253+
sessionVariables: { transaction_isolation: "serializable" },
254+
});
255+
return res.rows;
256+
}
257+
258+
/**
259+
* The highest CPU any of `replicaId`'s processes reported between `startDate`
260+
* and `endDate`, null when no sample falls in the window. Samples are a minute
261+
* apart, so this can miss the true peak. A lookup on
262+
* `mz_cluster_replica_metrics_history_ind`.
263+
*/
264+
export function buildReplicaCpuPeakQuery({
265+
replicaId,
266+
startDate,
267+
endDate,
268+
}: {
269+
replicaId: string;
270+
startDate: string;
271+
endDate: string;
272+
}) {
273+
return queryBuilder
274+
.selectFrom("mz_cluster_replica_metrics_history")
275+
.select(
276+
sql<number | null>`max(cpu_nano_cores)::float8`.as("peakCpuNanoCores"),
277+
)
278+
.where("replica_id", "=", replicaId)
279+
.where("occurred_at", ">=", timestampLit(startDate))
280+
.where("occurred_at", "<=", timestampLit(endDate));
281+
}
282+
283+
export async function fetchReplicaCpuPeak({
284+
params,
285+
queryKey,
286+
requestOptions,
287+
}: {
288+
params: Parameters<typeof buildReplicaCpuPeakQuery>[0];
289+
queryKey: QueryKey;
290+
requestOptions?: RequestInit;
291+
}) {
292+
const res = await executeSqlV2({
293+
queries: buildReplicaCpuPeakQuery(params).compile(),
294+
queryKey,
295+
requestOptions,
296+
sessionVariables: { transaction_isolation: "serializable" },
297+
});
298+
return res.rows[0]?.peakCpuNanoCores ?? null;
299+
}

‎console/src/api/materialize/cluster/replicaProcessMetrics.test.sql.ts‎

Lines changed: 4 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -7,56 +7,20 @@
77
// the Business Source License, use of this software will be governed
88
// by the Apache License, Version 2.0.
99

10-
import { Client } from "pg";
11-
12-
import { queryBuilder, SEARCH_PATH } from "~/api/materialize";
10+
import { SEARCH_PATH } from "~/api/materialize";
1311
import {
1412
executeSqlHttp,
1513
getMaterializeClient,
1614
} from "~/test/sql/materializeSqlClient";
1715
import { testdrive } from "~/test/sql/mzcompose";
16+
import { readSubscribeSnapshot } from "~/test/sql/subscribeSnapshot";
1817

1918
import {
2019
buildReplicaProcessMetricsSubscribe,
2120
buildReplicaSizeAllocationsQuery,
2221
} from "./replicaProcessMetrics";
2322

2423
const MOCKED_SEARCH_PATH = "internal_test, " + SEARCH_PATH;
25-
// Below the suite's `testTimeout`, so a snapshot that never completes fails on
26-
// the error below rather than on the timeout.
27-
const SUBSCRIBE_SNAPSHOT_TIMEOUT_MS = 30_000;
28-
29-
/**
30-
* A subscribe's snapshot, read through a cursor. The snapshot is complete at
31-
* the first progress message past the first message's timestamp.
32-
*/
33-
async function readSnapshot(
34-
client: Client,
35-
subscribe: ReturnType<typeof buildReplicaProcessMetricsSubscribe>,
36-
) {
37-
const { sql, parameters } = subscribe.compile(queryBuilder);
38-
const rows: Record<string, unknown>[] = [];
39-
await client.query("BEGIN");
40-
try {
41-
await client.query(`DECLARE c CURSOR FOR ${sql}`, [...parameters]);
42-
let firstTimestamp: bigint | undefined;
43-
const deadline = Date.now() + SUBSCRIBE_SNAPSHOT_TIMEOUT_MS;
44-
while (Date.now() < deadline) {
45-
const { rows: batch } = await client.query(
46-
`FETCH ALL c WITH (timeout='1s')`,
47-
);
48-
for (const row of batch) {
49-
const timestamp = BigInt(row.mz_timestamp);
50-
firstTimestamp ??= timestamp;
51-
if (!row.mz_progressed) rows.push(row);
52-
else if (timestamp > firstTimestamp) return rows;
53-
}
54-
}
55-
throw new Error("The subscribe's snapshot did not complete");
56-
} finally {
57-
await client.query("ROLLBACK");
58-
}
59-
}
6024

6125
describe("buildReplicaProcessMetricsSubscribe", () => {
6226
it("reads each process's samples for the given replicas from the literal start", async () => {
@@ -85,7 +49,7 @@ describe("buildReplicaProcessMetricsSubscribe", () => {
8549
const client = await getMaterializeClient();
8650
await client.query(`SET search_path TO ${MOCKED_SEARCH_PATH}`);
8751

88-
const rows = await readSnapshot(
52+
const rows = await readSubscribeSnapshot(
8953
client,
9054
buildReplicaProcessMetricsSubscribe({
9155
replicaIds: ["u1", "u3"],
@@ -161,7 +125,7 @@ describe("buildReplicaProcessMetricsSubscribe", () => {
161125
"SELECT r.id FROM mz_cluster_replicas r JOIN mz_clusters c ON c.id = r.cluster_id WHERE c.name = 'quickstart'",
162126
);
163127

164-
const rows = await readSnapshot(
128+
const rows = await readSubscribeSnapshot(
165129
client,
166130
buildReplicaProcessMetricsSubscribe({
167131
replicaIds: [replica.id],

‎console/src/api/materialize/subscribeSession.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,10 +107,20 @@ function createSubscribeSession<T extends object, R>(
107107
const unsubscribeChange = manager.onChange(() =>
108108
sink.apply(manager.getSnapshot()),
109109
);
110+
// NOTE: every connect sends the variables it's given, defaults included,
111+
// over those the manager was built with. Without this, each attempt would
112+
// drop the session's cluster and replica. Only set variables are passed, so
113+
// an unset cluster keeps the default rather than clearing it.
114+
const sessionVariables = Object.fromEntries(
115+
Object.entries(options.sessionVariables ?? {}).filter(
116+
([, value]) => value !== undefined,
117+
),
118+
);
110119
const connection = new WebsocketConnectionManager(
111120
manager,
112121
store,
113122
reconnectionStateAtom,
123+
{ getSessionVariables: () => sessionVariables },
114124
);
115125

116126
// The held rows belong to the previous region's catalog: reset rather than

0 commit comments

Comments
 (0)