Skip to content

Commit 24a0162

Browse files
committed
console: add data for the redesigned cluster page's resource usage
Second in the cluster details redesign stack. Adds the queries, hooks and chart model that the resource usage card in the next PR draws from. Nothing renders them yet. Queries, all scoped to literal replica or object ids so they plan as index lookups on mz_catalog_server, except where noted: - RAM and swap as shares of the heap limit, selected from the console utilization views on 26.44 and later only. Callers that don't ask for them keep their query keys. - Replica status history, for running and offline stretches and out-of-memory kills. - Hydration episodes from mz_replica_hydration_history (26.43 and later, where it gained process_id). This has no index, so it only runs for replicas that came back online inside the window. - How many compute objects each replica has yet to hydrate. - Heap limits per replica, fetched once and refetched only while a replica hasn't reported one. - Creation times of the cluster's indexes and materialized views, fetched once per id set. Per-process reads, for the workers drawer later in the stack: a subscribe builder for a replica's samples in mz_cluster_replica_metrics_history (a lookup on its replica_id index) and each size's process count and allocations from mz_cluster_replica_sizes. The subscribe starts at a literal time, which useRollingWindowStartMs moves forward once per window, since a mz_now() filter would hold every sample's pending retraction on mz_catalog_server. useUtilizationTier holds the tier choice that the replica-level paths share. useReplicaUtilizationHistory gains an includeOfflineEvents option. The new page reads offline stretches from status history, so it skips the 20s offline-events poll on windows of 3h or less. The chart model holds the pure logic: status timelines, replica colors, memory bar segments, and merging buckets so bars stay readable on wide windows.
1 parent f74dd3d commit 24a0162

21 files changed

Lines changed: 2610 additions & 55 deletions

‎console/doc/guide-testing.md‎

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ CONSOLE_DEPLOYMENT_MODE='flexible-deployment' yarn start
171171

172172
# Terminal 3: Run tests (--workers=1 to avoid tests fighting over the same instance)
173173
cd console
174-
yarn playwright test scalability/ --project=chromium --workers=1
174+
yarn test:e2e:scalability scalability/ --workers=1
175175
```
176176

177177
#### Setup (staging)
@@ -183,7 +183,7 @@ export E2E_EMAIL=<your-staging-email>
183183
export E2E_PASSWORD=<your-staging-password>
184184
export CLUSTER_NAME=<cluster-name>
185185
export CLUSTER_ID=<cluster-id>
186-
yarn playwright test scalability/ --project=chromium
186+
yarn test:e2e:scalability scalability/
187187
```
188188

189189
#### Test files
@@ -199,11 +199,11 @@ Pages tested: cluster detail, cluster list, sources list, sinks list, object exp
199199

200200
```shell
201201
# Run all audits
202-
yarn playwright test scalability/query-audit --project=chromium
202+
yarn test:e2e:scalability scalability/query-audit
203203

204204
# Run one page
205-
yarn playwright test scalability/query-audit -g "cluster detail" --project=chromium
206-
yarn playwright test scalability/query-audit -g "object explorer" --project=chromium
205+
yarn test:e2e:scalability scalability/query-audit -g "cluster detail"
206+
yarn test:e2e:scalability scalability/query-audit -g "object explorer"
207207
```
208208

209209
**`tab-stress.spec.ts`** — Multi-tab stress tests
@@ -218,7 +218,7 @@ Two tests that measure how the console behaves with many tabs open:
218218
error checks.
219219

220220
```shell
221-
yarn playwright test scalability/tab-stress --project=chromium
221+
yarn test:e2e:scalability scalability/tab-stress
222222
```
223223

224224
**`page-load.spec.ts`** — UI snappiness
@@ -234,14 +234,14 @@ can be run individually.
234234

235235
```shell
236236
# All cold loads + navigation
237-
yarn playwright test scalability/page-load --project=chromium
237+
yarn test:e2e:scalability scalability/page-load
238238

239239
# Individual cold loads
240-
yarn playwright test scalability/page-load -g "cold load: cluster detail" --project=chromium
241-
yarn playwright test scalability/page-load -g "cold load: cluster list" --project=chromium
242-
yarn playwright test scalability/page-load -g "cold load: sources list" --project=chromium
243-
yarn playwright test scalability/page-load -g "cold load: sinks list" --project=chromium
244-
yarn playwright test scalability/page-load -g "cold load: object explorer" --project=chromium
240+
yarn test:e2e:scalability scalability/page-load -g "cold load: cluster detail"
241+
yarn test:e2e:scalability scalability/page-load -g "cold load: cluster list"
242+
yarn test:e2e:scalability scalability/page-load -g "cold load: sources list"
243+
yarn test:e2e:scalability scalability/page-load -g "cold load: sinks list"
244+
yarn test:e2e:scalability scalability/page-load -g "cold load: object explorer"
245245
```
246246

247247
#### Running tests per page
@@ -251,32 +251,32 @@ to avoid tests contending for the same Materialize instance.
251251

252252
**Cluster detail:**
253253
```shell
254-
yarn playwright test scalability/ -g "cluster detail|progressive tab" --project=chromium --workers=1
254+
yarn test:e2e:scalability scalability/ -g "cluster detail|progressive tab" --workers=1
255255
```
256256

257257
**Cluster list:**
258258
```shell
259-
yarn playwright test scalability/ -g "cluster list|navigation flow: cluster|mixed page" --project=chromium --workers=1
259+
yarn test:e2e:scalability scalability/ -g "cluster list|navigation flow: cluster|mixed page" --workers=1
260260
```
261261

262262
**Sources list:**
263263
```shell
264-
yarn playwright test scalability/ -g "sources list" --project=chromium --workers=1
264+
yarn test:e2e:scalability scalability/ -g "sources list" --workers=1
265265
```
266266

267267
**Sinks list:**
268268
```shell
269-
yarn playwright test scalability/ -g "sinks list" --project=chromium --workers=1
269+
yarn test:e2e:scalability scalability/ -g "sinks list" --workers=1
270270
```
271271

272272
**Object explorer:**
273273
```shell
274-
yarn playwright test scalability/ -g "object explorer|cold load: object explorer" --project=chromium --workers=1
274+
yarn test:e2e:scalability scalability/ -g "object explorer|cold load: object explorer" --workers=1
275275
```
276276

277277
**Everything:**
278278
```shell
279-
yarn playwright test scalability/ --project=chromium --workers=1
279+
yarn test:e2e:scalability scalability/ --workers=1
280280
```
281281

282282
#### Configuration

‎console/e2e-tests/k6/cluster-detail.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,15 @@ const POLL_INTERVAL_MS: Record<string, number> = {
7171
"clusters.maintainedObjectNames": Infinity,
7272
"clusters.replicaOfflineEvents": 20_000,
7373
"clusters.deploymentLineage": 300_000,
74+
// Redesigned cluster page (UI preview clusterDetailsRedesign).
75+
"clusters.unhydratedComputeObjects": 30_000,
76+
"clusters.replicaHydrationEpisodes": 60_000,
77+
// Fetched once. Status history refetches only when a replica's status
78+
// changes, and heap limits only while a replica's metrics fetch fails.
79+
"clusters.replicaStatusHistory": Infinity,
80+
"clusters.replicaHeapLimits": Infinity,
81+
"clusters.objectCreationTimes": Infinity,
82+
"clusters.replicaSizeAllocations": Infinity,
7483
"roles.owners": 300_000,
7584
// No refetchInterval in the console: fetched on mount and window refocus
7685
// only, so replay them once per VU rather than at the 5s default.
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
// Copyright Materialize, Inc. and contributors. All rights reserved.
2+
//
3+
// Use of this software is governed by the Business Source License
4+
// included in the LICENSE file.
5+
//
6+
// As of the Change Date specified in that file, in accordance with
7+
// the Business Source License, use of this software will be governed
8+
// by the Apache License, Version 2.0.
9+
10+
import { QueryKey } from "@tanstack/react-query";
11+
import { sql } from "kysely";
12+
13+
import { executeSqlV2, queryBuilder } from "~/api/materialize";
14+
15+
/**
16+
* When each of the given indexes and materialized views was created, for those
17+
* created at or after `startDate`. The literal id list makes this a lookup on
18+
* `mz_object_lifetimes_ind`.
19+
*/
20+
export function buildObjectCreationTimesQuery({
21+
objectIds,
22+
startDate,
23+
}: {
24+
objectIds: string[];
25+
startDate: string;
26+
}) {
27+
return (
28+
queryBuilder
29+
.selectFrom("mz_object_lifetimes")
30+
.select(["id", "occurred_at as occurredAt"])
31+
.where("id", "in", objectIds)
32+
// Clusters, replicas, roles and schemas share the `u<N>` id form, so an id
33+
// alone can match another kind's creation.
34+
.where("object_type", "in", ["index", "materialized-view"])
35+
.where("event_type", "=", "create")
36+
.where("occurred_at", ">=", sql<Date>`${sql.lit(startDate)}::timestamptz`)
37+
);
38+
}
39+
40+
export async function fetchObjectCreationTimes({
41+
params,
42+
queryKey,
43+
requestOptions,
44+
}: {
45+
params: Parameters<typeof buildObjectCreationTimesQuery>[0];
46+
queryKey: QueryKey;
47+
requestOptions?: RequestInit;
48+
}) {
49+
const res = await executeSqlV2({
50+
queries: buildObjectCreationTimesQuery(params).compile(),
51+
queryKey,
52+
requestOptions,
53+
sessionVariables: { transaction_isolation: "serializable" },
54+
});
55+
return res.rows;
56+
}
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
// Copyright Materialize, Inc. and contributors. All rights reserved.
2+
//
3+
// Use of this software is governed by the Business Source License
4+
// included in the LICENSE file.
5+
//
6+
// As of the Change Date specified in that file, in accordance with
7+
// the Business Source License, use of this software will be governed
8+
// by the Apache License, Version 2.0.
9+
10+
import { QueryKey } from "@tanstack/react-query";
11+
12+
import { executeSqlV2, queryBuilder } from "~/api/materialize";
13+
14+
/**
15+
* Each process's heap limit (RAM plus swap) for the given replicas, from their
16+
* latest metrics. The literal id list makes this a lookup on
17+
* `mz_cluster_replica_metrics_ind`.
18+
*
19+
* NOTE: callers fold the per-process rows, because a GROUP BY here turns the
20+
* fast-path lookup into a dataflow.
21+
*/
22+
export function buildReplicaHeapLimitsQuery({
23+
replicaIds,
24+
}: {
25+
replicaIds: string[];
26+
}) {
27+
return queryBuilder
28+
.selectFrom("mz_cluster_replica_metrics")
29+
.select([
30+
"replica_id as replicaId",
31+
"process_id as processId",
32+
"heap_limit as heapLimit",
33+
"memory_bytes as memoryBytes",
34+
])
35+
.where("replica_id", "in", replicaIds);
36+
}
37+
38+
export async function fetchReplicaHeapLimits({
39+
params,
40+
queryKey,
41+
requestOptions,
42+
}: {
43+
params: Parameters<typeof buildReplicaHeapLimitsQuery>[0];
44+
queryKey: QueryKey;
45+
requestOptions?: RequestInit;
46+
}) {
47+
const res = await executeSqlV2({
48+
queries: buildReplicaHeapLimitsQuery(params).compile(),
49+
queryKey,
50+
requestOptions,
51+
sessionVariables: { transaction_isolation: "serializable" },
52+
});
53+
return res.rows;
54+
}

0 commit comments

Comments
 (0)