Skip to content

Commit dfe1420

Browse files
jdonelsonclaude
andcommitted
console: Resolve freshness object names client-side
The cluster freshness query joined mz_objects, mz_clusters and mz_object_fully_qualified_names to name its lag readings. That last one is a view over three more relations, so one request planned as three full scans and no lookup. The useAllObjects subscribe already holds every name, schema, database and type, so useClusterFreshness now takes the objects to report on and sends only their IDs. On v26.45.0-dev that goes from five indexes and three full scans to a single index lookup. A literal IN list is the only form that reaches a lookup. It degrades to a full scan past 330 IDs, where an unnested array would hold a differential join. Taking the common case rather than encoding that threshold in our code. buildLagHistoryQuery keeps its joins for Environment Overview and the object panel, which do read names from SQL. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent d69a1ea commit dfe1420

4 files changed

Lines changed: 270 additions & 122 deletions

File tree

‎console/src/api/materialize/freshness/lagHistory.ts‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,86 @@ export type LagHistoryParameters = {
4545
includeSystemObjects?: boolean;
4646
};
4747

48+
/**
49+
* Binned lag readings for a known set of objects, with no name resolution.
50+
*
51+
* The Console already holds every object's name, schema, database and type in
52+
* the `useAllObjects` subscribe, so a caller that knows which objects it wants
53+
* needs three columns back, not eleven. `buildLagHistoryQuery` still joins
54+
* `mz_objects`, `mz_clusters` and `mz_object_fully_qualified_names` (itself a
55+
* view over `mz_objects`, `mz_schemas` and `mz_databases`) because the
56+
* environment-wide and single-object callers do depend on those names.
57+
*
58+
* NOTE: the IDs go in as a literal `IN` list because that is the only form
59+
* that reaches an index lookup on `mz_wallclock_global_lag_recent_history_ind`.
60+
* Measured on v26.45.0-dev: a literal list plans as `lookup` up to 330 entries
61+
* and as a full scan from 331, while an `unnest`ed array and an `IN (SELECT
62+
* ...)` subquery both plan as a differential join at every size. A cluster
63+
* past 330 objects therefore loses the lookup and reads the whole index.
64+
*/
65+
export function buildObjectLagHistoryQuery({
66+
objectIds,
67+
lookbackMs,
68+
}: {
69+
objectIds: string[];
70+
lookbackMs: number;
71+
}) {
72+
const bucketSizeMs = sql.raw(
73+
`${calculateBucketSizeFromLookback(lookbackMs)}`,
74+
);
75+
76+
return (
77+
queryBuilder
78+
.with("readings", (cte) =>
79+
cte
80+
.selectFrom("mz_wallclock_global_lag_recent_history")
81+
.select([
82+
sql<Date>`date_bin(
83+
'${bucketSizeMs} MILLISECONDS',
84+
occurred_at,
85+
TIMESTAMP '1970-01-01'
86+
)`.as("bucket_start"),
87+
"lag",
88+
"object_id",
89+
])
90+
.where("object_id", "in", objectIds)
91+
.where(
92+
(eb) =>
93+
sql`${eb.ref("occurred_at")} + INTERVAL '${sql.raw(`${lookbackMs}`)} MILLISECONDS'`,
94+
">=",
95+
sql<Date>`mz_now()`,
96+
),
97+
)
98+
.selectFrom("readings")
99+
// The reading kept per bin is the worst one in it, matching what
100+
// `buildLagHistoryQuery` reports.
101+
.distinctOn(["bucket_start", "object_id"])
102+
.select(["bucket_start as bucketStart", "object_id as objectId", "lag"])
103+
.orderBy(["bucket_start desc", "object_id", "lag desc"])
104+
);
105+
}
106+
107+
export async function fetchObjectLagHistory({
108+
objectIds,
109+
lookbackMs,
110+
queryKey,
111+
requestOptions,
112+
}: {
113+
objectIds: string[];
114+
lookbackMs: number;
115+
queryKey: QueryKey;
116+
requestOptions?: RequestInit;
117+
}) {
118+
return executeSqlV2({
119+
queries: buildObjectLagHistoryQuery({ objectIds, lookbackMs }).compile(),
120+
queryKey,
121+
requestOptions,
122+
sessionVariables: {
123+
transaction_isolation: "serializable",
124+
},
125+
});
126+
}
127+
48128
export function buildLagHistoryQuery(
49129
params: LagHistoryParameters = {
50130
lookback: { type: "getLatest" },

‎console/src/platform/clusters/ClusterOverview/ClusterFreshness.tsx‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ import {
4141
CurrentClusterFreshnessData,
4242
LINE_MAX_COUNT,
4343
useClusterFreshness,
44+
useFreshnessObjects,
4445
} from "../queries";
4546

4647
const TIME_PERIOD_OPTIONS = {
@@ -114,12 +115,10 @@ const FreshnessGraphWrapper = ({
114115
bucketSizeMs: number;
115116
clusterId: string;
116117
}) => {
118+
const objects = useFreshnessObjects(clusterId);
117119
const {
118120
data: { historicalData, currentData, startTime, endTime, lines },
119-
} = useClusterFreshness({
120-
lookbackMs,
121-
clusterId,
122-
});
121+
} = useClusterFreshness({ lookbackMs, objects });
123122

124123
return (
125124
<VStack alignItems="flex-start" width="100%" spacing="0" padding="4">

0 commit comments

Comments
 (0)