Skip to content

console: add data for the redesigned cluster page's resource usage - #39507

Merged
leedqin merged 1 commit into
mainfrom
cluster-page-usage-data
Oct 21, 2026
Merged

leedqin merged 1 commit into
mainfrom
cluster-page-usage-data

Conversation

@leedqin

@leedqin leedqin commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Second PR in the cluster details redesign stack, on top of #39446. It adds the data behind the resource usage card: memory split into RAM and swap, replica status over time, hydration, heap limits and object creation times. The card itself is the next PR (#39508), so this one can be reviewed without the chart code. Nothing renders these queries yet.

Part of CNS-174.

Description

New queries. Each filters on literal replica or object ids, so it plans as an index lookup on mz_catalog_server, except where noted.

Data Source Refresh
RAM and swap as shares of the heap limit swap_of_ram_percent and heap_limit_percent on the console utilization views, 26.44 and later. Windows past 14 days use the ad-hoc query, which now computes both from mz_cluster_replica_metrics_history the way the views do. with the utilization data
Replica status transitions, and out-of-memory kills on any process mz_cluster_replica_status_history: process 0's rows, which carry a replica's status, plus any oom-killed row, since an OOM is recorded only on the process it killed when any of the replicas' processes changes status: the newest updated_at among the replica statuses that the app-wide clusters subscribe already carries, so the query doesn't poll. The cluster's own latestStatusUpdate can't serve, since it comes from mz_cluster_replica_history, which leaves out system clusters.
Finished hydrations mz_replica_hydration_history, 26.43 and later, where it gained process_id. No index, so it only runs for replicas that came back online inside the window. 60s
How many objects each current replica has yet to hydrate mz_compute_hydration_times 30s
How long each current replica took to hydrate each index and materialized view created in the window, null while it still is mz_compute_hydration_times 30s
Heap limit per replica mz_cluster_replica_metrics 60s, until every replica reports a heap limit, or reports memory with no limit, as the process orchestrator always does
Creation times of the cluster's indexes and materialized views mz_object_lifetimes, filtered to those object types since other kinds share the u<N> id form once per set of ids
A replica's per-process samples, for the Processes drawer in #39508 and the workers drawer in #39889 mz_cluster_replica_metrics_history, a lookup on its replica_id index live SUBSCRIBE while the drawer is open
Each size's process count and per-process CPU, memory and disk allocation, for the same drawers mz_cluster_replica_sizes, fast path on its index once

Per-process reads. The resource usage card charts replicas, not processes. This PR adds the per-process subscribe builder and the sizes query that the Processes drawer in #39508 and the workers drawer in #39889 read. The subscribe starts at a literal time one window back, and useRollingWindowStartMs moves that start forward once per window, so an open subscribe holds at most two windows of samples. A mz_now() temporal filter would instead hold each sample's pending retraction in the subscribe on mz_catalog_server: on a local emulator with 2 replicas of 16 processes over 24h, that measured ~12 MiB per open subscribe, against ~2 MiB with the literal bound. EXPLAIN on mz_catalog_server: a lookup on mz_cluster_replica_metrics_history_ind, with the bound as a plain filter. Per-process catalog views are tracked in CNS-203. useUtilizationTier holds the tier choice (3h subscribe, 24h subscribe, or poll) that the replica-level paths share.

useReplicaUtilizationHistory gets two options:

  • includeMemoryBreakdown adds the RAM and swap columns. Callers that don't pass it keep their query keys, so the classic page sends the same queries as before.
  • includeOfflineEvents: false skips the 20s offline-events poll on windows of 3h or less. The new page reads offline stretches from status history instead.

useClusterObjectsLive (in allObjectsCollection.ts) returns a cluster's objects of the given types as a TanStack DB live query over the existing allObjectsCollection. Filtering useAllObjects gives a new array on every DDL anywhere in the environment; the live query keeps its result until one of this cluster's matching objects changes. It reuses the app-wide objects subscribe, so it adds no catalog_server load.

resourceUsageModel.ts holds the chart logic as pure functions: status timelines, replica colors, memory bar segments, and merging buckets so bars stay readable on wide windows. Every merged group spans the same interval, clipped to the window, so replicas that start or stop mid-group still line up side by side.

The k6 cluster detail replay lists these queries' refresh intervals, since it replays unlisted queries every 5s. The testing guide's scalability commands now use yarn test:e2e:scalability, because the chromium project skips scalability/.

types/materialize.d.ts is edited by hand, with TODOs on the new columns, because yarn gen:types fails against current builds (pg_get_serial_sequence doesn't exist).

A replica's hydration in progress has no recorded start, so the timeline model doesn't draw one. The next PR shows it as a count of objects still hydrating at the now edge instead.

mz_replica_hydration_history only records a replica's hydration after it starts, so an object created on a running replica left no stretch. paintObjectHydration draws those from the per-object times. time_ns counts from when the replica installed the object's dataflow, so a stretch starts at the later of the object's creation and the replica's last start, and one still hydrating runs to the end of that run. Objects created before environmentd last started (now() - mz_uptime(), read in the same request) are left out: an upgrade, or a restart under the process orchestrator, puts every replica on new processes that rehydrate every object without a status-history row, so their times can't be placed.

Verification

  • replicaUtilizationBinning.test.ts: RAM and swap shares survive binning and come from the bucket's highest-heap sample.
  • allObjectsCollection.test.ts: useClusterObjectsLive returns the cluster's objects of the given types, keeps its result through changes on other clusters, and updates when one of its objects changes.
  • resourceUsageModel.test.ts: status timelines (state before the window, readings before the first transition once older history is gone, clipping, out-of-memory markers, hydration stretches), replica colors, memory bar segments, the RAM limit, and bucket merging.
  • replicaProcessMetrics.test.ts: the per-process subscribe bounds its samples with a literal start and no temporal filter.
  • replicaProcessMetrics.test.sql.ts, against a real Materialize:
    • the subscribe, read through a cursor over stand-in samples, returns only the given replicas' samples from the start, with process ids as text and a failed process's row kept with nulls;
    • it also runs against the catalog's own metrics history;
    • the sizes query matches mz_cluster_replica_sizes for the listed sizes.
  • replicaStatusTimeline.test.ts: status transitions come from process 0, and OOM kills from any process.
  • resourceUsageModel.test.ts also covers groups that start at different buckets, clipping to the window, a merged width capped at the window, and RAM limits that disagree between sizes that swap and sizes that don't.
  • queries.test.ts also covers: the newest status change across replicas and processes, system replicas included; the status history query refetching only when it moves, and keeping its rows meanwhile; the heap-limit poll stopping and continuing; and the rolling start being set on the first render.
  • resourceUsageModel.test.ts: paintObjectHydration anchors a stretch at the object's creation or the replica's last start, runs an unfinished one to the end of the run, and paints only running time.
  • queries.test.ts: useObjectHydrationStretches groups rows by replica, leaves out objects created before environmentd started, and stays idle without objects; past 14 days, the memory breakdown reaches the ad-hoc query on 26.44 and later only.
  • replicaUtilizationHistory.test.sql.ts: past 14 days, the ad-hoc query returns RAM and swap shares from each bucket's highest-heap sample, and none without the option.
  • replicaStatusTimeline.test.sql.ts: the hydration times query filters to the given replicas and objects, converts to milliseconds and keeps nulls, and reads a real index's hydration; environmentd's start lands before an index created during the test.
  • queries.test.ts: useRollingWindowStartMs starts one window back, moves once per window, never returns a start for another window, and takes a fresh start when a window comes back.
  • The plans in the table were checked with EXPLAIN on mz_catalog_server in a local v26.45 build, and the two per-process queries on a cloud environment.

🤖 Generated with Claude Code

@leedqin
leedqin added this pull request to stack #39509 October 2, 2026 23:17
@leedqin
leedqin force-pushed the cluster-page-usage-data branch 3 times, most recently from fc7f8c9 to d866e0e Compare October 5, 2026 15:19
@leedqin
leedqin marked this pull request as ready for review October 5, 2026 15:29
@leedqin
leedqin requested a review from a team as a code owner October 5, 2026 15:29
@leedqin
leedqin requested a review from Alphadelta14 October 5, 2026 15:29
@def-

def- commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Hydration history query is gated at 26.41 but needs a column added in 26.43

console/src/platform/clusters/queries.ts:798

REPLICA_HYDRATION_HISTORY_MIN_VERSION is "26.41.0", but buildReplicaHydrationEpisodesQuery filters on mz_replica_hydration_history.process_id, which first shipped in v26.43.0. On environments running v26.41.0 through v26.42.0 the gate lets the query through, and every poll fails with column "process_id" does not exist, so the redesigned card never shows finished hydration stretches there and reports "Hydration history is unavailable."

Details

At v26.41.0, v26.41.1, v26.41.2 and v26.42.0, the table in src/catalog/src/builtin/mz_internal.rs ends at the status column. process_id was added by #38864 (9297cd2), which git tag --contains places first in v26.43.0. The filter at console/src/api/materialize/cluster/replicaStatusTimeline.ts:124 (process_id IS NULL OR process_id = '0') therefore fails to plan on those versions. The IS NULL arm was clearly meant for rows written before the column existed, but that doesn't help when the column itself is missing. The hook keeps refetchInterval: 60_000 while it is in error, so a page left open sends a failing query to the environment every minute, plus react-query's retries. The Requires mz >= 26.41 doc at replicaStatusTimeline.ts:100 and the PR description's version table have the same off-by-two.

The 26.44 gate for the memory breakdown columns is correct: swap_of_ram_percent and heap_limit_percent are present in all three console utilization views at v26.44.0.

Fix: raise the constant to "26.43.0" and update the doc comment. Before 26.43 each episode is a single replica-wide row, so the other option is to add the process_id filter only at 26.43 and later. That keeps 26.41 and 26.42 supported at the cost of a second gate.

@leedqin
leedqin requested review from jdonelson and removed request for Alphadelta14 October 6, 2026 03:56
@linear-code

linear-code Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

CNS-174

CNS-173

@leedqin
leedqin force-pushed the cluster-page-usage-data branch from d866e0e to 423440e Compare October 6, 2026 22:20
@leedqin
leedqin force-pushed the cluster-page-usage-data branch from 423440e to 0e48230 Compare October 7, 2026 02:24
@leedqin

leedqin commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, confirmed: process_id first ships in v26.43.0. The hydration history gate is now 26.43.0, with the doc comment updated to match, in 0e48230.

Comment on lines +172 to +182

if (
isHydratingNow &&
currentRun?.state === "running" &&
currentRun.endMs === windowEndMs &&
currentRun.startMs > windowStartMs &&
!hydrationEpisodes.some(
(episode) => episode.startedAt.getTime() >= currentRun.startMs,
)
) {
segments = paintHydrating(segments, currentRun.startMs, windowEndMs);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the open question: I think the inference can go wrong in two ways. The collector visits one replica per hydration_history_collection_interval. In an environment with N replicas, a finished episode can take about N minutes to be recorded.

  • The stretch can be too long. Suppose a replica restarts and finishes hydrating, but the collector hasn't recorded that episode yet. Then someone creates a new index. The model sees an unhydrated object and no recorded episode since the restart. So it paints the whole time since the restart as hydrating. This lasts until the collector visits the replica.
  • The stretch can be missing. Suppose the collector has already recorded the restart episode. Then someone creates a new index. The model finds an episode that started after the restart. So it doesn't paint a hydrating stretch at all. The new index's hydration doesn't appear until its own episode is recorded.

Nothing on mz_catalog_server says when an in-progress hydration started. So any stretch drawn for it is a guess. Two options:

  1. Don't draw a stretch for in-progress hydration. Instead, show the current state at the "now" edge, for example "Hydrating · 3 objects". This is accurate, but less visual.
  2. Start the inferred stretch at the later of two times: the restart, or the earliest creation time among the replica's unhydrated objects. Then drop the episode check. This fixes both cases, but it's still an approximation. useHydratingReplicaIds would need to keep the object ids. They might also need mapping through mz_object_global_ids to match mz_object_lifetimes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a clear write-up. I went with option 1: the model no longer infers an in-progress stretch, and the status row shows a "Hydrating · N objects" badge at the now edge, counted per current replica from mz_compute_hydration_times (#39508). Finished episodes still draw from history.


// TODO: remove these gates once all environments are past them.
const REPLICA_MEMORY_BREAKDOWN_MIN_VERSION = "26.44.0";
const REPLICA_HYDRATION_HISTORY_MIN_VERSION = "26.43.0";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the commit message says hydration episodes need 26.41 and later, but the gate is 26.43 (for process_id), as the description says.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the commit message.

@leedqin
leedqin force-pushed the cluster-page-usage-data branch 2 times, most recently from 6c2b1c5 to aed7858 Compare October 8, 2026 04:11
@def-

def- commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Status timeline is blank before the first transition once a replica's older history has aged out

console/src/platform/clusters/ClusterPage/ResourceUsage/resourceUsageModel.ts:153

transformReplicaTimeline gets its starting state only from a transition before the window, and uses sampleRange only when there are no transitions at all. Replica status history older than 30 days is deleted on every environmentd start, including a replica's latest row, so a long-running replica that is OOM-killed inside the window shows an empty status row from the window start up to the kill, although its metrics show it running the whole time.

Details

partially_truncate_status_history with StatusHistoryRetentionPolicy::TimeWindow (src/storage-controller/src/collection_mgmt.rs:1676) retracts every row older than replica_status_history_retention_window (30 days by default). It does not keep the newest row per replica. For a replica that has been stable for more than 30 days, the only rows left after an OOM are offline(t1, oom-killed) and online(t2). The loop then emits segments starting at t1, and [windowStart, t1) has none. The card then draws only the background there, and stateAt returns undefined, so the tooltip shows no status. This is the main case the card is for: a long-running replica that starts running out of memory. The existing test "paints recorded hydration episodes over running time" passes with this gap ([offline(10), online(20)] over [0, 100] starts at 10).

The fix is to fill the stretch before the earliest transition from sampleRange (it has samples, so it was running) whenever that transition falls inside the window, and not only when sorted is empty.

2. LOW -- Heap-limit polling stops on a metrics row whose heap_limit is still null

console/src/platform/clusters/queries.ts:1139

useReplicaHeapLimits stops refetching once every replica has any row in mz_cluster_replica_metrics, even if that row's heap_limit is NULL. Combined with staleTime: Infinity, a replica that is new when the page fetches never gets its heap limit for the rest of the session, so the limit labels lose their byte values.

Details

The replica metrics task ticks right after provisioning (src/controller/src/clusters.rs:962). At that point the pod has no metrics yet, so get_metrics returns ServiceProcessMetrics::default() and the controller writes a row with every column NULL. The next tick is 60s later. A page opened on a just-created or just-resized cluster, or one whose currentReplicaIds change because of a resize, can fetch in that window and see the replica as "reported". This contradicts the PR's "refetched only while a replica hasn't reported one", and the comment at line 1132. The fix is to build reported only from rows with a non-null heapLimit.

@leedqin
leedqin force-pushed the cluster-page-usage-data branch from aed7858 to 3df23ed Compare October 12, 2026 23:36
@def-

def- commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Creation-times query matches clusters, replicas, roles and schemas that share an index's id

console/src/api/materialize/cluster/objectCreationTimes.ts:30

buildObjectCreationTimesQuery filters mz_object_lifetimes on id and event_type only. Cluster, replica, database, schema, role and network-policy ids use the same u<N> form as catalog item ids, so any of those created in the window under an index's or MV's id comes back as that object's creation. transformDdlEvents then labels it CREATE INDEX <name> at the wrong time, typically at a cluster resize.

Details

mz_object_lifetimes.id is details->>'id' for every audited object type, and details->>'replica_id' for replicas. Each of those kinds has its own id counter. On a fresh v26.47 dev image, I ran CREATE CLUSTER c SIZE '25cc', CREATE MATERIALIZED VIEW mv IN CLUSTER c ... (u2), CREATE INDEX i IN CLUSTER c ... (u3), then ALTER CLUSTER c SET (SIZE = '50cc'). The PR's query for ('u2', 'u3') returned five rows: the MV, the index, u2 cluster-replica, u3 cluster, and u3 cluster-replica. The last one is the resize. The card in #39508 draws a labeled marker for each row, so a resize shows up as CREATE INDEX i at the moment memory changes. That is the correlation the card exists to show. Two events for one id also give the markers duplicate React keys (key={event.id}).

The console's other mz_object_lifetimes joins (sourceList.ts, sinkList.ts, objectDetails.ts, fetchSecretsList.ts) also constrain object_type. The fix is to add .where("object_type", "in", ["index", "materialized-view"]), or to pass the caller's object types through.

@leedqin
leedqin force-pushed the cluster-page-usage-data branch 2 times, most recently from 24706e0 to 3763cce Compare October 15, 2026 15:19
@leedqin
leedqin force-pushed the cluster-page-usage-data branch 2 times, most recently from db7eda6 to f5713a4 Compare October 19, 2026 15:51
@def-

def- commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- OOM markers miss an OOM kill of any process other than 0 in a multi-process replica

console/src/api/materialize/cluster/replicaStatusTimeline.ts:37

buildReplicaStatusHistoryQuery reads only process 0's status rows, and transformReplicaTimeline places OOM markers from those rows' reason. Only the killed process records oom-killed. When it isn't process 0, process 0 is forced down with a plain crash, so the timeline shows an unexplained offline stretch with no OOM marker on exactly the replicas whose worst-process memory the card charts.

Details

The comment "Processes should share the same state" holds for online/offline but not for the reason. Timely processes are shared-fate (src/cluster/src/communication.rs:18). When process k is OOM-killed, the other processes hit a timely communication error and halt! with exit code 166 (src/timely-util/src/panic.rs:36, src/ore/src/process.rs:45). The Kubernetes orchestrator sets OfflineReason::OomKilled per pod, and only for exit codes 135, 137 and 167 (src/orchestrator-kubernetes/src/lib.rs:1368). So process k records offline / oom-killed, but process 0 records offline / NULL. message_cluster_event writes one row per process, with no replica-level aggregation.

As a result, oomAtMs stays empty, hasOomBetween is false, and in #39508 the red OOM bar, the OOM marker and the offlineOom cause all read as a plain offline. On a replica with N processes, that is every OOM that doesn't land on process 0. Meanwhile the memory bars come from the views' worst-process shares, which CONSOLE_MEMORY_SHARES_SQL picks because "a replica dies on one process being OOM-killed". So the chart can show one process climbing to its limit and then a restart with no OOM attached. The existing offline-events query and views make the same process-0 assumption, but nothing there places markers by reason.

One fix is to keep process 0 for state and also select OOM rows from every process, e.g. process_id = '0' OR reason = 'oom-killed'. This still filters after the replica_id lookup. Another process's OOM row then lands just before process 0's offline transition and puts the marker at the kill time.

@leedqin
leedqin force-pushed the cluster-page-usage-data branch 2 times, most recently from 7b5c631 to bddd20b Compare October 20, 2026 15:41
@leedqin
leedqin force-pushed the cluster-page-usage-data branch 2 times, most recently from a24cda1 to 51fc79d Compare October 20, 2026 19:39
@def-

def- commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Status history of system clusters is fetched once and never again

console/src/platform/clusters/queries.ts:1077

useReplicaStatusHistory refetches only when its latestStatusUpdate key changes, and the cluster's latestStatusUpdate is always NULL for system clusters. On the mz_catalog_server page, and on the other system clusters' pages, the status timeline stays as it was on first load. A replica that restarts or is OOM-killed while the page is open keeps showing as running, with no OOM marker.

Details

buildClustersQuery (console/src/api/materialize/cluster/clusterList.ts:26) takes max(occurred_at) through mz_cluster_replica_history. That view is built from audit events and filters out cluster_id !~~ 's%' (src/catalog/src/builtin/mz_internal.rs:7689). I checked this on a v26.47 dev image: s2 (mz_catalog_server) has status rows for replica s1, but its latest_status_update is NULL. After a kill -9 of that replica's clusterd, s1 recorded offline and then online, and latest_status_update stayed NULL.

The query has staleTime: Infinity and no interval, so with a fixed key it never refetches. Coming back to the page within react-query's 5-minute gc time reuses the old result too. replicasRestartedSince reads the same stale transitions, so the hydration episodes query is never enabled for the restarted replica. #39508 passes cluster.latestStatusUpdate, as the hook's doc suggests.

A key that works for every cluster is the latest updated_at across cluster.replicas[].statuses. The same subscribe carries it from mz_cluster_replica_statuses, and it changes on any process's transition, including an OOM kill of a process other than 0.

@leedqin
leedqin force-pushed the cluster-page-usage-data branch from 51fc79d to 24a0162 Compare October 20, 2026 20:31
@leedqin

leedqin commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Status history of system clusters is fetched once and never again is addressed in 24a0162, with the caller in 28cc31c. Thanks for checking it on a real system cluster. useReplicaStatusHistory is now keyed on replicaStatusesUpdatedAt(cluster.replicas): the newest updated_at across every replica process's status, which the clusters subscribe already carries from mz_cluster_replica_statuses. That covers system clusters, and an OOM kill of a process other than 0. queries.test.ts covers the helper, including system replicas. A new ResourceUsage.test.tsx test changes a replica's status while the cluster's latestStatusUpdate stays put, and the card refetches; it fails against the previous key.

@leedqin
leedqin force-pushed the cluster-page-usage-data branch from 24a0162 to 7b6dbc1 Compare October 20, 2026 23:23
@def-

def- commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Object hydration stretches are anchored at creation after an upgrade replaces the replica's processes

console/src/platform/clusters/ClusterPage/ResourceUsage/resourceUsageModel.ts:219

paintObjectHydration places time_ns at the later of the object's creation and the last run start in status history. An upgrade starts every replica on new clusterd processes, which reinstall every dataflow, but leaves no offline row in status history. So after each release, every index or MV created in the window before the upgrade gets a hydrating stretch at its creation time, and the stretch's length is the post-upgrade rehydration time. An object the new processes haven't hydrated yet is drawn as hydrating from its creation up to now.

Details

Replica services are named per deploy generation ({cluster}-replica-{replica}-gen-{n}, src/controller/src/clusters.rs:1034). A 0dt upgrade creates new processes for every replica. events_stream drops events for other generations (clusters.rs:660), and message_cluster_event writes no row in read-only mode or when the status hasn't changed (src/adapter/src/coord/message_handler.rs). So nothing marks the swap. At most a second online row appears, which mergeAdjacent and lastRunStartMs fold into the same run. time_ns counts from when the new process installed the export (ExportState::created_at, src/compute/src/logging/compute.rs). That includes the read-only rehydration, when all objects hydrate at once. The process orchestrator does the same thing: every environmentd restart relaunches every clusterd, and status history only gets an online row.

With the PR's model, transitions [online(-100), online(600)] over [0, 1000] and an object created at 200 with hydrationMs: 120 (the new generation's time) paint hydrating [200, 320). With hydrationMs: null they paint hydrating [200, 1000).

One conservative fix is to skip objects created before the environment's boot (now - mz_uptime()), because their time_ns can't be placed. The new processes install their dataflows at that boot, during a 0dt upgrade and on a process-orchestrator restart.

Base automatically changed from cluster-page-layout to main October 21, 2026 15:03
leedqin added a commit that referenced this pull request Oct 21, 2026
…39446)

### Motivation

Part of [CNS-173](https://linear.app/materializeinc/issue/CNS-173).

First PR in a stack that redesigns the cluster details page. The
redesign puts the object lists under one tab, moves cluster
configuration into a drawer, and replaces the overview charts with a
combined resource usage view. This PR lands only the page shell, so each
later PR adds one piece behind the same flag.

### Description

- The new page sits behind a new UI preview, `clusterDetailsRedesign`,
offered by the LaunchDarkly flag `cluster-details-page-cns-173`. Users
who are offered it opt in per browser from the pill in the breadcrumb
bar ("Show the new cluster details page" / "Show classic cluster details
page"). Self-managed keeps the classic page: it has no preview toggle
and treats unlisted flags as on, so `disabledFlexibleDeploymentFlags`
pins `cluster-details-page-cns-173` off.
- Tabs:
- **Metrics** renders the existing `ClusterOverview` until the resource
usage PR replaces it.
- **Objects** shows the existing materialized view, index, source and
sink lists behind a type switcher, at `objects/:objectType`.
- **Replicas** is temporary. The configuration drawer PR moves replica
management into the drawer and removes this tab.
- A title row shows the cluster name with its size and replica count, or
"Unmanaged" and "No replicas". It looks the cluster up itself, so a
status change on any other cluster in the environment doesn't re-render
the page. The configuration PR adds a gear here that opens the drawer.
- Links work across both pages. Classic tab URLs (`/indexes` and the
rest) redirect into the Objects tab, and `objects/<type>` URLs redirect
to the matching classic tab, or to Overview for an unknown type. A bare
`objects` link opens the first classic object tab.
- `ClusterDetailBreadcrumbs` moves to its own module, and both pages
render it from one shared header, which hosts the preview pill. Only the
title row, tabs and routes switch under it, so the pill stays mounted
and keeps keyboard focus when it switches pages.

Next in the stack: the resource usage data (#39507) and charts (#39508),
then memory by object, worker skew, and the configuration drawer.

The `cluster-details-page-cns-173` flag is on in LaunchDarkly's Staging
environment, which PR previews read. Production needs it too before the
preview can be offered there.

### Verification

`ClusterDetail.test.tsx` adds tests for:
- the preview switch in both directions, including opting in from a
classic tab URL, with keyboard focus staying on the pill;
- the new title and tabs, including an unmanaged cluster with no
replicas;
- the Objects type switcher;
- the redirects in both directions, including an unknown object type;
- keeping the current tab when switching clusters from the breadcrumb
menu.

`useUiPreview.test.tsx` adds a test that self-managed keeps the redesign
off.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@leedqin
leedqin force-pushed the cluster-page-usage-data branch from 7b6dbc1 to a0313dc Compare October 21, 2026 15:15
@leedqin

leedqin commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Object hydration stretches are anchored at creation after an upgrade replaces the replica's processes is addressed in 0362388. Thanks for tracing it through the generation filter. The hydration-times request now also reads now() - mz_uptime(), and useObjectHydrationStretches leaves out objects created before environmentd started, so neither the stretch at creation nor the one running to now is drawn. A replica restart after that start still shows its hydration from mz_replica_hydration_history. queries.test.ts covers an object created before the start being left out, and fails without the filter. replicaStatusTimeline.test.sql.ts checks that the start lands before an index created during the test.

@leedqin
leedqin force-pushed the cluster-page-usage-data branch from a0313dc to 0362388 Compare October 21, 2026 15:21
Second in the cluster details redesign stack. Adds the queries, hooks and
chart model that the resource usage card in the next PR draws from. Nothing
renders them yet.

Queries, all scoped to literal replica or object ids so they plan as index
lookups on mz_catalog_server, except where noted:
- RAM and swap as shares of the heap limit, selected from the console
  utilization views on 26.44 and later only. Callers that don't ask for
  them keep their query keys.
- Replica status history, for running and offline stretches and
  out-of-memory kills.
- Hydration episodes from mz_replica_hydration_history (26.43 and later,
  where it gained process_id). This has no index, so it only runs for
  replicas that came back online inside the window.
- How many compute objects each replica has yet to hydrate.
- Heap limits per replica, fetched once and refetched only while a
  replica hasn't reported one.
- Creation times of the cluster's indexes and materialized views, fetched
  once per id set.

Per-process reads, for the workers drawer later in the stack: a subscribe
builder for a replica's samples in mz_cluster_replica_metrics_history (a
lookup on its replica_id index) and each size's process count and
allocations from mz_cluster_replica_sizes. The subscribe starts at a
literal time, which useRollingWindowStartMs moves forward once per window,
since a mz_now() filter would hold every sample's pending retraction on
mz_catalog_server. useUtilizationTier holds the tier choice that the
replica-level paths share.

useReplicaUtilizationHistory gains an includeOfflineEvents option. The new
page reads offline stretches from status history, so it skips the 20s
offline-events poll on windows of 3h or less.

The chart model holds the pure logic: status timelines, replica colors,
memory bar segments, and merging buckets so bars stay readable on wide
windows.
@leedqin
leedqin force-pushed the cluster-page-usage-data branch from 0362388 to b7aa112 Compare October 21, 2026 21:16
@leedqin
leedqin merged commit df52e1f into main Oct 21, 2026
81 checks passed
@leedqin
leedqin deleted the cluster-page-usage-data branch October 21, 2026 22:07
leedqin added a commit that referenced this pull request Oct 21, 2026
…9508)

### Motivation

Third PR in the cluster details redesign stack, on top of #39507. It
replaces the Metrics tab's overview charts with the resource usage card
from the design. CPU, memory and replica status share one time axis and
one tooltip, so a spike, an out-of-memory kill and the restart after it
line up. Fixes
[CNS-174](https://linear.app/materializeinc/issue/CNS-174/show-cluster-memory-swap-metrics)

### Description

- **Toolbar:** above the cards, a Replica select ("All replicas", or one
replica as `r1 · 100cc`) and the time range picker shared with the
classic page. The selection is the `replica` search param, so a narrowed
page is a link, and a name no current replica has reads as All.
`useSelectedReplica` holds it for the page's other cards. With a replica
selected, the card shows that replica and dropped replicas of its name,
and the Processes drawer covers only it.
- **Header:** under All replicas, a toggle button per current replica
(color, name and size), usable from the keyboard. Dropped replicas, such
as a blue-green cluster's past deployments, fold into one "+N past
replicas" toggle that shows or hides them together, and its tooltip
lists them. Labels never wrap, and the header wraps at any count.
`useChartLegend`'s toggle now takes any click event, since it only reads
the modifier keys.
- **CPU:** a line per replica, as a share of the replica's CPU
allocation. When a reading passes 100%, the axis extends to fit it and a
dashed line marks the allocation.
- **Memory:** a bar per replica as a share of the heap limit (RAM plus
swap), on a fixed 0 to 100% axis so sizes with a lot of swap don't
stretch it. The replicas' bars at one time overlap by half a bar, so
each group reads as one point in time.
- On 26.44 and later the bar splits into memory and disk (swap), and the
legend gives the memory and heap limits in bytes.
  - Earlier environments show the bar as heap alone, labelled "Heap".
- Where no sample in the window has a heap limit, such as on the local
emulator, the bars are a share of the memory allocation, and the labels
say so. Only rows on 26.44 and later carry that, so older environments
keep the heap labels.
- **Replica status:** a row per replica, including one offline the whole
window. Running, finished hydration and offline stretches, a marker for
each out-of-memory kill, and a "Hydrating · N objects" badge at the now
edge while a replica has objects left to hydrate. An index or
materialized view created in the window adds its hydration on each
replica, from #39507's per-object times. A stretch narrower than 4px
draws at 4px, so a short hydration after a restart doesn't read as a
gap; the tooltip keeps the true times. A replica name wider than the
label column ends in an ellipsis, with the full name on hover.
- **No readings:** when no sample in the window has a reading, the card
says why from replica status: replicas ran but weren't measured, every
replica was offline after an out-of-memory kill (with the memory
troubleshooting guide), every replica was offline with no recorded
reason (with the hydration troubleshooting guide), or status is unknown.
Cloud adds a contact-support link; self-managed points at
metrics-server. Replica status always shows below.
- **Objects Created:** a lane under replica status, with its own heading
and a legend chip, and a tick where indexes and materialized views were
created. Creations less than 8px apart share a tick, which grows taller
with its count. Hovering or focusing a tick lists the time and objects
(the first five, then "and N more") and draws one faint guide through
the CPU and memory charts, which carry no DDL lines at rest. The objects
come from `useClusterObjectsLive`, so DDL elsewhere in the environment
doesn't redraw the charts. The cluster page mounts the objects
collection bridge, as Object Explorer does.
- **Replicas that share a name:** a scheduled cluster gets a new replica
id on every refresh, under the same name. Replicas of one cluster that
share a name share one toggle, one color, one status row and one tooltip
section. A past blue-green deployment's replicas keep their own, since
they run alongside the current ones while those hydrate. A time's memory
bars split only among the replicas reporting at that time, and bucket
merging counts the same. A CPU reading with no neighbor, such as a short
refresh merged into one bucket, draws as a dot.
- **Processes:** when a replica's size runs more than one process, a
Processes button on the CPU chart opens a drawer with a tab per such
replica; only the open tab reads. A menu picks what to compare processes
by, as on the Freshness page: right now, at any moment (peak, the
default), or on average. An average alone hides a process that is
climbing. Each tab leads with the busiest process for CPU and for memory
by that reading, or says the load is even. A process counts as busiest
only when it leads the others' mean by 1.25× and by at least 5
percentage points, so idle processes don't get named over a fraction of
a percent. Then one CPU chart and one memory chart draw every process as
its own line, and a reading with no neighbor as a dot. A legend on the
CPU chart names each drawn line by replica and process; past five
processes only the standouts get a color, and the rest share an "Other
processes" entry. A table (UniversalTable) lists each process's reading
right now, peak and average, computed from its samples, and bolds the
column the findings use. Right now is the newest round of samples, which
the controller writes for every process at one time, so a process that
is down there reads "-" and the drawer says it has no reading. A
checkbox per row draws that process's line, the header checkbox draws
all or none, and hovering a row highlights its line. CPU reads as a
share of one process's CPU allocation and memory as a share of its heap
limit, over the card's window up to 24h, and the drawer says so when the
card's window is longer. The drawer reads each process's samples from
`mz_cluster_replica_metrics_history` (a lookup on its `replica_id`
index) through a SUBSCRIBE that is open only while the drawer is,
starting at #39507's rolling literal start.
- **Tooltip:** per replica, its status and readings at the hovered time,
bytes first. CPU adds cores (the share times the size's CPU allocation;
sizes without a CPU limit get none). Memory and Disk read against their
own limits, the way the Clusters list's columns compute them from the
same 3h view (summed over processes, against the size's allocation times
its processes), so a recent bucket matches that page apart from the
bucket keeping its peak. A Total row gives memory and disk as a share of
the heap limit, so an out-of-memory kill reads at the limit.
- **Wide windows:** neighboring buckets merge, keeping each metric's
peak, so bars stay at least 10px wide. The time axis matches the classic
graphs.
- **Metrics tab:** the toolbar, the system cluster info box, this card,
then the existing resource intensive objects card. The freshness card is
off the new page; the classic page keeps it.

Loading and deriving live in `useResourceUsageData`. The chart marks are
memoized, in the card and in the drawer, so hovering only redraws the
cursor and tooltip.

The classic page's check for a cluster reporting no metrics moves into
`hasNoUtilizationMetrics`, which both pages now use. Its behavior is
unchanged.

Differences from the classic graphs: the CPU axis extends past 100%
instead of clamping, there's no disk chart, replica toggles replace the
replica dropdown, and windows of 3h or less don't poll offline events.

Not yet checked in Safari or Firefox.

### Verification
<img width="1285" height="767" alt="image"
src="https://github.com/user-attachments/assets/67fcd0d4-568c-44ab-bd9c-2e5d24b33baa"
/>

Verified using teleport to customer environments 
`ResourceUsage.test.tsx` renders the card against MSW handlers and
covers:
- each replica's CPU, memory and swap readings at a hovered time;
- heap alone on environments without the swap columns;
- memory labelled as a share of the allocation when no sample has a heap
limit;
- the CPU axis extending past 100% with the allocation line, and staying
at 100% otherwise;
- the out-of-memory marker in the tooltip;
- the limit labels with their byte sizes;
- a status row for a replica offline the whole window, and the status
rows in a window with no usage;
- the count of objects a replica has left to hydrate;
- hiding replicas with the toggles;
- the Objects Created lane: its heading and legend, a tick's tooltip and
chart guide on hover and on focus, a 7-object tick naming five and "and
2 more", and the empty lane;
- four past blue-green deployments folded into "+4 past replicas",
listed in its tooltip, shown and hidden together, with no label
wrapping;
- a long replica name truncated, with its full name as a title;
- a 25-minute hydration on a week's window drawn 4px wide, and an
object's hydration on a replica that was already running;
- tooltip rows against their own limits for a two-process replica, from
a fixture computed the way the Clusters list's view computes them, an
idle disk reading 0, CPU cores, and no cores for an unlimited size;
- on the real Metrics page: the Replica select narrowing the card and
writing `?replica=`, the URL selecting a replica, a stale name showing
all replicas, and the drawer covering only the selected replica;
- the empty window and the load error;
- a week of a scheduled cluster's refreshes: one toggle and one status
row for `r1`, a line per refresh with more than one point, and bars as
wide as their own bucket;
- one tooltip section per name while hovering a refresh; a refresh that
fits one merged bucket drawn as a dot; a past blue-green deployment's
`r1` kept apart from the current one;
- a Processes button only for sizes that run more than one process, and
a drawer naming the busiest process at any moment by default and on
average when picked, noting that it reads the last 24 hours of a 7-day
card, listing every process's latest, peak and average, and drawing a
line per process that the row and header checkboxes select or clear;
- the drawer's legend, which follows the checked processes; a process
down in the newest round, which has no reading right now; idle processes
read as even however large the ratio between them; and, past five
processes, color only for a standout, a neutral highlight for a hovered
grey line, a dot for a reading with no neighbor, and a capped list of
processes with no reading in the tooltip;
- status rows in the same order as the replica toggles;
- status history refetched when a replica changes status while the
cluster's latest status update stays put, as it always does on system
clusters;
- no fallback message while replica status is still loading.

`processReadingsModel.test.ts` covers each sample as a share of its
process's allocation, memory against the heap limit with the allocation
fallback, each process's peak per bucket, each process's latest, peak
and average from its samples (a process bursting one minute in twenty
doesn't outrank a steady one on average over a day), the first bucket
clipped to the window, the 5-point standout floor, and naming the
busiest process by each of them: a process climbing from 10% to 90% next
to a steady 50% is even on average, and busiest right now and at its
peak. The drawer test answers its SUBSCRIBE through a new
`mockSubscribeSocket` helper, which sends a completed snapshot over the
WebSocket.

`resourceUsageModel.test.ts` covers merging change ticks by pixel
distance and widening short status stretches.

`ClusterMetrics.test.tsx` covers the error shown when the clusters
subscribe fails before its first snapshot, and the freshness card
staying off with its flag on.

`ClusterDetail.test.tsx` mocks the new Metrics tab.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@bosconi bosconi mentioned this pull request Oct 23, 2026
@leedqin leedqin added the A-CONSOLE Area: Console label Oct 28, 2026
val-materialize added a commit that referenced this pull request Oct 30, 2026
Release notes for the **v26.48.0** train. One commit is added per
snapshot; the `final` snapshot sets the real dates. **Applied: `rc.1`,
`rc.2`, `rc.3`, `final` — the train is complete.**

- **Content**: 3 Feature sections (upsert hydration; Kafka sink headers
GA; MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS`), 9 Improvements, 2 Guides,
4 Bug Fixes. No **Agent Skills** section — every agent-skills PR in all
four windows was omitted.
- **Scope**: 123 PRs across the four snapshots — rc.1 114 (14 included),
rc.2 5 (0 included), rc.3 1 (1 included), final 3 (0 included).
- **Dates — now real, not provisional**: Cloud **2026-10-29**,
Self-Managed **2026-10-30**. The `rc.1`-anchored provisional estimate
was right, so `final` left both files unchanged. The `v26.48.0` tag's
commit date is 2026-10-28T18:35:05Z, a Wednesday; the first Thursday on
or after it is 2026-10-29, with Self-Managed the day after.
- **Operator compatibility**: row for `v26.48`, release date
`2026-10-30`. Unchanged by rc.2, rc.3, and final.
- **Branch base**: `main`, whose newest published section was `##
v26.46.1` when this branch was cut; v26.47.0 and v26.47.1 have since
merged in. `v26.48.0` stays at the top of the list.

### final increment — empty, and verified empty

`final`'s commit is empty. The tracking issue
([release#627](MaterializeInc/release#627))
lists exactly one item, the unassociated `4edd60a8d` "release: bump to
version v26.48.0", which the release-process-automation rule omits. The
three agent-skills PRs in the window
([#118](MaterializeInc/agent-skills#118),
[#119](MaterializeInc/agent-skills#119),
[#120](MaterializeInc/agent-skills#120)) were
all omitted.

This was **verified rather than assumed**, because a zero-PR snapshot
and a failed fetch produce identical output:
`compare/v26.48.0-rc.3...v26.48.0` returns `total_commits: 1`, and that
commit is the version bump. Nothing merged onto the release branch in
the 42 h 49 m between the `rc.3` tag and the ship tag. Across the train,
every agent-skills PR from
[#111](MaterializeInc/agent-skills#111) to
[#120](MaterializeInc/agent-skills#120) falls in
exactly one window, with no gaps or overlaps.

⚠️ **One decision is still open for the release owner — see "Upsert v2"
below.** Every flag this train held open was re-read in source at the
`v26.48.0` ship tag rather than inherited from the RC snapshots:

| Flag | Default at `v26.48.0` | Source | Held since | Disposition |
| -- | -- | -- | -- | -- |
| `enable_upsert_v2` | **`false`** |
`src/storage-types/src/dyncfgs.rs:505-509` | rc.2 | ⚠️ premise
contradicted — see below |
| `enable_upsert_chunked_stash` | **`false`** |
`src/storage-types/src/dyncfgs.rs:469-475` | rc.2 | same |
| `enable_upsert_paged_spill` | **`false`** |
`src/storage-types/src/dyncfgs.rs:450-455` | rc.2 | same |
| `enable_mcp_protocol_2026_07_28` | **`false`** |
`src/adapter-dyncfgs/src/dyncfgs.rs:295-299` | rc.1 | hold stands —
still omitted |
| `cluster-details-page-cns-173` | n/a (LaunchDarkly) | no source
default | rc.1 | hold stands — still omitted |

**The rc.2 hold on
[#40189](#40189) rests
on a premise this branch now contradicts.** rc.2 omitted it at
Borderline on exactly one ground — every measurement is taken on a path
`enable_upsert_v2` keeps off by default — and asked whether any
environment actually runs upsert v2. The branch answers that without
anyone feeding the answer back: the human-added **"Improved hydration
times and memory efficiency for upsert sources"** section states *"This
change has been rolled out to all Materialize Cloud environments and
applies to all existing and new clusters."* The three flags still read
`false` at the ship tag, which is exactly the
source-default-versus-LaunchDarkly divergence the rollout-lever rule
covers: the published docs call wins, and the sole stated reason for the
omit has fallen away.

- **Not already covered.** The published section is about **hydration**
— building a source from scratch, 206 min → 108–113 min, 542 GiB →
183–208 GiB. #40189 is about **catch-up after a backlog**, a different
operation with its own motivating incident
([CPU-320](https://linear.app/materializeinc/issue/CPU-320): a source
that *"could not catch up on any replica size after a producer rate
increase"*). A customer who read the hydration section would learn
nothing about catch-up.
- **Suggested Improvements bullet, to add only if you confirm** — it was
*not* applied in the `final` commit:
> **Faster catch-up for upsert sources with a large backlog**: An upsert
source working through a backlog now drains one timestamp per round
instead of merging and re-pushing the whole backlog each round, so
catch-up time grows linearly rather than quadratically with the backlog;
draining a 100-timestamp tail fell from 6.5 to 0.15 seconds, and a
400-timestamp tail drains in 0.67 seconds.
- **Caveat that must travel with the decision**: the baseline in those
timings is the *paged* flavor on `main`, not the configuration Cloud ran
before this rollout — so they are not a clean before/after for a given
customer. Dropping the two timings and keeping the
linear-versus-quadratic claim holds regardless of baseline.
- **Two weaker links, stated rather than buried**: that the rolled-out
configuration is the one #40189 was measured on is *inferred from the
published section's prose* (its "buffer pool … serve sequential work
from disk" reads as chunked stash plus paged spill), not confirmed by
anyone. And the author declared "Release notes: None", naming
`enable_upsert_v2` — so publishing this would be an **override** of an
author signal, justified by your own later rollout statement.
- **Third consecutive train** to hold this flag: `v26.44.0-rc.1`
([#39041](#39041)),
`v26.45.0-rc.1`
([#39044](#39044)),
`v26.48.0-rc.2` (#40189). If the rollout statement is accurate, the
earlier two are worth revisiting rather than left to age out.

**The other two holds stand, corroborated.**
`enable_mcp_protocol_2026_07_28` still reads `false` at the ship tag and
the five MCP PRs rc.1 held appear nowhere in the section; neither does
the Console cluster-details page stack. In both cases you had the whole
train in view and did not promote them — the confirmation rc.1 asked
for, arriving in the negative.

**No duplicates, re-checked at ship.** The seven PRs rc.1 flagged as
already covered by v26.47.0 are still absent, the `## v26.47.1` section
is untouched, and rc.3's entry
([#40065](#40065))
appears exactly once.

**Two notes on human edits to this branch**, both recorded as fact
rather than objection:

- `524adb8` removed the rc.1 feature section **"Native Google Cloud
Storage for persist"**
([#39983](#39983))
outright. Flagging it only because a published entry disappearing is
easy to miss in a long branch — if that was deliberate, nothing to do.
- Two rc.1 Improvements were promoted to their own feature sections with
docs links and a code sample (Kafka sink headers
[#39908](#39908);
MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS`
[#39873](#39873)), and
a **Guides** section was added that is not in the skill's template. The
lesson for next time: an option whose flag flipped to `true` in the same
release more likely warrants a feature section than the Improvements
bullet the rubric suggests by default.

**Carried forward from rc.3 and still open**: the `K8s recovery: envd on
failing node` nightly — the job that sank
[#40029](#40029), the
first attempt at what #40065 re-lands — **was not checked**; it needs
Buildkite, outside this run. The entry itself was verified correct in
source at both the rc.3 and ship tags, so this is a CI-confidence
question, not a notes-accuracy one.

### rc.3 increment

One Improvement, for #40065: a read-only `environmentd` that reboots to
pick up DDL changes during zero-downtime upgrade catch-up now restarts
in place rather than exiting its container, so Kubernetes no longer
applies crash-restart backoff. Not inherited from rc.1 — rc.1 correctly
omitted the first attempt (#40029) together with its revert (#40061) as
a no-op on `main`; rc.3 re-lands the behavior in narrowed form, verified
as a real change by comparing `entrypoint.sh` at the `v26.47.0` tag (no
retry loop) against `v26.48.0-rc.3` (re-exec on exit 75). The draft's
closing "so zero-downtime upgrades catch up faster" was cut: no cited PR
measures an upgrade duration, so the entry claims only the mechanism.

- **Not a Bug Fix.** 0dt upgrades completed on v26.47.0; the catch-up
reboots merely collected backoff delay. Existing behavior improved, so
Improvements.
- **Residual, deliberately unpublished.** A stale read-only process in a
replaced pod still restarts in place on these halts until the kubelet
kills its container, competing with its replacement for the generation's
replicas. The `MZ_RESTARTED_IN_PLACE` marker check keeps it from booting
writable, so it cannot fence the leader — the failure #40061 reverted.
Worth watching if 0dt reports look odd after this ships.

### rc.2 increment — empty, by design

rc.2's commit is empty: no entry was published and the provisional dates
did not move. It exists so later runs can see from the commit subject
that rc.2 was applied. Its single materialize PR, #40189, is the open
item discussed under the `final` increment above.

### rc.1 — standing warnings

⚠️ **Seven PRs in rc.1's tracking issue were already covered by the
v26.47.0 train, and none is republished here.** v26.47.0 took five RCs
whose later ones were cherry-picks onto the release branch; the same PRs
appear in this issue because they are also what landed on `main` for
v26.48.0. Four of the seven **shipped to Cloud in v26.47.0 on
2026-10-22** — #37653, #39910, #39927, #40036 — so republishing any of
them would put duplicate text in front of customers eight days later.
The other three (#39903, #39906, #40046) were held at Borderline in
v26.47.0 and are held again here, each reason re-checked rather than
inherited. The "snapshots are disjoint by construction" guarantee holds
*within* a train, not across trains, so this was caught mechanically by
grepping every PR number in the issue against the v26.47.0 snapshot
directories. All seven are listed under their own heading in
`rc.1/omitted.md`.

**Open questions for the release owner**

- **Upsert v2** (#40189, rc.2) — see the `final` increment above. The
rollout statement on this branch has changed the picture since rc.2;
this is the one item that needs a decision before merge.
- **MCP protocol revision 2026-07-28** (#39698, #39702, #39703, #39759,
#39761) — the stack is complete and `enable_mcp_protocol_2026_07_28`
still defaults to **`false`** at the ship tag. Omitted, with the absence
of any MCP text on this branch taken as confirmation. Drafted wording
remains in `rc.1/omitted.md` if the flag flips.
- **Redesigned cluster details page in the Console** (#39446, #39507,
#39508, #39510, #39607) — omitted for the same reason. Note that the
Improvements section does carry **"New cluster details page on
Self-Managed"**, added by hand, which links the v26.47 section rather
than re-describing the page.
- **Schema projection on the Console's Freshness page** (#40031) — a
genuine new way to diagnose lag, but the author states the whole
Freshness page "is currently behind a feature flag that is only enabled
for certain customers in impersonation mode". **Is the page generally
reachable yet?** This is the same page v26.47.0 held for the same
reason.
- **Time zone data 2026e timing** (#40045) — the author flags the
Canadian zone changes as effective **2026-11-01**, three days after the
Cloud date. Tight, but it holds.

*Resolved since rc.1: the Kafka sink headers framing question — the GA
section on this branch answers it.*

**Verification notes** — rc.1's Step 5 pass cut one drafted entry and
rewrote four:

- **Cut**: a Bug Fix drafted for #40032 ("halt with exit code 168 when a
live reader loses its lease"). The PR's own Problem section is about
alerting — `mz_persist_lease_timeout_read` cannot distinguish a dead
reader from a stuck one — and the follow-up it names is a Cloud alerting
change. Nothing a customer observes changes.
- **tzdata** (#40045): the draft said the Canadian zones move to
"permanent daylight saving time", the author's prose. The PR's data
table shows a permanent **−05/EST** and **−06/CST** — summer *offsets*
kept year-round, but with another zone's standard-time *abbreviations*,
which is what a user sees from `AT TIME ZONE`. Reworded to what the
table supports.
- **Hydration memory** (#39923): scoped to the consolidation step and
the measured outcome (~628 GiB peak, no restart, against every
pre-change run exiting on the 849 GiB limit). The PR explicitly says the
fix "covers this consolidation path, not the sink's whole footprint";
[CPU-317](https://linear.app/materializeinc/issue/CPU-317) tracks the
rest.
- **Kafka clients** (#40026): "55 to 24" is measured on a 4-worker,
11-source setup. The CPU throttling and degraded freshness are observed
symptoms of the *old* behavior at a few hundred sources, not a measured
post-change result at that scale, so the entry states the count as the
outcome and the throttling as what the count drove.
- **`environmentd` accept backlog** (#40035): the author's hedge is kept
— "less likely to be restarted", not eliminated, because the PR says
"This makes the queue fill later, not never."

**Judgment calls a reviewer may want to revisit** — #39605 is filed as
an **Improvement**, not a Bug Fix: nothing was incorrect, an unbounded
operation gained a bound and a knob. #40101 is filed as a **Bug Fix**
rather than omitted under the internal-logging rule, because credentials
reaching the logs is a security defect with a user-visible remedy; only
credentials a user inlined rather than wrapped in a `SECRET` were
affected.

Snapshot drafts (with PR links) in mz-skills:
- rc.1:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md)
- rc.2:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md)
- rc.3:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/omitted.md)
- final:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/omitted.md)

**Borderline PRs omitted:** rc.1: 12 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md#borderline),
rc.2: 1 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md#borderline),
rc.3: 0, final: 0

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Valerie Chiang <[email protected]>
Co-authored-by: Pranshu Maheshwari <[email protected]>
Co-authored-by: Pranshu Maheshwari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-CONSOLE Area: Console

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants