console: add redesigned cluster details layout behind a UI preview - #39446

…39507)

### Motivation

Second PR in the cluster details redesign stack, on top of #39446. It
adds the data behind the resource usage card: memory split into RAM and
swap, replica status over time, hydration, heap limits and object
creation times. The card itself is the next PR (#39508), so this one can
be reviewed without the chart code. Nothing renders these queries yet.

Part of
[CNS-174](https://linear.app/materializeinc/issue/CNS-174/show-cluster-memory-swap-metrics).

### Description

New queries. Each filters on literal replica or object ids, so it plans
as an index lookup on `mz_catalog_server`, except where noted.

| Data | Source | Refresh |
|---|---|---|
| RAM and swap as shares of the heap limit | `swap_of_ram_percent` and
`heap_limit_percent` on the console utilization views, 26.44 and later.
Windows past 14 days use the ad-hoc query, which now computes both from
`mz_cluster_replica_metrics_history` the way the views do. | with the
utilization data |
| Replica status transitions, and out-of-memory kills on any process |
`mz_cluster_replica_status_history`: process 0's rows, which carry a
replica's status, plus any `oom-killed` row, since an OOM is recorded
only on the process it killed | when any of the replicas' processes
changes status: the newest `updated_at` among the replica statuses that
the app-wide clusters subscribe already carries, so the query doesn't
poll. The cluster's own `latestStatusUpdate` can't serve, since it comes
from `mz_cluster_replica_history`, which leaves out system clusters. |
| Finished hydrations | `mz_replica_hydration_history`, 26.43 and later,
where it gained `process_id`. No index, so it only runs for replicas
that came back online inside the window. | 60s |
| How many objects each current replica has yet to hydrate |
`mz_compute_hydration_times` | 30s |
| How long each current replica took to hydrate each index and
materialized view created in the window, null while it still is |
`mz_compute_hydration_times` | 30s |
| Heap limit per replica | `mz_cluster_replica_metrics` | 60s, until
every replica reports a heap limit, or reports memory with no limit, as
the process orchestrator always does |
| Creation times of the cluster's indexes and materialized views |
`mz_object_lifetimes`, filtered to those object types since other kinds
share the `u<N>` id form | once per set of ids |
| A replica's per-process samples, for the Processes drawer in #39508
and the workers drawer in #39889 | `mz_cluster_replica_metrics_history`,
a lookup on its `replica_id` index | live SUBSCRIBE while the drawer is
open |
| Each size's process count and per-process CPU, memory and disk
allocation, for the same drawers | `mz_cluster_replica_sizes`, fast path
on its index | once |

**Per-process reads.** The resource usage card charts replicas, not
processes. This PR adds the per-process subscribe builder and the sizes
query that the Processes drawer in #39508 and the workers drawer in
#39889 read. The subscribe starts at a literal time one window back, and
`useRollingWindowStartMs` moves that start forward once per window, so
an open subscribe holds at most two windows of samples. A `mz_now()`
temporal filter would instead hold each sample's pending retraction in
the subscribe on `mz_catalog_server`: on a local emulator with 2
replicas of 16 processes over 24h, that measured ~12 MiB per open
subscribe, against ~2 MiB with the literal bound. EXPLAIN on
`mz_catalog_server`: a lookup on
`mz_cluster_replica_metrics_history_ind`, with the bound as a plain
filter. Per-process catalog views are tracked in
[CNS-203](https://linear.app/materializeinc/issue/CNS-203).
`useUtilizationTier` holds the tier choice (3h subscribe, 24h subscribe,
or poll) that the replica-level paths share.

`useReplicaUtilizationHistory` gets two options:
- `includeMemoryBreakdown` adds the RAM and swap columns. Callers that
don't pass it keep their query keys, so the classic page sends the same
queries as before.
- `includeOfflineEvents: false` skips the 20s offline-events poll on
windows of 3h or less. The new page reads offline stretches from status
history instead.

`useClusterObjectsLive` (in `allObjectsCollection.ts`) returns a
cluster's objects of the given types as a TanStack DB live query over
the existing `allObjectsCollection`. Filtering `useAllObjects` gives a
new array on every DDL anywhere in the environment; the live query keeps
its result until one of this cluster's matching objects changes. It
reuses the app-wide objects subscribe, so it adds no catalog_server
load.

`resourceUsageModel.ts` holds the chart logic as pure functions: status
timelines, replica colors, memory bar segments, and merging buckets so
bars stay readable on wide windows. Every merged group spans the same
interval, clipped to the window, so replicas that start or stop
mid-group still line up side by side.

The k6 cluster detail replay lists these queries' refresh intervals,
since it replays unlisted queries every 5s. The testing guide's
scalability commands now use `yarn test:e2e:scalability`, because the
`chromium` project skips `scalability/`.

`types/materialize.d.ts` is edited by hand, with TODOs on the new
columns, because `yarn gen:types` fails against current builds
(`pg_get_serial_sequence` doesn't exist).

A replica's hydration in progress has no recorded start, so the timeline
model doesn't draw one. The next PR shows it as a count of objects still
hydrating at the now edge instead.

`mz_replica_hydration_history` only records a replica's hydration after
it starts, so an object created on a running replica left no stretch.
`paintObjectHydration` draws those from the per-object times. `time_ns`
counts from when the replica installed the object's dataflow, so a
stretch starts at the later of the object's creation and the replica's
last start, and one still hydrating runs to the end of that run. Objects
created before environmentd last started (`now() - mz_uptime()`, read in
the same request) are left out: an upgrade, or a restart under the
process orchestrator, puts every replica on new processes that rehydrate
every object without a status-history row, so their times can't be
placed.

### Verification

- `replicaUtilizationBinning.test.ts`: RAM and swap shares survive
binning and come from the bucket's highest-heap sample.
- `allObjectsCollection.test.ts`: `useClusterObjectsLive` returns the
cluster's objects of the given types, keeps its result through changes
on other clusters, and updates when one of its objects changes.
- `resourceUsageModel.test.ts`: status timelines (state before the
window, readings before the first transition once older history is gone,
clipping, out-of-memory markers, hydration stretches), replica colors,
memory bar segments, the RAM limit, and bucket merging.
- `replicaProcessMetrics.test.ts`: the per-process subscribe bounds its
samples with a literal start and no temporal filter.
- `replicaProcessMetrics.test.sql.ts`, against a real Materialize:
- the subscribe, read through a cursor over stand-in samples, returns
only the given replicas' samples from the start, with process ids as
text and a failed process's row kept with nulls;
  - it also runs against the catalog's own metrics history;
- the sizes query matches `mz_cluster_replica_sizes` for the listed
sizes.
- `replicaStatusTimeline.test.ts`: status transitions come from process
0, and OOM kills from any process.
- `resourceUsageModel.test.ts` also covers groups that start at
different buckets, clipping to the window, a merged width capped at the
window, and RAM limits that disagree between sizes that swap and sizes
that don't.
- `queries.test.ts` also covers: the newest status change across
replicas and processes, system replicas included; the status history
query refetching only when it moves, and keeping its rows meanwhile; the
heap-limit poll stopping and continuing; and the rolling start being set
on the first render.
- `resourceUsageModel.test.ts`: `paintObjectHydration` anchors a stretch
at the object's creation or the replica's last start, runs an unfinished
one to the end of the run, and paints only running time.
- `queries.test.ts`: `useObjectHydrationStretches` groups rows by
replica, leaves out objects created before environmentd started, and
stays idle without objects; past 14 days, the memory breakdown reaches
the ad-hoc query on 26.44 and later only.
- `replicaUtilizationHistory.test.sql.ts`: past 14 days, the ad-hoc
query returns RAM and swap shares from each bucket's highest-heap
sample, and none without the option.
- `replicaStatusTimeline.test.sql.ts`: the hydration times query filters
to the given replicas and objects, converts to milliseconds and keeps
nulls, and reads a real index's hydration; environmentd's start lands
before an index created during the test.
- `queries.test.ts`: `useRollingWindowStartMs` starts one window back,
moves once per window, never returns a start for another window, and
takes a fresh start when a window comes back.
- The plans in the table were checked with EXPLAIN on
`mz_catalog_server` in a local v26.45 build, and the two per-process
queries on a cloud environment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Release notes for the **v26.48.0** train. One commit is added per
snapshot; the `final` snapshot sets the real dates. **Applied: `rc.1`,
`rc.2`, `rc.3`, `final` — the train is complete.**

- **Content**: 3 Feature sections (upsert hydration; Kafka sink headers
GA; MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS`), 9 Improvements, 2 Guides,
4 Bug Fixes. No **Agent Skills** section — every agent-skills PR in all
four windows was omitted.
- **Scope**: 123 PRs across the four snapshots — rc.1 114 (14 included),
rc.2 5 (0 included), rc.3 1 (1 included), final 3 (0 included).
- **Dates — now real, not provisional**: Cloud **2026-10-29**,
Self-Managed **2026-10-30**. The `rc.1`-anchored provisional estimate
was right, so `final` left both files unchanged. The `v26.48.0` tag's
commit date is 2026-10-28T18:35:05Z, a Wednesday; the first Thursday on
or after it is 2026-10-29, with Self-Managed the day after.
- **Operator compatibility**: row for `v26.48`, release date
`2026-10-30`. Unchanged by rc.2, rc.3, and final.
- **Branch base**: `main`, whose newest published section was `##
v26.46.1` when this branch was cut; v26.47.0 and v26.47.1 have since
merged in. `v26.48.0` stays at the top of the list.

### final increment — empty, and verified empty

`final`'s commit is empty. The tracking issue
([release#627](MaterializeInc/release#627))
lists exactly one item, the unassociated `4edd60a8d` "release: bump to
version v26.48.0", which the release-process-automation rule omits. The
three agent-skills PRs in the window
([#118](MaterializeInc/agent-skills#118),
[#119](MaterializeInc/agent-skills#119),
[#120](MaterializeInc/agent-skills#120)) were
all omitted.

This was **verified rather than assumed**, because a zero-PR snapshot
and a failed fetch produce identical output:
`compare/v26.48.0-rc.3...v26.48.0` returns `total_commits: 1`, and that
commit is the version bump. Nothing merged onto the release branch in
the 42 h 49 m between the `rc.3` tag and the ship tag. Across the train,
every agent-skills PR from
[#111](MaterializeInc/agent-skills#111) to
[#120](MaterializeInc/agent-skills#120) falls in
exactly one window, with no gaps or overlaps.

⚠️ **One decision is still open for the release owner — see "Upsert v2"
below.** Every flag this train held open was re-read in source at the
`v26.48.0` ship tag rather than inherited from the RC snapshots:

| Flag | Default at `v26.48.0` | Source | Held since | Disposition |
| -- | -- | -- | -- | -- |
| `enable_upsert_v2` | **`false`** |
`src/storage-types/src/dyncfgs.rs:505-509` | rc.2 | ⚠️ premise
contradicted — see below |
| `enable_upsert_chunked_stash` | **`false`** |
`src/storage-types/src/dyncfgs.rs:469-475` | rc.2 | same |
| `enable_upsert_paged_spill` | **`false`** |
`src/storage-types/src/dyncfgs.rs:450-455` | rc.2 | same |
| `enable_mcp_protocol_2026_07_28` | **`false`** |
`src/adapter-dyncfgs/src/dyncfgs.rs:295-299` | rc.1 | hold stands —
still omitted |
| `cluster-details-page-cns-173` | n/a (LaunchDarkly) | no source
default | rc.1 | hold stands — still omitted |

**The rc.2 hold on
[#40189](#40189) rests
on a premise this branch now contradicts.** rc.2 omitted it at
Borderline on exactly one ground — every measurement is taken on a path
`enable_upsert_v2` keeps off by default — and asked whether any
environment actually runs upsert v2. The branch answers that without
anyone feeding the answer back: the human-added **"Improved hydration
times and memory efficiency for upsert sources"** section states *"This
change has been rolled out to all Materialize Cloud environments and
applies to all existing and new clusters."* The three flags still read
`false` at the ship tag, which is exactly the
source-default-versus-LaunchDarkly divergence the rollout-lever rule
covers: the published docs call wins, and the sole stated reason for the
omit has fallen away.

- **Not already covered.** The published section is about **hydration**
— building a source from scratch, 206 min → 108–113 min, 542 GiB →
183–208 GiB. #40189 is about **catch-up after a backlog**, a different
operation with its own motivating incident
([CPU-320](https://linear.app/materializeinc/issue/CPU-320): a source
that *"could not catch up on any replica size after a producer rate
increase"*). A customer who read the hydration section would learn
nothing about catch-up.
- **Suggested Improvements bullet, to add only if you confirm** — it was
*not* applied in the `final` commit:
> **Faster catch-up for upsert sources with a large backlog**: An upsert
source working through a backlog now drains one timestamp per round
instead of merging and re-pushing the whole backlog each round, so
catch-up time grows linearly rather than quadratically with the backlog;
draining a 100-timestamp tail fell from 6.5 to 0.15 seconds, and a
400-timestamp tail drains in 0.67 seconds.
- **Caveat that must travel with the decision**: the baseline in those
timings is the *paged* flavor on `main`, not the configuration Cloud ran
before this rollout — so they are not a clean before/after for a given
customer. Dropping the two timings and keeping the
linear-versus-quadratic claim holds regardless of baseline.
- **Two weaker links, stated rather than buried**: that the rolled-out
configuration is the one #40189 was measured on is *inferred from the
published section's prose* (its "buffer pool … serve sequential work
from disk" reads as chunked stash plus paged spill), not confirmed by
anyone. And the author declared "Release notes: None", naming
`enable_upsert_v2` — so publishing this would be an **override** of an
author signal, justified by your own later rollout statement.
- **Third consecutive train** to hold this flag: `v26.44.0-rc.1`
([#39041](#39041)),
`v26.45.0-rc.1`
([#39044](#39044)),
`v26.48.0-rc.2` (#40189). If the rollout statement is accurate, the
earlier two are worth revisiting rather than left to age out.

**The other two holds stand, corroborated.**
`enable_mcp_protocol_2026_07_28` still reads `false` at the ship tag and
the five MCP PRs rc.1 held appear nowhere in the section; neither does
the Console cluster-details page stack. In both cases you had the whole
train in view and did not promote them — the confirmation rc.1 asked
for, arriving in the negative.

**No duplicates, re-checked at ship.** The seven PRs rc.1 flagged as
already covered by v26.47.0 are still absent, the `## v26.47.1` section
is untouched, and rc.3's entry
([#40065](#40065))
appears exactly once.

**Two notes on human edits to this branch**, both recorded as fact
rather than objection:

- `524adb8` removed the rc.1 feature section **"Native Google Cloud
Storage for persist"**
([#39983](#39983))
outright. Flagging it only because a published entry disappearing is
easy to miss in a long branch — if that was deliberate, nothing to do.
- Two rc.1 Improvements were promoted to their own feature sections with
docs links and a code sample (Kafka sink headers
[#39908](#39908);
MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS`
[#39873](#39873)), and
a **Guides** section was added that is not in the skill's template. The
lesson for next time: an option whose flag flipped to `true` in the same
release more likely warrants a feature section than the Improvements
bullet the rubric suggests by default.

**Carried forward from rc.3 and still open**: the `K8s recovery: envd on
failing node` nightly — the job that sank
[#40029](#40029), the
first attempt at what #40065 re-lands — **was not checked**; it needs
Buildkite, outside this run. The entry itself was verified correct in
source at both the rc.3 and ship tags, so this is a CI-confidence
question, not a notes-accuracy one.

### rc.3 increment

One Improvement, for #40065: a read-only `environmentd` that reboots to
pick up DDL changes during zero-downtime upgrade catch-up now restarts
in place rather than exiting its container, so Kubernetes no longer
applies crash-restart backoff. Not inherited from rc.1 — rc.1 correctly
omitted the first attempt (#40029) together with its revert (#40061) as
a no-op on `main`; rc.3 re-lands the behavior in narrowed form, verified
as a real change by comparing `entrypoint.sh` at the `v26.47.0` tag (no
retry loop) against `v26.48.0-rc.3` (re-exec on exit 75). The draft's
closing "so zero-downtime upgrades catch up faster" was cut: no cited PR
measures an upgrade duration, so the entry claims only the mechanism.

- **Not a Bug Fix.** 0dt upgrades completed on v26.47.0; the catch-up
reboots merely collected backoff delay. Existing behavior improved, so
Improvements.
- **Residual, deliberately unpublished.** A stale read-only process in a
replaced pod still restarts in place on these halts until the kubelet
kills its container, competing with its replacement for the generation's
replicas. The `MZ_RESTARTED_IN_PLACE` marker check keeps it from booting
writable, so it cannot fence the leader — the failure #40061 reverted.
Worth watching if 0dt reports look odd after this ships.

### rc.2 increment — empty, by design

rc.2's commit is empty: no entry was published and the provisional dates
did not move. It exists so later runs can see from the commit subject
that rc.2 was applied. Its single materialize PR, #40189, is the open
item discussed under the `final` increment above.

### rc.1 — standing warnings

⚠️ **Seven PRs in rc.1's tracking issue were already covered by the
v26.47.0 train, and none is republished here.** v26.47.0 took five RCs
whose later ones were cherry-picks onto the release branch; the same PRs
appear in this issue because they are also what landed on `main` for
v26.48.0. Four of the seven **shipped to Cloud in v26.47.0 on
2026-10-22** — #37653, #39910, #39927, #40036 — so republishing any of
them would put duplicate text in front of customers eight days later.
The other three (#39903, #39906, #40046) were held at Borderline in
v26.47.0 and are held again here, each reason re-checked rather than
inherited. The "snapshots are disjoint by construction" guarantee holds
*within* a train, not across trains, so this was caught mechanically by
grepping every PR number in the issue against the v26.47.0 snapshot
directories. All seven are listed under their own heading in
`rc.1/omitted.md`.

**Open questions for the release owner**

- **Upsert v2** (#40189, rc.2) — see the `final` increment above. The
rollout statement on this branch has changed the picture since rc.2;
this is the one item that needs a decision before merge.
- **MCP protocol revision 2026-07-28** (#39698, #39702, #39703, #39759,
#39761) — the stack is complete and `enable_mcp_protocol_2026_07_28`
still defaults to **`false`** at the ship tag. Omitted, with the absence
of any MCP text on this branch taken as confirmation. Drafted wording
remains in `rc.1/omitted.md` if the flag flips.
- **Redesigned cluster details page in the Console** (#39446, #39507,
#39508, #39510, #39607) — omitted for the same reason. Note that the
Improvements section does carry **"New cluster details page on
Self-Managed"**, added by hand, which links the v26.47 section rather
than re-describing the page.
- **Schema projection on the Console's Freshness page** (#40031) — a
genuine new way to diagnose lag, but the author states the whole
Freshness page "is currently behind a feature flag that is only enabled
for certain customers in impersonation mode". **Is the page generally
reachable yet?** This is the same page v26.47.0 held for the same
reason.
- **Time zone data 2026e timing** (#40045) — the author flags the
Canadian zone changes as effective **2026-11-01**, three days after the
Cloud date. Tight, but it holds.

*Resolved since rc.1: the Kafka sink headers framing question — the GA
section on this branch answers it.*

**Verification notes** — rc.1's Step 5 pass cut one drafted entry and
rewrote four:

- **Cut**: a Bug Fix drafted for #40032 ("halt with exit code 168 when a
live reader loses its lease"). The PR's own Problem section is about
alerting — `mz_persist_lease_timeout_read` cannot distinguish a dead
reader from a stuck one — and the follow-up it names is a Cloud alerting
change. Nothing a customer observes changes.
- **tzdata** (#40045): the draft said the Canadian zones move to
"permanent daylight saving time", the author's prose. The PR's data
table shows a permanent **−05/EST** and **−06/CST** — summer *offsets*
kept year-round, but with another zone's standard-time *abbreviations*,
which is what a user sees from `AT TIME ZONE`. Reworded to what the
table supports.
- **Hydration memory** (#39923): scoped to the consolidation step and
the measured outcome (~628 GiB peak, no restart, against every
pre-change run exiting on the 849 GiB limit). The PR explicitly says the
fix "covers this consolidation path, not the sink's whole footprint";
[CPU-317](https://linear.app/materializeinc/issue/CPU-317) tracks the
rest.
- **Kafka clients** (#40026): "55 to 24" is measured on a 4-worker,
11-source setup. The CPU throttling and degraded freshness are observed
symptoms of the *old* behavior at a few hundred sources, not a measured
post-change result at that scale, so the entry states the count as the
outcome and the throttling as what the count drove.
- **`environmentd` accept backlog** (#40035): the author's hedge is kept
— "less likely to be restarted", not eliminated, because the PR says
"This makes the queue fill later, not never."

**Judgment calls a reviewer may want to revisit** — #39605 is filed as
an **Improvement**, not a Bug Fix: nothing was incorrect, an unbounded
operation gained a bound and a knob. #40101 is filed as a **Bug Fix**
rather than omitted under the internal-logging rule, because credentials
reaching the logs is a security defect with a user-visible remedy; only
credentials a user inlined rather than wrapped in a `SECRET` were
affected.

Snapshot drafts (with PR links) in mz-skills:
- rc.1:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md)
- rc.2:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md)
- rc.3:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/omitted.md)
- final:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/omitted.md)

**Borderline PRs omitted:** rc.1: 12 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md#borderline),
rc.2: 1 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md#borderline),
rc.3: 0, final: 0

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Valerie Chiang <[email protected]>
Co-authored-by: Pranshu Maheshwari <[email protected]>
Co-authored-by: Pranshu Maheshwari <[email protected]>