Repository navigation
console: add data for the redesigned cluster page's resource usage - #39507
Conversation
fc7f8c9 to
d866e0e
Compare
QA LLM Review1. MEDIUM -- Hydration history query is gated at 26.41 but needs a column added in 26.43
DetailsAt The 26.44 gate for the memory breakdown columns is correct: Fix: raise the constant to |
d866e0e to
423440e
Compare
423440e to
0e48230
Compare
|
Thanks, confirmed: |
|
|
||
| if ( | ||
| isHydratingNow && | ||
| currentRun?.state === "running" && | ||
| currentRun.endMs === windowEndMs && | ||
| currentRun.startMs > windowStartMs && | ||
| !hydrationEpisodes.some( | ||
| (episode) => episode.startedAt.getTime() >= currentRun.startMs, | ||
| ) | ||
| ) { | ||
| segments = paintHydrating(segments, currentRun.startMs, windowEndMs); |
There was a problem hiding this comment.
On the open question: I think the inference can go wrong in two ways. The collector visits one replica per hydration_history_collection_interval. In an environment with N replicas, a finished episode can take about N minutes to be recorded.
- The stretch can be too long. Suppose a replica restarts and finishes hydrating, but the collector hasn't recorded that episode yet. Then someone creates a new index. The model sees an unhydrated object and no recorded episode since the restart. So it paints the whole time since the restart as hydrating. This lasts until the collector visits the replica.
- The stretch can be missing. Suppose the collector has already recorded the restart episode. Then someone creates a new index. The model finds an episode that started after the restart. So it doesn't paint a hydrating stretch at all. The new index's hydration doesn't appear until its own episode is recorded.
Nothing on mz_catalog_server says when an in-progress hydration started. So any stretch drawn for it is a guess. Two options:
- Don't draw a stretch for in-progress hydration. Instead, show the current state at the "now" edge, for example "Hydrating · 3 objects". This is accurate, but less visual.
- Start the inferred stretch at the later of two times: the restart, or the earliest creation time among the replica's unhydrated objects. Then drop the episode check. This fixes both cases, but it's still an approximation.
useHydratingReplicaIdswould need to keep the object ids. They might also need mapping throughmz_object_global_idsto matchmz_object_lifetimes.
There was a problem hiding this comment.
Thanks, this is a clear write-up. I went with option 1: the model no longer infers an in-progress stretch, and the status row shows a "Hydrating · N objects" badge at the now edge, counted per current replica from mz_compute_hydration_times (#39508). Finished episodes still draw from history.
|
|
||
| // TODO: remove these gates once all environments are past them. | ||
| const REPLICA_MEMORY_BREAKDOWN_MIN_VERSION = "26.44.0"; | ||
| const REPLICA_HYDRATION_HISTORY_MIN_VERSION = "26.43.0"; |
There was a problem hiding this comment.
nit: the commit message says hydration episodes need 26.41 and later, but the gate is 26.43 (for process_id), as the description says.
There was a problem hiding this comment.
Fixed the commit message.
6c2b1c5 to
aed7858
Compare
QA LLM Review1. MEDIUM -- Status timeline is blank before the first transition once a replica's older history has aged out
Details
The fix is to fill the stretch before the earliest transition from 2. LOW -- Heap-limit polling stops on a metrics row whose
|
aed7858 to
3df23ed
Compare
QA LLM Review1. MEDIUM -- Creation-times query matches clusters, replicas, roles and schemas that share an index's id
Details
The console's other |
24706e0 to
3763cce
Compare
db7eda6 to
f5713a4
Compare
QA LLM Review1. MEDIUM -- OOM markers miss an OOM kill of any process other than 0 in a multi-process replica
DetailsThe comment "Processes should share the same state" holds for online/offline but not for the reason. Timely processes are shared-fate ( As a result, One fix is to keep process 0 for state and also select OOM rows from every process, e.g. |
7b5c631 to
bddd20b
Compare
a24cda1 to
51fc79d
Compare
QA LLM Review1. MEDIUM -- Status history of system clusters is fetched once and never again
Details
The query has A key that works for every cluster is the latest |
51fc79d to
24a0162
Compare
|
Status history of system clusters is fetched once and never again is addressed in 24a0162, with the caller in 28cc31c. Thanks for checking it on a real system cluster. |
24a0162 to
7b6dbc1
Compare
QA LLM Review1. MEDIUM -- Object hydration stretches are anchored at creation after an upgrade replaces the replica's processes
DetailsReplica services are named per deploy generation ( With the PR's model, transitions One conservative fix is to skip objects created before the environment's boot ( |
…39446) ### Motivation Part of [CNS-173](https://linear.app/materializeinc/issue/CNS-173). First PR in a stack that redesigns the cluster details page. The redesign puts the object lists under one tab, moves cluster configuration into a drawer, and replaces the overview charts with a combined resource usage view. This PR lands only the page shell, so each later PR adds one piece behind the same flag. ### Description - The new page sits behind a new UI preview, `clusterDetailsRedesign`, offered by the LaunchDarkly flag `cluster-details-page-cns-173`. Users who are offered it opt in per browser from the pill in the breadcrumb bar ("Show the new cluster details page" / "Show classic cluster details page"). Self-managed keeps the classic page: it has no preview toggle and treats unlisted flags as on, so `disabledFlexibleDeploymentFlags` pins `cluster-details-page-cns-173` off. - Tabs: - **Metrics** renders the existing `ClusterOverview` until the resource usage PR replaces it. - **Objects** shows the existing materialized view, index, source and sink lists behind a type switcher, at `objects/:objectType`. - **Replicas** is temporary. The configuration drawer PR moves replica management into the drawer and removes this tab. - A title row shows the cluster name with its size and replica count, or "Unmanaged" and "No replicas". It looks the cluster up itself, so a status change on any other cluster in the environment doesn't re-render the page. The configuration PR adds a gear here that opens the drawer. - Links work across both pages. Classic tab URLs (`/indexes` and the rest) redirect into the Objects tab, and `objects/<type>` URLs redirect to the matching classic tab, or to Overview for an unknown type. A bare `objects` link opens the first classic object tab. - `ClusterDetailBreadcrumbs` moves to its own module, and both pages render it from one shared header, which hosts the preview pill. Only the title row, tabs and routes switch under it, so the pill stays mounted and keeps keyboard focus when it switches pages. Next in the stack: the resource usage data (#39507) and charts (#39508), then memory by object, worker skew, and the configuration drawer. The `cluster-details-page-cns-173` flag is on in LaunchDarkly's Staging environment, which PR previews read. Production needs it too before the preview can be offered there. ### Verification `ClusterDetail.test.tsx` adds tests for: - the preview switch in both directions, including opting in from a classic tab URL, with keyboard focus staying on the pill; - the new title and tabs, including an unmanaged cluster with no replicas; - the Objects type switcher; - the redirects in both directions, including an unknown object type; - keeping the current tab when switching clusters from the breadcrumb menu. `useUiPreview.test.tsx` adds a test that self-managed keeps the redesign off. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
7b6dbc1 to
a0313dc
Compare
|
Object hydration stretches are anchored at creation after an upgrade replaces the replica's processes is addressed in 0362388. Thanks for tracing it through the generation filter. The hydration-times request now also reads |
a0313dc to
0362388
Compare
Second in the cluster details redesign stack. Adds the queries, hooks and chart model that the resource usage card in the next PR draws from. Nothing renders them yet. Queries, all scoped to literal replica or object ids so they plan as index lookups on mz_catalog_server, except where noted: - RAM and swap as shares of the heap limit, selected from the console utilization views on 26.44 and later only. Callers that don't ask for them keep their query keys. - Replica status history, for running and offline stretches and out-of-memory kills. - Hydration episodes from mz_replica_hydration_history (26.43 and later, where it gained process_id). This has no index, so it only runs for replicas that came back online inside the window. - How many compute objects each replica has yet to hydrate. - Heap limits per replica, fetched once and refetched only while a replica hasn't reported one. - Creation times of the cluster's indexes and materialized views, fetched once per id set. Per-process reads, for the workers drawer later in the stack: a subscribe builder for a replica's samples in mz_cluster_replica_metrics_history (a lookup on its replica_id index) and each size's process count and allocations from mz_cluster_replica_sizes. The subscribe starts at a literal time, which useRollingWindowStartMs moves forward once per window, since a mz_now() filter would hold every sample's pending retraction on mz_catalog_server. useUtilizationTier holds the tier choice that the replica-level paths share. useReplicaUtilizationHistory gains an includeOfflineEvents option. The new page reads offline stretches from status history, so it skips the 20s offline-events poll on windows of 3h or less. The chart model holds the pure logic: status timelines, replica colors, memory bar segments, and merging buckets so bars stay readable on wide windows.
0362388 to
b7aa112
Compare
…9508) ### Motivation Third PR in the cluster details redesign stack, on top of #39507. It replaces the Metrics tab's overview charts with the resource usage card from the design. CPU, memory and replica status share one time axis and one tooltip, so a spike, an out-of-memory kill and the restart after it line up. Fixes [CNS-174](https://linear.app/materializeinc/issue/CNS-174/show-cluster-memory-swap-metrics) ### Description - **Toolbar:** above the cards, a Replica select ("All replicas", or one replica as `r1 · 100cc`) and the time range picker shared with the classic page. The selection is the `replica` search param, so a narrowed page is a link, and a name no current replica has reads as All. `useSelectedReplica` holds it for the page's other cards. With a replica selected, the card shows that replica and dropped replicas of its name, and the Processes drawer covers only it. - **Header:** under All replicas, a toggle button per current replica (color, name and size), usable from the keyboard. Dropped replicas, such as a blue-green cluster's past deployments, fold into one "+N past replicas" toggle that shows or hides them together, and its tooltip lists them. Labels never wrap, and the header wraps at any count. `useChartLegend`'s toggle now takes any click event, since it only reads the modifier keys. - **CPU:** a line per replica, as a share of the replica's CPU allocation. When a reading passes 100%, the axis extends to fit it and a dashed line marks the allocation. - **Memory:** a bar per replica as a share of the heap limit (RAM plus swap), on a fixed 0 to 100% axis so sizes with a lot of swap don't stretch it. The replicas' bars at one time overlap by half a bar, so each group reads as one point in time. - On 26.44 and later the bar splits into memory and disk (swap), and the legend gives the memory and heap limits in bytes. - Earlier environments show the bar as heap alone, labelled "Heap". - Where no sample in the window has a heap limit, such as on the local emulator, the bars are a share of the memory allocation, and the labels say so. Only rows on 26.44 and later carry that, so older environments keep the heap labels. - **Replica status:** a row per replica, including one offline the whole window. Running, finished hydration and offline stretches, a marker for each out-of-memory kill, and a "Hydrating · N objects" badge at the now edge while a replica has objects left to hydrate. An index or materialized view created in the window adds its hydration on each replica, from #39507's per-object times. A stretch narrower than 4px draws at 4px, so a short hydration after a restart doesn't read as a gap; the tooltip keeps the true times. A replica name wider than the label column ends in an ellipsis, with the full name on hover. - **No readings:** when no sample in the window has a reading, the card says why from replica status: replicas ran but weren't measured, every replica was offline after an out-of-memory kill (with the memory troubleshooting guide), every replica was offline with no recorded reason (with the hydration troubleshooting guide), or status is unknown. Cloud adds a contact-support link; self-managed points at metrics-server. Replica status always shows below. - **Objects Created:** a lane under replica status, with its own heading and a legend chip, and a tick where indexes and materialized views were created. Creations less than 8px apart share a tick, which grows taller with its count. Hovering or focusing a tick lists the time and objects (the first five, then "and N more") and draws one faint guide through the CPU and memory charts, which carry no DDL lines at rest. The objects come from `useClusterObjectsLive`, so DDL elsewhere in the environment doesn't redraw the charts. The cluster page mounts the objects collection bridge, as Object Explorer does. - **Replicas that share a name:** a scheduled cluster gets a new replica id on every refresh, under the same name. Replicas of one cluster that share a name share one toggle, one color, one status row and one tooltip section. A past blue-green deployment's replicas keep their own, since they run alongside the current ones while those hydrate. A time's memory bars split only among the replicas reporting at that time, and bucket merging counts the same. A CPU reading with no neighbor, such as a short refresh merged into one bucket, draws as a dot. - **Processes:** when a replica's size runs more than one process, a Processes button on the CPU chart opens a drawer with a tab per such replica; only the open tab reads. A menu picks what to compare processes by, as on the Freshness page: right now, at any moment (peak, the default), or on average. An average alone hides a process that is climbing. Each tab leads with the busiest process for CPU and for memory by that reading, or says the load is even. A process counts as busiest only when it leads the others' mean by 1.25× and by at least 5 percentage points, so idle processes don't get named over a fraction of a percent. Then one CPU chart and one memory chart draw every process as its own line, and a reading with no neighbor as a dot. A legend on the CPU chart names each drawn line by replica and process; past five processes only the standouts get a color, and the rest share an "Other processes" entry. A table (UniversalTable) lists each process's reading right now, peak and average, computed from its samples, and bolds the column the findings use. Right now is the newest round of samples, which the controller writes for every process at one time, so a process that is down there reads "-" and the drawer says it has no reading. A checkbox per row draws that process's line, the header checkbox draws all or none, and hovering a row highlights its line. CPU reads as a share of one process's CPU allocation and memory as a share of its heap limit, over the card's window up to 24h, and the drawer says so when the card's window is longer. The drawer reads each process's samples from `mz_cluster_replica_metrics_history` (a lookup on its `replica_id` index) through a SUBSCRIBE that is open only while the drawer is, starting at #39507's rolling literal start. - **Tooltip:** per replica, its status and readings at the hovered time, bytes first. CPU adds cores (the share times the size's CPU allocation; sizes without a CPU limit get none). Memory and Disk read against their own limits, the way the Clusters list's columns compute them from the same 3h view (summed over processes, against the size's allocation times its processes), so a recent bucket matches that page apart from the bucket keeping its peak. A Total row gives memory and disk as a share of the heap limit, so an out-of-memory kill reads at the limit. - **Wide windows:** neighboring buckets merge, keeping each metric's peak, so bars stay at least 10px wide. The time axis matches the classic graphs. - **Metrics tab:** the toolbar, the system cluster info box, this card, then the existing resource intensive objects card. The freshness card is off the new page; the classic page keeps it. Loading and deriving live in `useResourceUsageData`. The chart marks are memoized, in the card and in the drawer, so hovering only redraws the cursor and tooltip. The classic page's check for a cluster reporting no metrics moves into `hasNoUtilizationMetrics`, which both pages now use. Its behavior is unchanged. Differences from the classic graphs: the CPU axis extends past 100% instead of clamping, there's no disk chart, replica toggles replace the replica dropdown, and windows of 3h or less don't poll offline events. Not yet checked in Safari or Firefox. ### Verification <img width="1285" height="767" alt="image" src="https://github.com/user-attachments/assets/67fcd0d4-568c-44ab-bd9c-2e5d24b33baa" /> Verified using teleport to customer environments `ResourceUsage.test.tsx` renders the card against MSW handlers and covers: - each replica's CPU, memory and swap readings at a hovered time; - heap alone on environments without the swap columns; - memory labelled as a share of the allocation when no sample has a heap limit; - the CPU axis extending past 100% with the allocation line, and staying at 100% otherwise; - the out-of-memory marker in the tooltip; - the limit labels with their byte sizes; - a status row for a replica offline the whole window, and the status rows in a window with no usage; - the count of objects a replica has left to hydrate; - hiding replicas with the toggles; - the Objects Created lane: its heading and legend, a tick's tooltip and chart guide on hover and on focus, a 7-object tick naming five and "and 2 more", and the empty lane; - four past blue-green deployments folded into "+4 past replicas", listed in its tooltip, shown and hidden together, with no label wrapping; - a long replica name truncated, with its full name as a title; - a 25-minute hydration on a week's window drawn 4px wide, and an object's hydration on a replica that was already running; - tooltip rows against their own limits for a two-process replica, from a fixture computed the way the Clusters list's view computes them, an idle disk reading 0, CPU cores, and no cores for an unlimited size; - on the real Metrics page: the Replica select narrowing the card and writing `?replica=`, the URL selecting a replica, a stale name showing all replicas, and the drawer covering only the selected replica; - the empty window and the load error; - a week of a scheduled cluster's refreshes: one toggle and one status row for `r1`, a line per refresh with more than one point, and bars as wide as their own bucket; - one tooltip section per name while hovering a refresh; a refresh that fits one merged bucket drawn as a dot; a past blue-green deployment's `r1` kept apart from the current one; - a Processes button only for sizes that run more than one process, and a drawer naming the busiest process at any moment by default and on average when picked, noting that it reads the last 24 hours of a 7-day card, listing every process's latest, peak and average, and drawing a line per process that the row and header checkboxes select or clear; - the drawer's legend, which follows the checked processes; a process down in the newest round, which has no reading right now; idle processes read as even however large the ratio between them; and, past five processes, color only for a standout, a neutral highlight for a hovered grey line, a dot for a reading with no neighbor, and a capped list of processes with no reading in the tooltip; - status rows in the same order as the replica toggles; - status history refetched when a replica changes status while the cluster's latest status update stays put, as it always does on system clusters; - no fallback message while replica status is still loading. `processReadingsModel.test.ts` covers each sample as a share of its process's allocation, memory against the heap limit with the allocation fallback, each process's peak per bucket, each process's latest, peak and average from its samples (a process bursting one minute in twenty doesn't outrank a steady one on average over a day), the first bucket clipped to the window, the 5-point standout floor, and naming the busiest process by each of them: a process climbing from 10% to 90% next to a steady 50% is even on average, and busiest right now and at its peak. The drawer test answers its SUBSCRIBE through a new `mockSubscribeSocket` helper, which sends a completed snapshot over the WebSocket. `resourceUsageModel.test.ts` covers merging change ticks by pixel distance and widening short status stretches. `ClusterMetrics.test.tsx` covers the error shown when the clusters subscribe fails before its first snapshot, and the freshness card staying off with its flag on. `ClusterDetail.test.tsx` mocks the new Metrics tab. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Release notes for the **v26.48.0** train. One commit is added per snapshot; the `final` snapshot sets the real dates. **Applied: `rc.1`, `rc.2`, `rc.3`, `final` — the train is complete.** - **Content**: 3 Feature sections (upsert hydration; Kafka sink headers GA; MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS`), 9 Improvements, 2 Guides, 4 Bug Fixes. No **Agent Skills** section — every agent-skills PR in all four windows was omitted. - **Scope**: 123 PRs across the four snapshots — rc.1 114 (14 included), rc.2 5 (0 included), rc.3 1 (1 included), final 3 (0 included). - **Dates — now real, not provisional**: Cloud **2026-10-29**, Self-Managed **2026-10-30**. The `rc.1`-anchored provisional estimate was right, so `final` left both files unchanged. The `v26.48.0` tag's commit date is 2026-10-28T18:35:05Z, a Wednesday; the first Thursday on or after it is 2026-10-29, with Self-Managed the day after. - **Operator compatibility**: row for `v26.48`, release date `2026-10-30`. Unchanged by rc.2, rc.3, and final. - **Branch base**: `main`, whose newest published section was `## v26.46.1` when this branch was cut; v26.47.0 and v26.47.1 have since merged in. `v26.48.0` stays at the top of the list. ### final increment — empty, and verified empty `final`'s commit is empty. The tracking issue ([release#627](MaterializeInc/release#627)) lists exactly one item, the unassociated `4edd60a8d` "release: bump to version v26.48.0", which the release-process-automation rule omits. The three agent-skills PRs in the window ([#118](MaterializeInc/agent-skills#118), [#119](MaterializeInc/agent-skills#119), [#120](MaterializeInc/agent-skills#120)) were all omitted. This was **verified rather than assumed**, because a zero-PR snapshot and a failed fetch produce identical output: `compare/v26.48.0-rc.3...v26.48.0` returns `total_commits: 1`, and that commit is the version bump. Nothing merged onto the release branch in the 42 h 49 m between the `rc.3` tag and the ship tag. Across the train, every agent-skills PR from [#111](MaterializeInc/agent-skills#111) to [#120](MaterializeInc/agent-skills#120) falls in exactly one window, with no gaps or overlaps.⚠️ **One decision is still open for the release owner — see "Upsert v2" below.** Every flag this train held open was re-read in source at the `v26.48.0` ship tag rather than inherited from the RC snapshots: | Flag | Default at `v26.48.0` | Source | Held since | Disposition | | -- | -- | -- | -- | -- | | `enable_upsert_v2` | **`false`** | `src/storage-types/src/dyncfgs.rs:505-509` | rc.2 |⚠️ premise contradicted — see below | | `enable_upsert_chunked_stash` | **`false`** | `src/storage-types/src/dyncfgs.rs:469-475` | rc.2 | same | | `enable_upsert_paged_spill` | **`false`** | `src/storage-types/src/dyncfgs.rs:450-455` | rc.2 | same | | `enable_mcp_protocol_2026_07_28` | **`false`** | `src/adapter-dyncfgs/src/dyncfgs.rs:295-299` | rc.1 | hold stands — still omitted | | `cluster-details-page-cns-173` | n/a (LaunchDarkly) | no source default | rc.1 | hold stands — still omitted | **The rc.2 hold on [#40189](#40189) rests on a premise this branch now contradicts.** rc.2 omitted it at Borderline on exactly one ground — every measurement is taken on a path `enable_upsert_v2` keeps off by default — and asked whether any environment actually runs upsert v2. The branch answers that without anyone feeding the answer back: the human-added **"Improved hydration times and memory efficiency for upsert sources"** section states *"This change has been rolled out to all Materialize Cloud environments and applies to all existing and new clusters."* The three flags still read `false` at the ship tag, which is exactly the source-default-versus-LaunchDarkly divergence the rollout-lever rule covers: the published docs call wins, and the sole stated reason for the omit has fallen away. - **Not already covered.** The published section is about **hydration** — building a source from scratch, 206 min → 108–113 min, 542 GiB → 183–208 GiB. #40189 is about **catch-up after a backlog**, a different operation with its own motivating incident ([CPU-320](https://linear.app/materializeinc/issue/CPU-320): a source that *"could not catch up on any replica size after a producer rate increase"*). A customer who read the hydration section would learn nothing about catch-up. - **Suggested Improvements bullet, to add only if you confirm** — it was *not* applied in the `final` commit: > **Faster catch-up for upsert sources with a large backlog**: An upsert source working through a backlog now drains one timestamp per round instead of merging and re-pushing the whole backlog each round, so catch-up time grows linearly rather than quadratically with the backlog; draining a 100-timestamp tail fell from 6.5 to 0.15 seconds, and a 400-timestamp tail drains in 0.67 seconds. - **Caveat that must travel with the decision**: the baseline in those timings is the *paged* flavor on `main`, not the configuration Cloud ran before this rollout — so they are not a clean before/after for a given customer. Dropping the two timings and keeping the linear-versus-quadratic claim holds regardless of baseline. - **Two weaker links, stated rather than buried**: that the rolled-out configuration is the one #40189 was measured on is *inferred from the published section's prose* (its "buffer pool … serve sequential work from disk" reads as chunked stash plus paged spill), not confirmed by anyone. And the author declared "Release notes: None", naming `enable_upsert_v2` — so publishing this would be an **override** of an author signal, justified by your own later rollout statement. - **Third consecutive train** to hold this flag: `v26.44.0-rc.1` ([#39041](#39041)), `v26.45.0-rc.1` ([#39044](#39044)), `v26.48.0-rc.2` (#40189). If the rollout statement is accurate, the earlier two are worth revisiting rather than left to age out. **The other two holds stand, corroborated.** `enable_mcp_protocol_2026_07_28` still reads `false` at the ship tag and the five MCP PRs rc.1 held appear nowhere in the section; neither does the Console cluster-details page stack. In both cases you had the whole train in view and did not promote them — the confirmation rc.1 asked for, arriving in the negative. **No duplicates, re-checked at ship.** The seven PRs rc.1 flagged as already covered by v26.47.0 are still absent, the `## v26.47.1` section is untouched, and rc.3's entry ([#40065](#40065)) appears exactly once. **Two notes on human edits to this branch**, both recorded as fact rather than objection: - `524adb8` removed the rc.1 feature section **"Native Google Cloud Storage for persist"** ([#39983](#39983)) outright. Flagging it only because a published entry disappearing is easy to miss in a long branch — if that was deliberate, nothing to do. - Two rc.1 Improvements were promoted to their own feature sections with docs links and a code sample (Kafka sink headers [#39908](#39908); MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS` [#39873](#39873)), and a **Guides** section was added that is not in the skill's template. The lesson for next time: an option whose flag flipped to `true` in the same release more likely warrants a feature section than the Improvements bullet the rubric suggests by default. **Carried forward from rc.3 and still open**: the `K8s recovery: envd on failing node` nightly — the job that sank [#40029](#40029), the first attempt at what #40065 re-lands — **was not checked**; it needs Buildkite, outside this run. The entry itself was verified correct in source at both the rc.3 and ship tags, so this is a CI-confidence question, not a notes-accuracy one. ### rc.3 increment One Improvement, for #40065: a read-only `environmentd` that reboots to pick up DDL changes during zero-downtime upgrade catch-up now restarts in place rather than exiting its container, so Kubernetes no longer applies crash-restart backoff. Not inherited from rc.1 — rc.1 correctly omitted the first attempt (#40029) together with its revert (#40061) as a no-op on `main`; rc.3 re-lands the behavior in narrowed form, verified as a real change by comparing `entrypoint.sh` at the `v26.47.0` tag (no retry loop) against `v26.48.0-rc.3` (re-exec on exit 75). The draft's closing "so zero-downtime upgrades catch up faster" was cut: no cited PR measures an upgrade duration, so the entry claims only the mechanism. - **Not a Bug Fix.** 0dt upgrades completed on v26.47.0; the catch-up reboots merely collected backoff delay. Existing behavior improved, so Improvements. - **Residual, deliberately unpublished.** A stale read-only process in a replaced pod still restarts in place on these halts until the kubelet kills its container, competing with its replacement for the generation's replicas. The `MZ_RESTARTED_IN_PLACE` marker check keeps it from booting writable, so it cannot fence the leader — the failure #40061 reverted. Worth watching if 0dt reports look odd after this ships. ### rc.2 increment — empty, by design rc.2's commit is empty: no entry was published and the provisional dates did not move. It exists so later runs can see from the commit subject that rc.2 was applied. Its single materialize PR, #40189, is the open item discussed under the `final` increment above. ### rc.1 — standing warnings⚠️ **Seven PRs in rc.1's tracking issue were already covered by the v26.47.0 train, and none is republished here.** v26.47.0 took five RCs whose later ones were cherry-picks onto the release branch; the same PRs appear in this issue because they are also what landed on `main` for v26.48.0. Four of the seven **shipped to Cloud in v26.47.0 on 2026-10-22** — #37653, #39910, #39927, #40036 — so republishing any of them would put duplicate text in front of customers eight days later. The other three (#39903, #39906, #40046) were held at Borderline in v26.47.0 and are held again here, each reason re-checked rather than inherited. The "snapshots are disjoint by construction" guarantee holds *within* a train, not across trains, so this was caught mechanically by grepping every PR number in the issue against the v26.47.0 snapshot directories. All seven are listed under their own heading in `rc.1/omitted.md`. **Open questions for the release owner** - **Upsert v2** (#40189, rc.2) — see the `final` increment above. The rollout statement on this branch has changed the picture since rc.2; this is the one item that needs a decision before merge. - **MCP protocol revision 2026-07-28** (#39698, #39702, #39703, #39759, #39761) — the stack is complete and `enable_mcp_protocol_2026_07_28` still defaults to **`false`** at the ship tag. Omitted, with the absence of any MCP text on this branch taken as confirmation. Drafted wording remains in `rc.1/omitted.md` if the flag flips. - **Redesigned cluster details page in the Console** (#39446, #39507, #39508, #39510, #39607) — omitted for the same reason. Note that the Improvements section does carry **"New cluster details page on Self-Managed"**, added by hand, which links the v26.47 section rather than re-describing the page. - **Schema projection on the Console's Freshness page** (#40031) — a genuine new way to diagnose lag, but the author states the whole Freshness page "is currently behind a feature flag that is only enabled for certain customers in impersonation mode". **Is the page generally reachable yet?** This is the same page v26.47.0 held for the same reason. - **Time zone data 2026e timing** (#40045) — the author flags the Canadian zone changes as effective **2026-11-01**, three days after the Cloud date. Tight, but it holds. *Resolved since rc.1: the Kafka sink headers framing question — the GA section on this branch answers it.* **Verification notes** — rc.1's Step 5 pass cut one drafted entry and rewrote four: - **Cut**: a Bug Fix drafted for #40032 ("halt with exit code 168 when a live reader loses its lease"). The PR's own Problem section is about alerting — `mz_persist_lease_timeout_read` cannot distinguish a dead reader from a stuck one — and the follow-up it names is a Cloud alerting change. Nothing a customer observes changes. - **tzdata** (#40045): the draft said the Canadian zones move to "permanent daylight saving time", the author's prose. The PR's data table shows a permanent **−05/EST** and **−06/CST** — summer *offsets* kept year-round, but with another zone's standard-time *abbreviations*, which is what a user sees from `AT TIME ZONE`. Reworded to what the table supports. - **Hydration memory** (#39923): scoped to the consolidation step and the measured outcome (~628 GiB peak, no restart, against every pre-change run exiting on the 849 GiB limit). The PR explicitly says the fix "covers this consolidation path, not the sink's whole footprint"; [CPU-317](https://linear.app/materializeinc/issue/CPU-317) tracks the rest. - **Kafka clients** (#40026): "55 to 24" is measured on a 4-worker, 11-source setup. The CPU throttling and degraded freshness are observed symptoms of the *old* behavior at a few hundred sources, not a measured post-change result at that scale, so the entry states the count as the outcome and the throttling as what the count drove. - **`environmentd` accept backlog** (#40035): the author's hedge is kept — "less likely to be restarted", not eliminated, because the PR says "This makes the queue fill later, not never." **Judgment calls a reviewer may want to revisit** — #39605 is filed as an **Improvement**, not a Bug Fix: nothing was incorrect, an unbounded operation gained a bound and a knob. #40101 is filed as a **Bug Fix** rather than omitted under the internal-logging rule, because credentials reaching the logs is a security defect with a user-visible remedy; only credentials a user inlined rather than wrapped in a `SECRET` were affected. Snapshot drafts (with PR links) in mz-skills: - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 12 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md#borderline), rc.2: 1 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md#borderline), rc.3: 0, final: 0 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]>
Motivation
Second PR in the cluster details redesign stack, on top of #39446. It adds the data behind the resource usage card: memory split into RAM and swap, replica status over time, hydration, heap limits and object creation times. The card itself is the next PR (#39508), so this one can be reviewed without the chart code. Nothing renders these queries yet.
Part of CNS-174.
Description
New queries. Each filters on literal replica or object ids, so it plans as an index lookup on
mz_catalog_server, except where noted.swap_of_ram_percentandheap_limit_percenton the console utilization views, 26.44 and later. Windows past 14 days use the ad-hoc query, which now computes both frommz_cluster_replica_metrics_historythe way the views do.mz_cluster_replica_status_history: process 0's rows, which carry a replica's status, plus anyoom-killedrow, since an OOM is recorded only on the process it killedupdated_atamong the replica statuses that the app-wide clusters subscribe already carries, so the query doesn't poll. The cluster's ownlatestStatusUpdatecan't serve, since it comes frommz_cluster_replica_history, which leaves out system clusters.mz_replica_hydration_history, 26.43 and later, where it gainedprocess_id. No index, so it only runs for replicas that came back online inside the window.mz_compute_hydration_timesmz_compute_hydration_timesmz_cluster_replica_metricsmz_object_lifetimes, filtered to those object types since other kinds share theu<N>id formmz_cluster_replica_metrics_history, a lookup on itsreplica_idindexmz_cluster_replica_sizes, fast path on its indexPer-process reads. The resource usage card charts replicas, not processes. This PR adds the per-process subscribe builder and the sizes query that the Processes drawer in #39508 and the workers drawer in #39889 read. The subscribe starts at a literal time one window back, and
useRollingWindowStartMsmoves that start forward once per window, so an open subscribe holds at most two windows of samples. Amz_now()temporal filter would instead hold each sample's pending retraction in the subscribe onmz_catalog_server: on a local emulator with 2 replicas of 16 processes over 24h, that measured ~12 MiB per open subscribe, against ~2 MiB with the literal bound. EXPLAIN onmz_catalog_server: a lookup onmz_cluster_replica_metrics_history_ind, with the bound as a plain filter. Per-process catalog views are tracked in CNS-203.useUtilizationTierholds the tier choice (3h subscribe, 24h subscribe, or poll) that the replica-level paths share.useReplicaUtilizationHistorygets two options:includeMemoryBreakdownadds the RAM and swap columns. Callers that don't pass it keep their query keys, so the classic page sends the same queries as before.includeOfflineEvents: falseskips the 20s offline-events poll on windows of 3h or less. The new page reads offline stretches from status history instead.useClusterObjectsLive(inallObjectsCollection.ts) returns a cluster's objects of the given types as a TanStack DB live query over the existingallObjectsCollection. FilteringuseAllObjectsgives a new array on every DDL anywhere in the environment; the live query keeps its result until one of this cluster's matching objects changes. It reuses the app-wide objects subscribe, so it adds no catalog_server load.resourceUsageModel.tsholds the chart logic as pure functions: status timelines, replica colors, memory bar segments, and merging buckets so bars stay readable on wide windows. Every merged group spans the same interval, clipped to the window, so replicas that start or stop mid-group still line up side by side.The k6 cluster detail replay lists these queries' refresh intervals, since it replays unlisted queries every 5s. The testing guide's scalability commands now use
yarn test:e2e:scalability, because thechromiumproject skipsscalability/.types/materialize.d.tsis edited by hand, with TODOs on the new columns, becauseyarn gen:typesfails against current builds (pg_get_serial_sequencedoesn't exist).A replica's hydration in progress has no recorded start, so the timeline model doesn't draw one. The next PR shows it as a count of objects still hydrating at the now edge instead.
mz_replica_hydration_historyonly records a replica's hydration after it starts, so an object created on a running replica left no stretch.paintObjectHydrationdraws those from the per-object times.time_nscounts from when the replica installed the object's dataflow, so a stretch starts at the later of the object's creation and the replica's last start, and one still hydrating runs to the end of that run. Objects created before environmentd last started (now() - mz_uptime(), read in the same request) are left out: an upgrade, or a restart under the process orchestrator, puts every replica on new processes that rehydrate every object without a status-history row, so their times can't be placed.Verification
replicaUtilizationBinning.test.ts: RAM and swap shares survive binning and come from the bucket's highest-heap sample.allObjectsCollection.test.ts:useClusterObjectsLivereturns the cluster's objects of the given types, keeps its result through changes on other clusters, and updates when one of its objects changes.resourceUsageModel.test.ts: status timelines (state before the window, readings before the first transition once older history is gone, clipping, out-of-memory markers, hydration stretches), replica colors, memory bar segments, the RAM limit, and bucket merging.replicaProcessMetrics.test.ts: the per-process subscribe bounds its samples with a literal start and no temporal filter.replicaProcessMetrics.test.sql.ts, against a real Materialize:mz_cluster_replica_sizesfor the listed sizes.replicaStatusTimeline.test.ts: status transitions come from process 0, and OOM kills from any process.resourceUsageModel.test.tsalso covers groups that start at different buckets, clipping to the window, a merged width capped at the window, and RAM limits that disagree between sizes that swap and sizes that don't.queries.test.tsalso covers: the newest status change across replicas and processes, system replicas included; the status history query refetching only when it moves, and keeping its rows meanwhile; the heap-limit poll stopping and continuing; and the rolling start being set on the first render.resourceUsageModel.test.ts:paintObjectHydrationanchors a stretch at the object's creation or the replica's last start, runs an unfinished one to the end of the run, and paints only running time.queries.test.ts:useObjectHydrationStretchesgroups rows by replica, leaves out objects created before environmentd started, and stays idle without objects; past 14 days, the memory breakdown reaches the ad-hoc query on 26.44 and later only.replicaUtilizationHistory.test.sql.ts: past 14 days, the ad-hoc query returns RAM and swap shares from each bucket's highest-heap sample, and none without the option.replicaStatusTimeline.test.sql.ts: the hydration times query filters to the given replicas and objects, converts to milliseconds and keeps nulls, and reads a real index's hydration; environmentd's start lands before an index created during the test.queries.test.ts:useRollingWindowStartMsstarts one window back, moves once per window, never returns a start for another window, and takes a fresh start when a window comes back.mz_catalog_serverin a local v26.45 build, and the two per-process queries on a cloud environment.🤖 Generated with Claude Code