console: add data for the redesigned cluster page's hydration tab by leedqin · Pull Request #40044 · MaterializeInc/materialize

leedqin · 2026-10-20T23:21:41Z

Motivation

Ninth PR in the cluster details redesign stack (#39509), on top of #40043. The redesign's prototype has a Hydration report tab. This PR adds the data a Hydration tab needs, built on the hydration history the database now records. The next PR in the stack, #40074, adds the tab. Part of CNS-173.

Description

History, read with react-query since the history tables have no index:
- a cluster's hydration episodes from mz_replica_hydration_history, one row per episode with the highest peaks across its processes. Gated on 26.43, which records a row per process, and leaves out older rows. Episodes with no objects stay in: on a replica with nothing to hydrate, such as one running only sources, they're the record that a boot finished;
- the cluster's replicas from mz_cluster_replica_history, so a dropped replica (a resize, a burst replica) keeps its name and size. Fetched again when the cluster's current replicas change, and not kept once unused, so a set of replicas that comes back, as after a burst replica is dropped, fetches again. The same cluster's answer stays up meanwhile, filtered to the window as it moves on;
- an episode's objects from mz_object_hydration_history, with catalog ids through mz_object_global_ids. Fetched each minute until they reach the episode's object count, since the collector can record an episode a visit before some of its objects;
- the highest CPU sampled during an episode, from mz_cluster_replica_metrics_history.
Live state, in SUBSCRIBE-fed TanStack DB collections:
- whether each user object on each of the cluster's replicas is hydrated. A lookup on mz_compute_hydration_times_ind, on the fast path;
- each object's state on one replica (waiting for a hydration slot, working, hydrated), from that replica's mz_compute_hydration_times_per_worker in a session pinned to the replica. Gated on 26.39. It runs on the replica, not on mz_catalog_server, so the tab mounts it only for a replica that is hydrating.
Model (hydrationModel.ts):
- an episode is a full hydration when it starts within two minutes of its replica coming online or being created, or covers every object the cluster had when it finished. The second rule catches a zero-downtime upgrade's rehydration, which runs while the environment is read-only and so records no boot. Objects created during an episode raise its threshold as much as its count, so a burst of new objects isn't full. Any other episode is additive: it covers objects added to an already-hydrated replica. The database records no cause, so this is inferred from status history and the objects' creation times;
- a replica that goes offline before it finishes hydrating, such as an out-of-memory restart loop, is an unfinished hydration. The history only records episodes that finish, so these come from status history. An attempt runs from the replica's previous restart or its creation, not its last online row, since a zero-downtime upgrade writes that row after the replica hydrated. An attempt that began before the window is skipped unless an object was created in the window during it, since the episodes before the window aren't read. End events within a minute of each other are one restart, unless the replica came back online between them. An offline row counts as a restart only after an online one, since a new replica's pod on Kubernetes records an offline row before it first comes online. An object created during an attempt starts a hydration of its own, so the attempt finished only if a hydration finished after the last such object. The creation times come from the cluster's current objects and, while a restart in the window reads as finished with those, also from its indexes and materialized views created in the window and dropped since, read from mz_object_history, so dropping the index behind an out-of-memory restart doesn't hide it. A dropped object counts only for the restarts while it existed, since dropping one unfinished records no finished hydration;
- an episode's objects are split into time waiting for one of the hydration slots and time working.
useObjectCreationTimes keeps its earlier answer up while the objects overlap, as after one is created, so a new index doesn't blank what depends on it.
useClusterHydrationData and useHydrationDetail load and derive what the tab shows. The detail's object timings and CPU sample load and fail on their own, and the timings don't wait on a catalog that failed.
Fix: SubscribeManager starts a clean snapshot when it resends a subscribe the server ended, as when the subscribe's target replica restarts. The old run's rows and error used to stay, so a per-replica subscribe read as failed for as long as it stayed mounted. The held state stays until the new run updates it, so a statement that fails every time keeps its error without flickering. Resends of a statement that keeps failing without a snapshot back off from 1s to 30s, so a replica-pinned subscribe the user lacks USAGE for isn't resent as fast as the round trip. The first resend stays immediate. The delay is the reconnect backoff, now shared as backoffDelayMs.
Fix: collection subscribes now keep their session variables across connection attempts. Each attempt sent the defaults instead, so a cluster passed to a collection subscribe never reached the server. No existing caller passed one. Only variables that are set are passed, so unpinned subscribes keep running on mz_catalog_server.
types/materialize.d.ts is regenerated with yarn gen:types. Most of its diff is the generator catching up with the catalog: builtin materialized views are no longer Generated<>, and relations added since the last run (including the two hydration histories and mz_object_global_ids) appear.

yarn gen:types doesn't run against Materialize as is: kysely 0.28's Postgres introspector calls pg_get_serial_sequence, which Materialize lacks, and lists only tables and views, which leaves out the catalog's builtin materialized views. The types here were generated with both patched locally in node_modules. Fixing the script is a follow-up.

Cost on mz_catalog_server: the episode query and the object query read storage, scoped to one cluster or replica and a time bound, as the redesign's status timeline already does. Episodes poll at the history's 60s collection interval. The replica history scans its index, fetched once per change to the replicas and per mount of the tab. An episode's objects poll each minute only while the open detail is missing some. The dropped objects read scans mz_object_history's index, which holds every object the environment has created: 8.5 ms at 32,000 objects on v26.47, plus about 0.23 ms per 1,000. It runs only while a restart reads as finished, and again only when those restarts change. Looking the window's ids up instead costs about 2 ms plus 60 µs an id and takes several round trips to find them, so it beats the scan only past roughly 70,000 objects.

The Metrics tab's "Hydrating · N objects" badge counts introspection dataflows as well as user objects, so for a moment after a replica boots it can disagree with this PR's per-replica progress.

Tests

hydrationHistory.test.sql.ts: each builder against shadowed relations (grouping and peaks per episode, the 26.43 filter, episodes with no objects, dropped replicas in the window, user objects only, per-worker states combined into one per object, catalog ids for dropped objects), and against the real catalog, including a new index's hydration streamed from a session pinned to the quickstart replica.
hydrationModel.test.ts: full and additive episodes, a zero-downtime upgrade's rehydration counted as full by the objects it covered, a burst of new objects that outnumbers the existing ones, out-of-memory restarts that never finished, a restart merged across processes, restarts after a finished hydration and on drop, a restart while hydrating an object created after the replica finished, a restart after the replica came back online kept apart from the one before it, a crash loop a new object started on a replica up since before the window, the restarts an object missing from the creation times could explain, an object dropped since counted only for the restarts while it existed, an attempt that began before the window, an upgrade's late online row, a new pod's offline row, a boot with nothing to hydrate, the latest full hydration by finish, dropped replicas outside the window, heap limits, and object spans.
useLiveHydration.test.ts: progress counted per replica from the stream on mz_catalog_server, progress complete at once for a cluster with no replicas, per-object states read in a session pinned to the replica, following the replica's new process after its subscribe fails over a restart, and no subscribe before 26.39.
SubscribeManager.test.ts: a subscribe the server ends and runs again drops its error and old rows, one that fails every time keeps reporting its error, and one that fails every time is resent with growing delays.
queries.test.ts: the replica history fetched again when a set of replicas comes back, an episode's objects fetched until all are recorded, including on reopening the episode, and the dropped objects read only once a restart needs them.
objectCreationTimes.test.sql.ts: against the real catalog, the dropped-objects query returns when the cluster's indexes and materialized views created since the start and dropped since were created and dropped, and leaves out ones still there, created before the start, or of another type.
replicaProcessMetrics.test.sql.ts reads its snapshot through the new shared readSubscribeSnapshot helper.

🤖 Generated with Claude Code

linear-code · 2026-10-20T23:21:48Z

CNS-173

def- · 2026-10-21T00:09:14Z

QA LLM Review

1. MEDIUM -- A restart after a hydration that finished before the window is reported as an unfinished hydration

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:229

findUnfinishedHydrations takes the replica's last boot from the full retained status history, but only checks for a finished hydration in hydrations, and the episodes query only returns episodes finished inside the window. So when a long-running replica goes offline inside the window, for example a steady-state OOM, the model reports a hydration that "went offline before it finished" and dates its start to the boot days earlier. With the default 1-hour window this is the usual case for any replica that restarts, not a rare one.

Details

The bounds don't match:

replicaBootTimes (hydrationModel.ts:119) gets boots from createdAt and from every process-0 online row. buildReplicaStatusHistoryQuery has no time bound, and mergeHydrationReplicas gives current replicas their original created_at.
useClusterHydrationEpisodes (queries.ts:1316) fetches finished_at >= now - timePeriodMinutes (hydrationHistory.ts:62).
bootMs is the latest boot before the offline event, which can be before the window. The finished check at hydrationModel.ts:232 then looks for an episode finishing between bootMs and the offline event among only the in-window episodes, finds none, and emits the entry.

I checked this by running the model on: replica created and online at 2026-10-18 10:00, OOM-killed at 2026-10-20 11:30, back online at 11:31 and rehydrated by 11:35, window starting at 11:00. The output is [{startedAt: "2026-10-18T10:00:30Z", endedAt: "2026-10-20T11:30:00Z", reason: "oom-killed"}]. When the 2026-10-18 episode is also passed in, which the window excludes, the output is []. The existing test "skips a restart after the replica finished hydrating" only covers a hydration inside the window.

Two possible fixes: skip a restart whose bootMs is before startMs, which only loses the first attempt of a crash loop that began before the window, or also fetch each replica's latest episode that finished before the window.

def- · 2026-10-21T16:11:13Z

QA LLM Review

1. MEDIUM -- A zero-downtime upgrade's rehydration is classified as a new-objects episode

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:152

On Kubernetes, the full rehydration a 0dt upgrade runs on every replica finishes before status history records any boot for it, so transformHydrations labels it newObjects. Releases make this the most common full hydration in cloud. Each one drops out of latestFullHydration and typicalFullHydration and is shown as a hydration of new objects.

Details

The new generation runs each replica on new -gen-N processes (src/controller/src/clusters.rs:1034). These install and hydrate every dataflow while environmentd is read-only, and the episode's started_at is their earliest installed_at.
A read-only environmentd writes no status-history rows (src/adapter/src/coord/message_handler.rs:1077). The first online row for those processes is written when the promoted generation reboots as leader and its orchestrator watch reports the already-running pods ready. That row is stamped with the reboot time, since the pod Ready condition's last_probe_time is normally unset (src/orchestrator-kubernetes/src/lib.rs:1401).
Promotion waits until clusters have stayed hydrated for with_0dt_caught_up_check_stability_period. That is 10 minutes by default, and the flag has no LaunchDarkly override (KNOWN_MISSING_FROM_LD). So the online row always comes more than the hydration time plus 10 minutes after the episode started, far outside BOOT_WINDOW_MS.

I ran the model on that timeline: the gen-N processes boot at 10:00, the episode runs from 10:00:05 to 10:06, and the leader's online row lands at 10:16:30. The output is kind: "newObjects", and latestFullHydration returns undefined. The promotion row also becomes the replica's latest boot. So a later OOM at 10:40 is reported as an unfinished hydration that started at 10:16:30, although the replica had been hydrated since 10:06.

Boot detection needs a signal that also sees read-only boots. now() - mz_uptime(), which the base stack already uses for the resource usage timeline, doesn't help here: it marks the leader reboot, the same moment as the online row.

def- · 2026-10-21T22:12:50Z

QA LLM Review

1. MEDIUM -- Every restart of a replica with no indexes or materialized views is reported as an unfinished hydration

console/src/api/materialize/cluster/hydrationHistory.ts:61

The episodes query drops zero-object episodes, but findUnfinishedHydrations (hydrationModel.ts:232) treats a restart as finished only if some episode finished between the replica's boot and the restart. So on a cluster without indexes or materialized views, such as a source or sink cluster, or a cluster used only for ad hoc queries, every OOM kill or other offline event in the window is listed as a hydration that never finished, however long the replica had been up.

Details

The collector does record an episode after each boot of such a replica. Since 26.43, object_count counts only non-introspection exports, so that episode has object_count 0 (test/testdrive/hydration-status.td:69 expects hydrated 0 for a replica with no objects). .where("object_count", ">", "0") removes exactly those rows, so hydrations never holds an entry for the replica.

I ran the model on a source-only replica that came online at 2026-10-21 14:01 and was OOM-killed at 2026-10-22 03:00. The output is an unfinished hydration from 14:01 to 03:00 with reason oom-killed. When the zero-object episode recorded at 14:01 is passed in, the output is empty.

This is separate from the earlier window-boundary finding. The boot here is inside the window, and any online row in the 7-day default window (an earlier restart, an environmentd restart, a release) puts it there. In #40074 these entries become unfinished rows in the history table and chart. The verdict is unaffected, since live progress counts zero objects and reads as hydrated.

One option is to keep zero-object episodes for the unfinished check and filter them only from the hydrations that are displayed.

leedqin · 2026-10-21T22:40:52Z

Thanks, all three are addressed in 9a2a02e:

Window boundary: an attempt now runs from the replica's previous restart or its creation, and one that began before the window is skipped rather than reported.
Zero-downtime upgrades: an episode that covers every object the cluster had when it began counts as full, and online rows no longer bound an attempt, so the promotion's late online row doesn't produce a false restart. Objects dropped since aren't counted, so after a drop a partial hydration can read as full. That's noted in the code.
Zero-object episodes: the episodes query keeps them, and they count as a finished boot. The tab still leaves them out of the chart and table.

Each has a regression test in hydrationModel.test.ts, and the SQL test now expects the zero-object episode.

def- · 2026-10-22T00:15:08Z

QA LLM Review

1. MEDIUM -- On Kubernetes, every new replica reports an unfinished hydration at its creation

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:210

findUnfinishedHydrations treats every process-0 offline row as a restart. On Kubernetes, a new replica writes an offline row with no reason as soon as its pod is created, before it has ever come online. So every replica created in the window (a new cluster, a resize, a burst replica) gets an unfinished hydration that runs from its creation to a fraction of a second later. In #40074, a replica doing its first hydration then reads as restarting, and the tab shows an error verdict.

Details

How the row gets written:

The coordinator starts each process of a new replica at Offline(Some(Initializing)) (src/adapter/src/coord/ddl.rs:639, src/adapter/src/coord.rs:1723).
The pod watch passes the pod's ADDED event through (touched_objects). A pod that was just created has no Ready condition, so the event maps to Offline(None), stamped with the current time (src/orchestrator-kubernetes/src/lib.rs:1389).
Offline(None) is not equal to Offline(Some(Initializing)), so the cluster-event handler writes an offline row with a NULL reason (src/adapter/src/coord/message_handler.rs:1059, :1077). The process orchestrator reports online straight away, which is why local runs don't show the row. I traced this through the code and haven't seen it on a live cluster.

I ran the model on this case: replica created at 10:00:00.000, offline/NULL at 10:00:00.300, online at 10:01:30, an episode from 10:01:32 to 10:06:00, and a window starting at 09:00. The output is [{startedAt: 10:00:00.000, endedAt: 10:00:00.300, reason: null}], both while the replica is hydrating and after it finishes. Without the pending-pod row the output is [].

In #40074, summarizeReplicasNow counts that entry as a restart because nothing has finished yet. So during the first hydration, the verdict says the replica "hasn't finished hydrating since 10:00" and "restarted once". Once hydration finishes, the history still keeps a "Restarted before finishing" row. None of the fixtures in hydrationModel.test.ts has an offline row before the first online.

One fix: count an offline row as a restart only when the replica's previous process-0 transition was online.

leedqin · 2026-10-22T14:59:44Z

Thanks, fixed in 44073ca. An offline row now counts as a restart only when the replica's previous process-0 transition was online, so the row a new pod records before it first comes online no longer reads as an unfinished hydration. There's a regression test in hydrationModel.test.ts.

This push also makes live progress complete at once for a cluster with no replicas, for the finding on #40074.

def- · 2026-10-22T16:28:54Z

QA LLM Review

1. MEDIUM -- New objects created together are classified as a full hydration once they outnumber the cluster's existing objects

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:164

coversAll compares the episode's object count with the cluster's object count when the episode started. Objects created after the episode started add to the episode's count but not to that threshold. So an episode that hydrates only new objects counts as full whenever it covers more objects than the cluster already had, even though it never touched the existing ones. That partial episode then becomes the replica's latestFullHydration and enters typicalFullHydration.

Details

At its start the cluster holds its N existing objects plus the first new one, which is created just before it is installed (useClusterHydrationData.ts:122). So the threshold is N+1, and any burst of K ≥ N+1 new objects whose hydrations overlap, and so form one episode, is labelled full.

I ran the model on this case: a replica up since 08:00 with one index, then two new objects created at 10:00:00 and 10:00:01 that hydrate together by 10:03. The output is kind: "full", and latestFullHydration and the typical duration are both that 3-minute episode. Without objectCountAt the same episode is newObjects. With 5 existing objects and 6 new ones, the result is also full.

In #40074 that duration feeds "Its last full hydration took …" while a replica rehydrates after a restart, and also the hydrated verdict and the per-size typical value. Both then understate how long a full rehydration takes. The NOTE about dropped objects doesn't cover this case, since nothing was dropped.

Comparing against the objects the cluster had when the episode finished (objectCountAt(finishedMs)) would make objects created during the episode raise the threshold by as much as they raise the count.

def- · 2026-10-29T20:54:08Z

QA LLM Review

1. MEDIUM -- The replica-pinned live subscribe is resent in a tight loop for viewers without USAGE on the cluster

console/src/platform/clusters/ClusterPage/Hydration/useLiveHydration.ts:144

Pinning useLiveObjectHydrations to a replica runs its SUBSCRIBE on the user's cluster, so it fails on every run for a viewer whose role lacks USAGE on that cluster. SubscribeManager resends a subscribe on every ReadyForQuery with no backoff, and the server sends one after every error. Such a viewer therefore sends failing SUBSCRIBEs as fast as the round trip allows for as long as the replica hydrates. In #40074, LiveObjects renders nothing on error, so nobody sees it happen.

Details

With cluster_replica set, the statement isn't routed to mz_catalog_server (src/adapter/src/coord/catalog_serving.rs:176), and a SUBSCRIBE needs USAGE on its target cluster (src/sql/src/rbac.rs:982). The other collection subscribes run on mz_catalog_server, where PUBLIC has USAGE. A replica with INTROSPECTION INTERVAL = 0 fails the same way on every run, with cannot read log sources of replica with disabled introspection (src/adapter/src/coord/sequencer.rs:998).
The WebSocket API follows every request with ReadyForQuery, failed or not (src/environmentd/src/http/sql.rs:446). onReadyForQuery then resets and resends (console/src/api/materialize/SubscribeManager.ts:315). Each round makes the coordinator plan a SUBSCRIBE, and the client calls captureException.
I ran SubscribeManager against mockSubscribeSocket with every run ending in endWithError, which is how the server answers a denied SUBSCRIBE. It sent 500 SUBSCRIBEs on one connection in 168 ms. In a browser the round trip sets the pace, so a single viewer sends tens per second. That lasts for the whole hydration, or, in console: add a hydration tab to the redesigned cluster page #40074's restarting state, for the whole crash loop.
The new test "keeps reporting the error of a subscribe that fails every time it runs" ends the second run with error, which sends no ReadyForQuery. The test stops after two runs, but the server would keep the loop going.

One fix is to resend only after an error that a retry can clear, such as the target replica failing, and to leave other errors in place until the session restarts. Backing off between resends would also bound the loop.

def- · 2026-11-02T20:15:54Z

QA LLM Review

1. MEDIUM -- Merging nearby restarts can erase a failed rehydration

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:249

findUnfinishedHydrations merges every restart within one minute of the previous one, even when the replica came online between them. If a steady-state OOM is followed by another OOM during rehydration, this can discard the failed hydration entirely, hiding the start of a crash loop.

Details

For example, a replica created at 09:59 finishes hydrating at 10:04, is OOM-killed at 10:05:00, comes online at 10:05:10, and is OOM-killed again at 10:05:40 before rehydrating. With all events inside the window, the model returns []: the second restart is merged into the first, and the first is then excluded because the 10:04 hydration finished before it. The unfinished attempt from 10:05:10 to 10:05:40 disappears.

The one-minute proximity check should not merge events across an intervening online transition. Preserving that boundary makes this reproduction report the failed attempt while retaining the existing test that merges multiple processes' reports of one restart.

2. MEDIUM -- An early detail fetch stays cached after object history catches up

console/src/platform/clusters/queries.ts:1414

useEpisodeObjectHydrations marks its first result fresh forever, but a finished replica episode can become visible before all its object-history rows exist. Opening its details during that gap leaves the object list empty or incomplete while the detail stays mounted, even after collection catches up.

Details

The backend runs object collection and replica collection as separate mutations, in that order (src/adapter/src/coord/hydration_history.rs:610, :622). An object can finish between their reads, and replica collection also proceeds when object collection fails. The replica episode can therefore be recorded one sweep before its remaining object rows.

The detail query's key contains only the replica and episode timestamps, so later history polling does not refresh it. A hook probe returned an empty first response, made the completed object available, then remounted the same episode: it still returned zero rows after only one request. Allow recently completed episode details to refetch while their object history catches up.

First of two PRs adding a Hydration tab to the redesigned cluster page, on top of the configuration drawer. This one adds the data the tab reads. The next one adds the tab. - History, read with react-query since the history tables have no index: a cluster's hydration episodes from mz_replica_hydration_history (26.43 and later, one row per episode with the highest peaks across its processes), the cluster's replicas from mz_cluster_replica_history so dropped ones keep their name and size, an episode's objects from mz_object_hydration_history, and the highest CPU sampled during an episode. - Live state, in SUBSCRIBE-fed TanStack DB collections: whether each user object on each replica is hydrated (a lookup on mz_compute_hydration_times_ind), and each object's queued, working or hydrated state on one replica, from that replica's own introspection (26.39 and later) in a session pinned to it. - A model that counts an episode as a full hydration when it starts within two minutes of its replica booting, finds replicas that went offline before they finished hydrating, such as out-of-memory restarts, and splits an episode's objects into time waiting for a hydration slot and time working. - Collection subscribes keep their session variables across connection attempts. Each attempt sent the defaults instead, so a cluster given to a collection subscribe never reached the server. No existing caller passed one. - types/materialize.d.ts regenerated with yarn gen:types.

- An end after a replica came back online is a restart of its own, so a failed rehydration shortly after a restart is no longer merged away. - An attempt that began before the window counts once an object was created in the window during it, dated from that object. - An episode's object hydrations are fetched each minute until they reach the episode's object count, since the collector can record an episode a visit before some of its objects. - The cluster's replica history isn't kept once unused, so a set of replicas that comes back, as after a burst replica is dropped, fetches again. - SubscribeManager backs off resending a statement that keeps failing without a snapshot, such as a replica-pinned subscribe the user lacks USAGE for. The first resend stays immediate. The delay reuses the reconnect backoff, now shared as `backoffDelayMs`. - The creation times' placeholder checks overlap with a set.

leedqin · 2026-11-03T04:04:13Z

Thanks, these are fixed in 79e79a4:

Nearby restarts: an end after the replica came back online is now a restart of its own, so the 10:05:10 to 10:05:40 attempt is reported. There's a regression test with your timeline.
Detail fetched before its objects are recorded: an episode's object hydrations are fetched each minute until they reach its object count, and reopening an incomplete one fetches again.
Replica history cache: the history isn't kept once unused, so a set of replicas that comes back fetches again. There's a test with the burst sequence.
OOM on a replica up since before the window: an attempt that began before the window now counts once an object was created in the window during it, dated from that object. There's a test with your crash loop. Restarts caused by an object dropped since are still missed, since the creation times come from the cluster's current objects.
Resend loop: resends of a statement that keeps failing without a snapshot now back off from 1s to 30s, using the reconnect backoff. The first resend stays immediate, so a replica restart still recovers at once. There's a test with a subscribe denied on every run.

A restart is unfinished when the replica was hydrating an object created during the attempt. The creation times came only from the cluster's current objects, so once the object that caused an out-of-memory restart was dropped, the restart read as finished. The tab now also reads the cluster's indexes and materialized views created over the window and dropped since, from `mz_object_history`. That relation is indexed only by id, so the read scans every object the environment has created: 8.5 ms at 32,000 objects on v26.47, plus about 0.23 ms per 1,000. It runs only while a restart in the window reads as finished with the current objects, and again only when those restarts change. `findRestarts` is split out of `findUnfinishedHydrations` so the hook can tell which restarts read as finished.

leedqin · 2026-11-03T16:49:30Z

Follow-up on item 4 above: restarts caused by an object dropped since are now covered in 5deb050. While a restart in the window reads as finished, the tab also reads the cluster's indexes and materialized views created in the window and dropped since, from mz_object_history. That's a scan of its index, 8.5 ms at 32,000 objects on v26.47, so it waits for such a restart and doesn't run otherwise.

def- · 2026-11-03T17:10:39Z

QA LLM Review

1. MEDIUM -- Objects dropped before a restart produce false hydration failures

console/src/platform/clusters/ClusterPage/Hydration/useClusterHydrationData.ts:225

The new dropped-object input treats an object canceled before a restart as still hydrating at that restart. After a user drops an unfinished index or materialized view, a later steady-state OOM can therefore appear as a failed hydration even though the replica had returned to being fully hydrated.

Details

For example, a replica finishes hydrating at 10:02, starts a new object at 10:10, drops it unfinished at 10:11, and suffers an unrelated OOM at 10:20. The model returns an unfinished hydration from 10:10 to 10:20. Without the newly added dropped-object timestamps, it correctly returns none.

buildDroppedObjectCreationTimesQuery filters on dropped_at but does not return it (console/src/api/materialize/cluster/objectCreationTimes.ts:70). Dropping an export retracts its hydration log row (src/compute/src/logging/compute.rs:957), so cancellation produces no finished episode to satisfy finishedBetween. A local backend probe confirmed that dropping an unhydrated MV restores the replica's hydrated status while leaving the latest episode unchanged, and the new query returns that MV's creation time.

Return the drop time and consider each dropped object's creation only for restarts that occurred while that object still existed.

Dropping an object that hadn't finished hydrating records no finished hydration, so a later restart read as cutting it short. The dropped-object read now returns when each object was dropped, and an object counts toward a restart only if it still existed then.

leedqin · 2026-11-03T19:29:23Z

Thanks, fixed in 66b9083. The dropped-object read now returns when each object was dropped, and an object counts toward a restart only if it still existed then. There's a model test with your timeline.

DROP INDEX now fails while a newer index or a materialized view on the same table reads from the index (#39098). The test dropped its oldest index first, so it failed on builds with that change. Drop the materialized view first and the indexes newest first, and put the kept index on its own table so it doesn't read from the dropped ones.

### Motivation Tenth PR in the cluster details redesign stack (#39509), on top of #40044, which adds the data. The redesign's prototype has a Hydration report tab. This adds a Hydration tab built on what the database records about hydration, and says which parts the Console infers. Part of [CNS-173](https://linear.app/materializeinc/issue/CNS-173) and [CNS-209](https://linear.app/materializeinc/issue/CNS-209). There's no Figma design; the layout follows the prototype and the PM's feedback on it. ### Description - **Tab:** Hydration, after Objects. Its link carries the `?replica=` choice like the Metrics and Workers tabs, and the classic page redirects a Hydration link to its overview. - **Controls:** the replica select shared with the other tabs, and a time range of 24 hours, 7 days (the default) or 30 days. Restarts come from the replica status history, which is kept 30 days. - **Alert:** only when something needs attention. A replica that keeps restarting with no finished hydration recorded, with when it began (date and time), how many times, and a link to the docs on hydration failures. A replica that's offline, whatever its last progress said, or one that hasn't reported yet on a cluster with objects. A hydrating replica gets no alert, since the progress card shows it. A cluster without indexes or materialized views says it has nothing to hydrate. - **Hydration in Progress:** shown while a replica isn't hydrated, with elapsed times that tick every second, dated from its boot or, when every object it hasn't hydrated is newer than the boot, from the first one's creation. A replica that hasn't reported reads as starting only for two minutes after it boots. How many objects it has hydrated so far, and "Working on `<object>` for `<time>`" for each object it works on, read from the replica itself and mounted only while it hydrates. A replica offline between restarts isn't hydrating: it gets no timer, progress bar, in-progress mark or live subscribe. - **History chart:** a dot per hydration by replica, filled for a full hydration and hollow for an additive one, with restarts that never finished in a lane below. Restarts close together are one mark, counted in its tooltip, so a restart loop stays readable. The y axis is hydration time, ticked at round durations, and the x axis shows dates, or times of day over 24 hours. Replicas of different names are drawn a dot's width apart, at most 48px in all, and replicas of one name, such as the ones a schedule or a resize replaced, share a place, a color and a legend entry. Each mark is a button that selects its hydration. - **Detail:** the last successful hydration by default, or the one selected. Its duration, peak memory, disk and heap, each as a share of its limit with the bytes (heap turns red near the limit), and the highest CPU sample during it. Below, a waterfall of each object's working time, slowest first, or of its time waiting for a slot and working in start order, 30 objects a page. For a restart, why there is nothing to break down. - **Past Hydrations:** every hydration, timed by when it began or, for a restart, when the replica restarted, with its peak memory as a bar like the cluster list's, and its peak disk and heap; each row's time selects it. - **Replica select:** a retired replica, such as a burst replica, stays selectable while its history is in the window. - **Labels:** tooltips say what each figure is. The Console's inferences (full or additive, restarts before finishing) say "Inferred". The collector visits one replica per interval and can miss a hydration that finished just before a restart, so a restart says only that no finished hydration was recorded. Without hydration history (before 26.43) no restarts are inferred. - **Loading:** each section renders as soon as its own data arrives and shows its own error. The history doesn't wait for the live progress, and the alert waits for both, since telling a restart before finishing from a crash after hydrating needs the history. The history's restarts come from the current replicas' status, so it waits for that and fails with it, and keeps its last answer while a resize or burst changes the replica set. Creation times are read for every object, not only the window's, and the elapsed time waits for a new object's. The history waits for the cluster's objects and their creation times, which tell a full hydration from an additive one, and falls back to replica starts alone when the catalog can't be read. A new replica or index keeps the history up while its data loads, and the detail holds its height while another hydration's object timings load. Shared pieces: the stack's `ReplicaSelect`, which can now mark a replica retired, a `HYDRATION_TAB_PATH` in `tabLinks.ts`, `percentOf` exported from `processReadingsModel.ts`, `replicaStatus` from the configuration drawer, each replica's size allocation from `useClusterHydrationData`, `PercentBar` from the cluster list, a `keepOverlapping` option on `useReplicaStatusHistory`, an optional range on `useObjectCreationTimes`, and `PaginationControls`, split out of `TablePagination` so the waterfall pages the same way tables do. This is larger than the stack's other PRs, about 2,400 lines of components and 1,200 of tests, because it adds the whole tab at once. ### Verification Not yet checked in a real browser. Worth a look: dark mode, chart spacing with many hydrations and a restart loop, the waterfall's pages on a cluster with many objects, the red heap stat, and a replica that is hydrating, for the live object list. `ClusterHydration.test.tsx` renders the tab against MSW handlers and the subscribe socket mock and covers: - the last successful hydration shown by default with no alert, and selecting another from the table; - a hydration's objects ordered slowest first by default, without waiting, or by start with it, and paged; - Past Hydrations with each hydration's peak memory and no kind filter; - each replica named once in the legend, and a dated time axis; - object timings kept when the CPU sample fails to load; - a replica restarting before it finishes, with the restart count, the docs link, and its restarts merged into one mark; - a replica offline between restarts kept off the clock; - a restart counted once the index it was hydrating is dropped, from the dropped objects #40044 reads; - a hydrating replica's working objects, with no alert and no Hydrating label; - an offline replica called out rather than its last progress; - a cluster without indexes or materialized views; - a retired replica offered while its history is in the window, with no alert about the current replicas; - the history kept up while a new replica's or object's data loads; - elapsed times that move on, and restarts that went offline labelled as such; - a tooltip opening on keyboard focus; - the history showing without waiting for live progress; - narrowing to the replica chosen on the cluster page; - live progress kept when the history fails to load; - the history waiting for the catalog before it classifies hydrations and counts new objects, and falling back to replica starts, with object timings by id, when the catalog fails to load; - live progress only, and no inferred restarts, on environments older than 26.43; - the history's error when the replicas' status fails to load; - a new object's elapsed time left out until its creation time loads; - a new object's hydration dated from its creation rather than the replica's boot; - a cluster with no replicas. `HydrationHistoryChart.test.tsx` covers a replica replaced every hour for a week, whose marks all land in the plot in time order, and replicas of different names hydrating at once, set apart within a bounded spread. `hydrationView.test.ts` covers ordering finished and unfinished hydrations, restarts counted only since a replica last finished, when the current hydration began, an offline replica between restarts, offline and starting replicas, and each alert, including none once every replica is hydrated or hydrating. `ClusterDetail.test.tsx` covers the Hydration tab in the tab list, opening it from its link, and the classic page's redirect. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- <a href="https://app.devin.ai/review/materializeinc/materialize/pull/40074" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a>

### Motivation `yarn gen:types` fails with `function "pg_get_serial_sequence" does not exist`. kysely-codegen runs kysely's Postgres introspector for its table query. Since the bump to kysely 0.28 (#35669), that query calls `pg_get_serial_sequence`, which Materialize lacks. kysely 0.29 still does. The query also lists only relkinds `r`, `v` and `p`, so the catalog's builtin materialized views (`m`) never reached the types under kysely-codegen 0.17. #40044's types were generated with both patched by hand in `node_modules`. This also bumps kysely to 0.29 and kysely-codegen to 0.20, superseding #39686. ### Description - `bin/kysely-codegen.ts` runs kysely-codegen programmatically with an introspector that overrides only the table query. kysely-codegen's materialized view query, enum handling and serializer still apply. - The script writes the copyright header to `types/materialize.d.ts`. The codegen output lacks it, and the copyright lint fails without it. - `bin/generate-types` exits on errors, pulls the requested image (a cached `latest` went stale), and stops the container on exit. Before, each run left a container bound to port 6875, and the next run generated types from that old container. - kysely 0.28.17 → 0.29.8 and kysely-codegen 0.17 → 0.20.0: - 0.29 rejects `limit(sql.raw(...))`. `rawLimit` and the lag history limit now call `query.limit(sql.lit(n))`, which still inlines the limit as a literal, as Materialize requires. - `rawLimit` used to append its limit with `modifyEnd`. In 0.29, compiling end modifiers calls `Array.prototype.toSorted`, which Chrome 109 lacks. No console query uses end modifiers now. Neither `rawLimit` caller has an OFFSET or another end modifier, so the compiled SQL is unchanged. - kysely 0.29 is ESM-only and kysely-codegen 0.20 `require`s it, so `yarn gen:types` needs Node 22.12 or later. CI and the Volta pin run Node 26, and #39683 updates `engines.node`. - `types/materialize.d.ts` is regenerated against v26.49.1. kysely-codegen 0.17 and 0.20 produce identical output. - Adds `mz_builtin_columns` and `mz_builtin_index_columns`. - Builtin columns lose their `Generated<>` wrappers, because #40316 changed `pg_attribute.atthasdef` to report no default for them. The console only selects from these relations, where `Generated<T>` and `T` read the same. - Drops the two hand-added `TODO` markers from #39992. The generator now emits `mz_cluster_auto_scaling_strategies` and `mz_cluster_reconfigurations`, and `AUTO_SCALING_MIN_VERSION` still gates them at runtime. The file is now exactly the generator output. Worth checking on the preview, since these run the changed limits: - the Query History list; - a sink's Errors tab; - the freshness lag charts on the environment overview and the cluster pages. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

The final snapshot adds no new entries: the v26.50.0-rc.1...v26.50.0 compare holds exactly one commit, the version bump, and no agent-skills PR merged in the window, so the increment is empty. Dates are now real rather than provisional (Cloud 2026-11-12, Self-Managed 2026-11-13); rc.1 had estimated the same pair, so no date text changes. The whole-release pass found three entries that main already publishes in its v26.49.1 section, because the v26.49.1 patch train cherry-picked four Console PRs that also merged to main inside this train's rc.1 window. They are removed here so they are not published twice: - "Diagnose hydration from the cluster details page" (#40044, #40074), which main carries as the feature section "Right size your clusters, using the new Hydration tab on the console", with a screenshot. - "The Freshness page on Self-Managed" (#40511), which main carries as "Freshness and cluster details pages on Self-Managed", covering both pages. - "Swap usage in the Console's Disk column" (#40492), which main carries as a Bug Fix. That leaves v26.50.0 with 5 Improvements and 7 Bug Fixes. Draft with PR links: https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/release_notes.md Co-Authored-By: Claude Opus 5 <[email protected]>

Dates in RC snapshots are provisional until the release ships. One commit is added per snapshot; the final snapshot sets the real dates. **The `final` snapshot has been applied: the dates below are the real ones** (Cloud 2026-11-12, Self-Managed 2026-11-13). The `final` snapshot adds no entries — `v26.50.0-rc.1...v26.50.0` is one commit, the version bump, and no agent-skills PR merged in the window. Its substantive work was the dates, a rebase onto `main`, and the three removals below. **Rebased onto `main`.** The branch was 107 commits behind and conflicted: `main`'s `## v26.49.1` section replaced the stale `## v26.49.0` stub the branch was cut from. Resolved by keeping `main`'s section verbatim and re-inserting `## v26.50.0` above it; same for the compatibility YAML (`v26.50`, then `v26.49.1`, then `v26.48.2`). **Three entries removed as already published on `main`.** The `v26.49.1` patch train cherry-picked four Console PRs that also merged to `main` inside this train's `rc.1` window, so both trains classified them independently and `main` already carries them under `## v26.49.1`: | PR(s) | Published on `main` in `## v26.49.1` as | Removed from `## v26.50.0` | |---|---|---| | [#40044](#40044), [#40074](#40074) | the feature section *"Right size your clusters, using the new Hydration tab on the console"*, with a screenshot | *"**Diagnose hydration from the cluster details page**"* | | [#40511](#40511) | *"**Freshness and cluster details pages on Self-Managed**"*, covering both pages | *"**The Freshness page on Self-Managed**"* | | [#40492](#40492) | the Bug Fix about the Console's clusters list showing a dash in the Disk column | *"**Swap usage in the Console's Disk column**"* | `main`'s wording is the later and fuller treatment in all three cases. That leaves `## v26.50.0` with **5 Improvements and 7 Bug Fixes**. Decisions required: - **MCP protocol default flip** ([#39762](#39762)): published as an Improvement because `enable_mcp_protocol_2026_07_28` is `true` at the tag and the flip *is* the change. The PR was held in draft "until the 2026-07-28 code ships in a release and passes a test on a deployed environment" — is it confirmed on? - **Columnar accumulators** ([#40364](#40364)): omitted as Borderline — the default is `true` but the PR states no measured user-facing effect. Is there a memory or CPU figure worth publishing? - **MySQL snapshot/replication concurrency** ([#40206](#40206), [#40207](#40207)): omitted as Borderline — gated on `storage_persist_sink_description_lookahead`, which is `Duration::ZERO` at the tag, same as the PostgreSQL counterparts held back in v26.47.0. If the flag is served on, both halves want announcing together. - **MCP servers GA** ([#40448](#40448)): omitted as Borderline — the preview banners are gone, but the GA call belongs to the release owner. Publish it? - **Balancer connection metrics** ([#40192](#40192)): omitted as Borderline — `mz_balancer_connection_status{status="error"}` changes meaning, so Self-Managed dashboards and alerts on that series will see a step change. Worth a note? - **Console query comments** ([#40075](#40075)): omitted as Borderline — every Console statement's `sql_hash` changes once when this deploys. Worth calling out? Every flag above was re-read in source at the `v26.50.0` ship tag; not one default moved between `rc.1` and ship, so all six holds still read correctly. Snapshot drafts (with PR links) in [mz-skills](https://github.com/MaterializeInc/mz-skills): - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 7 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md#borderline), final: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md#borderline) (no PRs in scope; `rc.1`'s six holds re-examined at the ship tag and unchanged) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]> Co-authored-by: Valerie Chiang <[email protected]>

leedqin requested a review from a team as a code owner October 20, 2026 23:21

leedqin requested a review from Alphadelta14 October 20, 2026 23:21

leedqin changed the base branch from cluster-page-configuration to cluster-page-autoscaling-form October 20, 2026 23:21

leedqin added this pull request to stack #39509 October 20, 2026 23:21

leedqin force-pushed the cluster-page-hydration-data branch from 1b6ea8c to 709a135 Compare October 20, 2026 23:26

leedqin force-pushed the cluster-page-hydration-data branch from 709a135 to 809b437 Compare October 21, 2026 15:17

leedqin force-pushed the cluster-page-hydration-data branch from 809b437 to c6aeab0 Compare October 21, 2026 15:23

leedqin force-pushed the cluster-page-hydration-data branch from c6aeab0 to 2114ccc Compare October 21, 2026 21:25

leedqin mentioned this pull request Oct 21, 2026

console: add a hydration tab to the redesigned cluster page #40074

Merged

leedqin force-pushed the cluster-page-hydration-data branch from 2114ccc to 9c941b8 Compare October 21, 2026 22:07

leedqin force-pushed the cluster-page-hydration-data branch from 9c941b8 to 9a2a02e Compare October 21, 2026 22:39

leedqin force-pushed the cluster-page-hydration-data branch from 9a2a02e to 31e12c7 Compare October 21, 2026 23:19

leedqin requested review from jdonelson and removed request for Alphadelta14 October 22, 2026 14:36

leedqin force-pushed the cluster-page-hydration-data branch from 31e12c7 to 44073ca Compare October 22, 2026 14:59

leedqin force-pushed the cluster-page-hydration-data branch from 44073ca to 5a32288 Compare October 22, 2026 15:48

leedqin force-pushed the cluster-page-hydration-data branch from a3d347c to 0021de5 Compare October 29, 2026 19:00

leedqin force-pushed the cluster-page-hydration-data branch from 0021de5 to 5e618dc Compare October 29, 2026 19:35

leedqin force-pushed the cluster-page-hydration-data branch from 5e618dc to ad6886e Compare October 29, 2026 20:22

leedqin force-pushed the cluster-page-hydration-data branch 2 times, most recently from bcf56e9 to c5e2aac Compare November 2, 2026 19:29

leedqin force-pushed the cluster-page-hydration-data branch 2 times, most recently from 0f92770 to f55f514 Compare November 3, 2026 00:12

Base automatically changed from cluster-page-autoscaling-form to main November 3, 2026 00:37

leedqin force-pushed the cluster-page-hydration-data branch from f55f514 to b1f1ab3 Compare November 3, 2026 00:37

leedqin merged commit 616f9c3 into main Nov 4, 2026
82 checks passed

leedqin deleted the cluster-page-hydration-data branch November 4, 2026 19:14

leedqin mentioned this pull request Nov 6, 2026

console: bump kysely to 0.29 and fix yarn gen:types #40550

Merged

bosconi mentioned this pull request Nov 11, 2026

v26.50.0 release notes #40532

Merged

Conversation

leedqin commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Tests

Uh oh!

linear-code Bot commented Oct 20, 2026

Uh oh!

def- commented Oct 21, 2026

QA LLM Review

1. MEDIUM -- A restart after a hydration that finished before the window is reported as an unfinished hydration

Uh oh!

def- commented Oct 21, 2026

QA LLM Review

1. MEDIUM -- A zero-downtime upgrade's rehydration is classified as a new-objects episode

Uh oh!

def- commented Oct 21, 2026

QA LLM Review

1. MEDIUM -- Every restart of a replica with no indexes or materialized views is reported as an unfinished hydration

Uh oh!

leedqin commented Oct 21, 2026

Uh oh!

def- commented Oct 22, 2026

QA LLM Review

1. MEDIUM -- On Kubernetes, every new replica reports an unfinished hydration at its creation

Uh oh!

leedqin commented Oct 22, 2026

Uh oh!

def- commented Oct 22, 2026

QA LLM Review

1. MEDIUM -- New objects created together are classified as a full hydration once they outnumber the cluster's existing objects

Uh oh!

def- commented Oct 29, 2026

QA LLM Review

1. MEDIUM -- The replica-pinned live subscribe is resent in a tight loop for viewers without USAGE on the cluster

Uh oh!

def- commented Nov 2, 2026

QA LLM Review

1. MEDIUM -- Merging nearby restarts can erase a failed rehydration

2. MEDIUM -- An early detail fetch stays cached after object history catches up

Uh oh!

leedqin commented Nov 3, 2026

Uh oh!

leedqin commented Nov 3, 2026

Uh oh!

def- commented Nov 3, 2026

QA LLM Review

1. MEDIUM -- Objects dropped before a restart produce false hydration failures

Uh oh!

leedqin commented Nov 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leedqin commented Oct 20, 2026 •

edited

Loading