storage: make the chunked upsert v2 stash time-major by DAlperin · Pull Request #40189 · MaterializeInc/materialize

DAlperin · 2026-10-26T14:36:42Z

The upsert v2 operator drains one source timestamp per persist round trip, but its stash was a single key-sorted batcher sealed at the input frontier. When a source falls behind, every round merged and extracted the whole backlog, processed the one timestamp at the persist frontier, and pushed everything else back through a resident Vec outside the buffer pool budget. Per-round cost grew with the backlog, so catch-up was quadratic in it.

In production this showed up as a paged-flavor source that could not catch up on any replica size after a producer rate increase. CPU went to pushing ineligible data back (re-paging and re-compressing it) and to sealing. Doubling workers doubled CPU without raising the drain rate. See CPU-320.

#40190, stacked on this one, splits the module into submodules as a pure move.

Changes

Seal just past the persist frontier (seal_upper: the start of the next mz_repr::Timestamp, capped at the input frontier), in both flavors, so a round extracts only what it can process.
Make the chunked stash time-major. It stores ((time, key), time, diff), so a seal cuts a prefix of the batcher's chains. ColumnChunk's merge passes chunks with disjoint data ranges through, and its extract passes chunks wholly on one side of the upper through by their time bounds, so a seal rewrites only the boundary chunk. Within one time, entries stay sorted by key, which the drain's monotone probes rely on.
The paged flavor keeps its key-major batcher. Its seal (extract_chain) reads every chunk whatever the upper, so it stays quadratic, but the narrower seal halves its catch-up time.
UpsertStashArm gains a per-arm chunker and flush.
Count pushed-back updates. Adds mz_storage_upsert_restashed_total, the number of updates the v2 operator extracted before the output reached their time and pushed back. On a source stuck the way CPU-320 describes, it climbs every round.

An earlier revision of this PR bucketed the stash by time with a BucketChain of key-major batchers instead. Review pointed out that time-major order removes the need for it. The time-major version needs no changes to mz_timely_util::temporal, is about 200 lines smaller, and drains faster when the stash spills.

Performance

Optimized build, draining a tail of N timestamps × 10k updates over 1M keys, one timestamp per persist advance. Drain seconds, paged / chunked:

	100 ts	200 ts	400 ts
`main`	6.5 / 4.0	26.9 / 16.9
`seal_upper` alone, key-major	3.3 / 3.6	12.8 / 14.2
`BucketChain` (earlier revision)	0.50 / 0.47	1.11 / 1.19	2.67 / 2.63
this PR, chunked stash resident	n/a / 0.48	n/a / 1.48	n/a / 5.35
this PR, chunked stash spilled	n/a / 0.15	n/a / 0.32	n/a / 0.67

seal_upper alone stays quadratic. With the chunked stash spilling to the buffer pool (enable_upsert_paged_spill), the time-major stash drains linearly and about 4x faster than the bucketed one, and writes a quarter as many chunks to the pool (647 against 2636 at 400 timestamps). With spilling off, chunks stay resident and recompute their time bounds by scanning on every seal, so the drain grows faster than linear. Caching the time bounds on resident ColumnChunks would remove that, as a follow-up in mz-timely-util.

Tests

tail_catch_up_restashes_nothing: drains a 20-timestamp tail one persist advance at a time and asserts nothing is pushed back. On main the same scenario pushes back 1710 updates.
later_subtime_is_pushed_back_and_drained: an update at a later subtime of the persist frontier's timestamp is pushed back exactly once, then drained against the earlier value.
persist_jump_into_tail: a persist frontier that jumps into a stashed tail drops the times it passed, drains the one it lands on, and keeps the rest without extracting it.
spilled_tail_drains_across_split_chunks: with spilling forced, a 12-timestamp tail with more than a chunk per timestamp drains correctly, and the pool sees at most 5 inserts per timestamp (34 in practice). A seal that rewrote every chunk past its upper would add about 100.
refined_input_frontier_drains_finalized_subtimes, far_ahead_data_drains, source_tail_ahead_of_persist, source_tail_across_rounds: draining multi-timestamp tails.
open_time_across_rounds_consolidates, time_ahead_of_frontier_across_rounds_consolidates: two updates for one key at one time, arriving in separate operator rounds, consolidate to the later one.
paged_stash_matches_model, chunked_stash_matches_model: each arm's batcher against a model through random pushes, seals and push-backs. The chunked check also asserts sealed output is time-major.
seal_upper_stops_past_persist_upper: the seal upper computation, including the maximum timestamp.

All v2 operator tests run against both stash flavors and assert equal output.

Release notes

None. Upsert v2 is gated behind enable_upsert_v2.

Generated with Claude Code

devin-ai-integration

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

def- · 2026-10-26T17:09:15Z

QA LLM Review

1. MEDIUM -- Paged stash re-copies each tick's data about 11 times in steady state

src/storage/src/upsert_continual_feedback_v2.rs:1099

For a caught-up source, each tick's data lands about one tick ahead of the chain start, so peel and restore split its bucket about 11 times before it drains, and on the paged flavor every StashBucket::split is a full ColumnMergeBatcher::seal that loads and copies every record in the bucket. This makes the upsert v2 operator roughly 20-45% slower than on main for a paged source in steady state, while the chunked flavor is unaffected.

Details

After a tick at t drains, the chain starts at t+1, and the next tick's data at t+1000 sits in a bucket about 2^10 wide. When persist reaches it, peel splits that bucket down to one timestamp. For ColumnMergeBatcher, each seal runs extract_chain over a FetchIter of the whole merged chain, copying each record into keep/ship buffers. The shipped half comes back resident via pager.take and is re-paged and re-merged in a fresh batcher. The doc comment's "spilled bodies move without loading" holds only for the chunked flavor, whose ColumnChunk::extract passes whole chunks through by their time bounds.

Measured on a single-worker harness with epoch-ms timestamps and 1000 ms ticks: 120 ticks of 3000 upserts over 20k keys, with persist fed back from the operator's output. It saw 1329 non-empty splits, about 11 per tick. With --profile optimized, Paged with persist caught up took 613 ms on main and 738-880 ms on this PR (three runs). With persist one tick behind it was 628 ms vs 748-926 ms. In a debug build it was 8.05 s vs 11.8-12.0 s. Chunked took 570-605 ms on both. The harness ran with the stash pager disabled. With spill enabled, each split also routes every chunk through the pager.

Suggested fix: when a bucket's data lies entirely on one side of the split time, have split hand over the whole bucket without sealing. Return (empty, self) when frontier is at or past timestamp. Return (self, empty) when a tracked upper bound on the bucket's times is below timestamp. Maintain that bound in Stash::push, and set it to the split time on a lower half created by a real split. A prototype of this brought Paged back to 592-603 ms with zero non-empty splits when caught up. A randomized differential test against a reference upsert model still passed: 6000 seeds, both flavors, including restarts and refined subtimes.

The v2 upsert drain processes one timestamp per persist round trip, but the stash was a single key-sorted batcher sealed at the input frontier. While catching up on a source tail spanning many timestamps, every round merged and extracted the whole tail and re-pushed everything above the persist frontier through a resident Vec outside the buffer pool budget, making catch-up quadratic in the tail. Seal just past the persist frontier, and keep the stash as a BucketChain of batchers, the time bucketing compute's temporal filters use. A round peels and seals only the buckets below the seal upper, one at a time. The chain keeps buckets near its start small and splits farther ones, fueled, as the frontier approaches. A bucket whose times all fall on one side of a split point splits without a seal. Its buckets partition time, which the drain requires: each bucket consolidates on its own, so one (key, time) in two buckets would emit both values. Bucket boundaries fall on whole mz_repr::Timestamps, via a BucketTimestamp impl for product timestamps that advances the outer coordinate, and peel keeps a one-step bucket the frontier falls inside instead of splitting it. Updates pushed back below the chain's start go to a front batcher. Locally (debug build), draining a 100-timestamp tail of 400k updates took 40.7 s (paged) and 16.4 s (chunked) before, and 3.7 s and 3.4 s now. Draining 400k updates at one timestamp takes 0.9 s and 0.4 s; with every split sealing, it took 34.7 s and 1.6 s.

…ches up Add mz_storage_upsert_restashed_total, the number of updates the v2 operator extracts from its stash at a time the output has not reached and pushes back. When the drain extracts more than the timestamp it can process, this grows with the backlog every persist round trip, which is otherwise visible only in a CPU profile. tail_catch_up_restashes_nothing drains a 20-timestamp tail one persist advance at a time and asserts nothing is pushed back. Before the stash was bucketed and sealed just past the persist frontier, the same scenario pushed back 1710 updates.

antiguru

I think we can merge this, but should follow up with a change to reduce some of the wrinkles we're introducing here. This could share some code with MV sinks, but it's not practical to call it due to the code living in a dependent crate, and it would need slightly different generic bounds.

My agent has some comments, also inline, I'm repeating the comments verbatim:

Overall: right diagnosis, wrong layer for the fix.

What's good:

The diagnosis is correct. Sealing at input_upper and pushing the backlog back every round is quadratic during catch-up, and the restashed metric makes that visible.
seal_upper (just past persist_upper, capped at the input frontier) is the real fix. It's small and could stand alone.
The tests are good: both flavors run with restore fuel 1, the restash count is asserted, and the subtime cases are covered.
I found no correctness bug. The partition invariant holds, and the straddling bucket is handled.

The problem: the PR fixes a key-major sort order by adding structure on top of it.

BucketChain over key-major batchers needs a lot of supporting machinery: the pair BucketTimestamp impl, the peel contract change, front, max_time for one-sided splits, the straddling iterator, and fueled restore. All of it exists to cut key-major data along time.
Even so, any split or seal inside a bucket rewrites the whole bucket, and the paged split materializes the lower half resident.
The pair impl changes the meaning of a shared utility (the advance contract and peel's guarantee) to serve one caller, and it conflicts with #39928's approach to the same problem.

What I'd push for:

Make the stash time-major, ((T, key), T, diff). Most of the machinery then becomes unnecessary or cheap.
Then decide whether to bucket at all. Correction V2 keeps a BucketChain over time-major chains to keep merges within a time range. A single time-major batcher works only if its merges and extracts pass through time-disjoint chunks without loading them. The paged extract_chain doesn't do that today.
If bucketing stays, project to mz_repr::Timestamp the way BucketTime does, and keep the subtime straddle in Stash. Leave BucketChain and BucketTimestamp unchanged.
Look at reusing correction V2's chain types (made generic over the semigroup diff) instead of maintaining a second bucket-over-batcher implementation.

On the numbers: the speedup is real, but it was measured on a debug build with a 100-timestamp tail. The baseline is quadratic, so most of the 10x comes from seal_upper, not from the bucket design. It would be worth benchmarking seal_upper alone on the old single batcher. If that captures most of the gain, the simpler change could merge first and the stash redesign could follow time-major.

My verdict is request changes on the design, not on correctness. My most useful single ask: split seal_upper and the metric into their own PR.

antiguru · 2026-10-26T18:47:31Z

+            // `seal_upper`, just past persist_upper) to extract the
+            // source-finalized entries that can be processed now as sorted,
+            // consolidated chunks. A bucket's seal merges all its chains
+            // (O(N) linear merge of sorted data) and splits by time: entries


Nit: It's not a k-way merge, which would be linear time, but potentially the whole cascade, so closer to $$O(n\cdot\log n)$$.

antiguru · 2026-10-26T18:58:17Z

+/// minimum, so all boundaries have a minimal inner coordinate and a bucket
+/// holds every refinement of the outer timestamps it spans. A frontier inside
+/// an outer timestamp therefore lands in a bucket [`BucketChain::peel`] keeps.
+impl<TOuter: BucketTimestamp, TInner: Timestamp> BucketTimestamp for (TOuter, TInner) {


Suggested change

impl<TOuter: BucketTimestamp, TInner: Timestamp> BucketTimestamp for (TOuter, TInner) {

impl<TOuter: BucketTimestamp, TInner: Timestamp+TotalOrder> BucketTimestamp for (TOuter, TInner) {

This is only well-defined if the inner timestamp has a total order.

antiguru · 2026-10-26T18:58:39Z

    fn advance_by_power_of_two(&self, exponent: u32) -> Option<Self>;
 }

+/// A product timestamp buckets by its outer coordinate: advancing moves the


Suggested change

/// A product timestamp buckets by its outer coordinate: advancing moves the

/// A lexicographic pair timestamp buckets by its outer coordinate: advancing moves the

antiguru · 2026-10-26T19:00:16Z

+    const DOMAIN: usize = TOuter::DOMAIN;
+
+    fn advance_by_power_of_two(&self, exponent: u32) -> Option<Self> {
+        Some((self.0.advance_by_power_of_two(exponent)?, TInner::minimum()))


This break the trait's invariants when self.1 isn't the minimum, it doesn't return $$\text{self}+2^e$$.

antiguru · 2026-10-26T19:30:04Z

            if upper.is_none() && !frontier.is_empty()
                || upper.is_some() && frontier.less_than(&upper.unwrap())
            {
+                if bits == 0 {


This is the same root cause as the pair impl's advance_by_power_of_two: the step is one outer timestamp, so a zero-bit bucket holds many times and a frontier can land strictly inside it. That weakens peel's contract for every user ("returned buckets contain all data strictly less than the frontier" now depends on the timestamp type), and callers have to know to seal the kept bucket themselves.

Alternative, as in #39928's BucketTime: keep the chain over mz_repr::Timestamp, route (o, i) to o in Stash::push, peel at the frontier's outer projection, and seal the bucket for o at the refined seal_upper in Stash. That drops both the pair impl and this branch, and keeps BucketChain unchanged. It would also avoid two competing ways to bucket refined times if #39928 lands.

antiguru · 2026-10-26T19:30:08Z

+/// times from its start, and the front holds the times below it. Each bucket
+/// consolidates and drains on its own, so two updates for one `(key, time)` in
+/// different buckets would both be emitted against the same prior value.
+struct Stash<T, B>


Each bucket's batcher still sorts (UpsertKey, T, UpsertDiff), so every chunk spans the bucket's whole time range. Any seal or split that falls inside a bucket loads and rewrites all of its chunks: the chunked extract time-bounds pass-through effectively never fires, and the paged extract_chain has none. peel also splits straddling buckets regardless of fuel, so the first rounds after a backlog rewrite the wide bucket about log(span) times.

Storing ((T, UpsertKey), T, diff) time-major, as correction V2 and #39760 do, makes a split or seal touch only the boundary chunk. The drain still works: eligible entries are all at persist_upper, so they stay key-sorted for the monotone seek_key, and equal (key, time) pairs stay adjacent for consolidation.

antiguru · 2026-10-26T19:30:10Z

+        }
+        let (frontier, max_time) = (self.frontier.clone(), self.max_time.clone());
+        let mut lower = empty;
+        for chunk in self.seal(Antichain::from_elem(timestamp.clone())) {


For the paged flavor, seal runs pager.take on every ship-side chunk, so the whole lower half of the bucket is resident at once, outside the pager budget. That's the same class of problem this PR removes for the ineligible Vec. Each chunk then goes through page, take, and page again, and is pushed as a singleton chain, so a sorted chain gets re-merged geometrically. A push_chain on ColumnMergeBatcher that inserts the kept-paged chain as-is would avoid all three.

DAlperin · 2026-10-26T20:30:32Z

Posted by Claude (Claude Code) on Dov's behalf.

Thanks for the review. I took the time-major route in this PR instead of following up later. The chunked stash now stores ((time, key), time, diff) in a single ChunkBatcher and seals at seal_upper, so the BucketChain, the pair BucketTimestamp impl, and the peel change are all gone. mz_timely_util::temporal is untouched.

On your question about seal_upper alone: on an optimized build it halves paged's catch-up but stays quadratic in both flavors (3.3 → 12.8 s for 100 → 200 timestamps). Time-major with the stash spilling drains linearly: 0.15 / 0.32 / 0.67 s for 100 / 200 / 400 timestamps, about 4x faster than the bucketed version. With spilling off, resident chunks recompute their time bounds on every seal, so it grows faster than linear there. I'd fix that by caching bounds on resident ColumnChunks in a separate PR. Numbers are in the description. Paged keeps its key-major batcher, since extract_chain has no time pass-through.

Could you take another look?

antiguru

Looks solid! Thank you

Replace the bucketed stash with a single time-major chunked batcher: the chunked flavor stores ((time, key), time, diff), so a seal just past the persist frontier cuts a prefix of the chains and only the boundary chunk is rewritten. Both flavors now seal at seal_upper. The shared-utility changes (the pair BucketTimestamp impl and the peel change) are reverted.

Stacked on #40189. `upsert_continual_feedback_v2.rs` had grown to 2.8k lines. This moves its code into submodules along the seams it already had: * `feedback`: the persist-feedback arrangement types, `encode_feedback`, and the `UpsertValue` to `Row` codec. * `stash`: `UpsertDiff`, the stash batcher types, `seal_upper`, and the per-flavor `UpsertStashArm` impls. * `drain`: classifying sealed stash entries against the persist frontier and emitting eligible output, one function per flavor. * `tests`: the unit tests. The parent keeps the module docs, `UpsertStashFlavor`, `upsert_inner`, and the operator loop. This is a pure move. Apart from rustfmt reflows, the only differences are module headers, explicit imports per file, `pub(super)` on the moved items, and two intra-doc links that now need paths. The crate-visible surface is unchanged: `upsert_value_to_row` and the fuzzing-gated `datum_seq_to_upsert_value` are re-exported from the parent. ### Release notes None. Generated with [Claude Code](https://claude.com/claude-code)

Release notes for the **v26.48.0** train. One commit is added per snapshot; the `final` snapshot sets the real dates. **Applied: `rc.1`, `rc.2`, `rc.3`, `final` — the train is complete.** - **Content**: 3 Feature sections (upsert hydration; Kafka sink headers GA; MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS`), 9 Improvements, 2 Guides, 4 Bug Fixes. No **Agent Skills** section — every agent-skills PR in all four windows was omitted. - **Scope**: 123 PRs across the four snapshots — rc.1 114 (14 included), rc.2 5 (0 included), rc.3 1 (1 included), final 3 (0 included). - **Dates — now real, not provisional**: Cloud **2026-10-29**, Self-Managed **2026-10-30**. The `rc.1`-anchored provisional estimate was right, so `final` left both files unchanged. The `v26.48.0` tag's commit date is 2026-10-28T18:35:05Z, a Wednesday; the first Thursday on or after it is 2026-10-29, with Self-Managed the day after. - **Operator compatibility**: row for `v26.48`, release date `2026-10-30`. Unchanged by rc.2, rc.3, and final. - **Branch base**: `main`, whose newest published section was `## v26.46.1` when this branch was cut; v26.47.0 and v26.47.1 have since merged in. `v26.48.0` stays at the top of the list. ### final increment — empty, and verified empty `final`'s commit is empty. The tracking issue ([release#627](MaterializeInc/release#627)) lists exactly one item, the unassociated `4edd60a8d` "release: bump to version v26.48.0", which the release-process-automation rule omits. The three agent-skills PRs in the window ([#118](MaterializeInc/agent-skills#118), [#119](MaterializeInc/agent-skills#119), [#120](MaterializeInc/agent-skills#120)) were all omitted. This was **verified rather than assumed**, because a zero-PR snapshot and a failed fetch produce identical output: `compare/v26.48.0-rc.3...v26.48.0` returns `total_commits: 1`, and that commit is the version bump. Nothing merged onto the release branch in the 42 h 49 m between the `rc.3` tag and the ship tag. Across the train, every agent-skills PR from [#111](MaterializeInc/agent-skills#111) to [#120](MaterializeInc/agent-skills#120) falls in exactly one window, with no gaps or overlaps. ⚠️ **One decision is still open for the release owner — see "Upsert v2" below.** Every flag this train held open was re-read in source at the `v26.48.0` ship tag rather than inherited from the RC snapshots: | Flag | Default at `v26.48.0` | Source | Held since | Disposition | | -- | -- | -- | -- | -- | | `enable_upsert_v2` | **`false`** | `src/storage-types/src/dyncfgs.rs:505-509` | rc.2 | ⚠️ premise contradicted — see below | | `enable_upsert_chunked_stash` | **`false`** | `src/storage-types/src/dyncfgs.rs:469-475` | rc.2 | same | | `enable_upsert_paged_spill` | **`false`** | `src/storage-types/src/dyncfgs.rs:450-455` | rc.2 | same | | `enable_mcp_protocol_2026_07_28` | **`false`** | `src/adapter-dyncfgs/src/dyncfgs.rs:295-299` | rc.1 | hold stands — still omitted | | `cluster-details-page-cns-173` | n/a (LaunchDarkly) | no source default | rc.1 | hold stands — still omitted | **The rc.2 hold on [#40189](#40189) rests on a premise this branch now contradicts.** rc.2 omitted it at Borderline on exactly one ground — every measurement is taken on a path `enable_upsert_v2` keeps off by default — and asked whether any environment actually runs upsert v2. The branch answers that without anyone feeding the answer back: the human-added **"Improved hydration times and memory efficiency for upsert sources"** section states *"This change has been rolled out to all Materialize Cloud environments and applies to all existing and new clusters."* The three flags still read `false` at the ship tag, which is exactly the source-default-versus-LaunchDarkly divergence the rollout-lever rule covers: the published docs call wins, and the sole stated reason for the omit has fallen away. - **Not already covered.** The published section is about **hydration** — building a source from scratch, 206 min → 108–113 min, 542 GiB → 183–208 GiB. #40189 is about **catch-up after a backlog**, a different operation with its own motivating incident ([CPU-320](https://linear.app/materializeinc/issue/CPU-320): a source that *"could not catch up on any replica size after a producer rate increase"*). A customer who read the hydration section would learn nothing about catch-up. - **Suggested Improvements bullet, to add only if you confirm** — it was *not* applied in the `final` commit: > **Faster catch-up for upsert sources with a large backlog**: An upsert source working through a backlog now drains one timestamp per round instead of merging and re-pushing the whole backlog each round, so catch-up time grows linearly rather than quadratically with the backlog; draining a 100-timestamp tail fell from 6.5 to 0.15 seconds, and a 400-timestamp tail drains in 0.67 seconds. - **Caveat that must travel with the decision**: the baseline in those timings is the *paged* flavor on `main`, not the configuration Cloud ran before this rollout — so they are not a clean before/after for a given customer. Dropping the two timings and keeping the linear-versus-quadratic claim holds regardless of baseline. - **Two weaker links, stated rather than buried**: that the rolled-out configuration is the one #40189 was measured on is *inferred from the published section's prose* (its "buffer pool … serve sequential work from disk" reads as chunked stash plus paged spill), not confirmed by anyone. And the author declared "Release notes: None", naming `enable_upsert_v2` — so publishing this would be an **override** of an author signal, justified by your own later rollout statement. - **Third consecutive train** to hold this flag: `v26.44.0-rc.1` ([#39041](#39041)), `v26.45.0-rc.1` ([#39044](#39044)), `v26.48.0-rc.2` (#40189). If the rollout statement is accurate, the earlier two are worth revisiting rather than left to age out. **The other two holds stand, corroborated.** `enable_mcp_protocol_2026_07_28` still reads `false` at the ship tag and the five MCP PRs rc.1 held appear nowhere in the section; neither does the Console cluster-details page stack. In both cases you had the whole train in view and did not promote them — the confirmation rc.1 asked for, arriving in the negative. **No duplicates, re-checked at ship.** The seven PRs rc.1 flagged as already covered by v26.47.0 are still absent, the `## v26.47.1` section is untouched, and rc.3's entry ([#40065](#40065)) appears exactly once. **Two notes on human edits to this branch**, both recorded as fact rather than objection: - `524adb8` removed the rc.1 feature section **"Native Google Cloud Storage for persist"** ([#39983](#39983)) outright. Flagging it only because a published entry disappearing is easy to miss in a long branch — if that was deliberate, nothing to do. - Two rc.1 Improvements were promoted to their own feature sections with docs links and a code sample (Kafka sink headers [#39908](#39908); MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS` [#39873](#39873)), and a **Guides** section was added that is not in the skill's template. The lesson for next time: an option whose flag flipped to `true` in the same release more likely warrants a feature section than the Improvements bullet the rubric suggests by default. **Carried forward from rc.3 and still open**: the `K8s recovery: envd on failing node` nightly — the job that sank [#40029](#40029), the first attempt at what #40065 re-lands — **was not checked**; it needs Buildkite, outside this run. The entry itself was verified correct in source at both the rc.3 and ship tags, so this is a CI-confidence question, not a notes-accuracy one. ### rc.3 increment One Improvement, for #40065: a read-only `environmentd` that reboots to pick up DDL changes during zero-downtime upgrade catch-up now restarts in place rather than exiting its container, so Kubernetes no longer applies crash-restart backoff. Not inherited from rc.1 — rc.1 correctly omitted the first attempt (#40029) together with its revert (#40061) as a no-op on `main`; rc.3 re-lands the behavior in narrowed form, verified as a real change by comparing `entrypoint.sh` at the `v26.47.0` tag (no retry loop) against `v26.48.0-rc.3` (re-exec on exit 75). The draft's closing "so zero-downtime upgrades catch up faster" was cut: no cited PR measures an upgrade duration, so the entry claims only the mechanism. - **Not a Bug Fix.** 0dt upgrades completed on v26.47.0; the catch-up reboots merely collected backoff delay. Existing behavior improved, so Improvements. - **Residual, deliberately unpublished.** A stale read-only process in a replaced pod still restarts in place on these halts until the kubelet kills its container, competing with its replacement for the generation's replicas. The `MZ_RESTARTED_IN_PLACE` marker check keeps it from booting writable, so it cannot fence the leader — the failure #40061 reverted. Worth watching if 0dt reports look odd after this ships. ### rc.2 increment — empty, by design rc.2's commit is empty: no entry was published and the provisional dates did not move. It exists so later runs can see from the commit subject that rc.2 was applied. Its single materialize PR, #40189, is the open item discussed under the `final` increment above. ### rc.1 — standing warnings ⚠️ **Seven PRs in rc.1's tracking issue were already covered by the v26.47.0 train, and none is republished here.** v26.47.0 took five RCs whose later ones were cherry-picks onto the release branch; the same PRs appear in this issue because they are also what landed on `main` for v26.48.0. Four of the seven **shipped to Cloud in v26.47.0 on 2026-10-22** — #37653, #39910, #39927, #40036 — so republishing any of them would put duplicate text in front of customers eight days later. The other three (#39903, #39906, #40046) were held at Borderline in v26.47.0 and are held again here, each reason re-checked rather than inherited. The "snapshots are disjoint by construction" guarantee holds *within* a train, not across trains, so this was caught mechanically by grepping every PR number in the issue against the v26.47.0 snapshot directories. All seven are listed under their own heading in `rc.1/omitted.md`. **Open questions for the release owner** - **Upsert v2** (#40189, rc.2) — see the `final` increment above. The rollout statement on this branch has changed the picture since rc.2; this is the one item that needs a decision before merge. - **MCP protocol revision 2026-07-28** (#39698, #39702, #39703, #39759, #39761) — the stack is complete and `enable_mcp_protocol_2026_07_28` still defaults to **`false`** at the ship tag. Omitted, with the absence of any MCP text on this branch taken as confirmation. Drafted wording remains in `rc.1/omitted.md` if the flag flips. - **Redesigned cluster details page in the Console** (#39446, #39507, #39508, #39510, #39607) — omitted for the same reason. Note that the Improvements section does carry **"New cluster details page on Self-Managed"**, added by hand, which links the v26.47 section rather than re-describing the page. - **Schema projection on the Console's Freshness page** (#40031) — a genuine new way to diagnose lag, but the author states the whole Freshness page "is currently behind a feature flag that is only enabled for certain customers in impersonation mode". **Is the page generally reachable yet?** This is the same page v26.47.0 held for the same reason. - **Time zone data 2026e timing** (#40045) — the author flags the Canadian zone changes as effective **2026-11-01**, three days after the Cloud date. Tight, but it holds. *Resolved since rc.1: the Kafka sink headers framing question — the GA section on this branch answers it.* **Verification notes** — rc.1's Step 5 pass cut one drafted entry and rewrote four: - **Cut**: a Bug Fix drafted for #40032 ("halt with exit code 168 when a live reader loses its lease"). The PR's own Problem section is about alerting — `mz_persist_lease_timeout_read` cannot distinguish a dead reader from a stuck one — and the follow-up it names is a Cloud alerting change. Nothing a customer observes changes. - **tzdata** (#40045): the draft said the Canadian zones move to "permanent daylight saving time", the author's prose. The PR's data table shows a permanent **−05/EST** and **−06/CST** — summer *offsets* kept year-round, but with another zone's standard-time *abbreviations*, which is what a user sees from `AT TIME ZONE`. Reworded to what the table supports. - **Hydration memory** (#39923): scoped to the consolidation step and the measured outcome (~628 GiB peak, no restart, against every pre-change run exiting on the 849 GiB limit). The PR explicitly says the fix "covers this consolidation path, not the sink's whole footprint"; [CPU-317](https://linear.app/materializeinc/issue/CPU-317) tracks the rest. - **Kafka clients** (#40026): "55 to 24" is measured on a 4-worker, 11-source setup. The CPU throttling and degraded freshness are observed symptoms of the *old* behavior at a few hundred sources, not a measured post-change result at that scale, so the entry states the count as the outcome and the throttling as what the count drove. - **`environmentd` accept backlog** (#40035): the author's hedge is kept — "less likely to be restarted", not eliminated, because the PR says "This makes the queue fill later, not never." **Judgment calls a reviewer may want to revisit** — #39605 is filed as an **Improvement**, not a Bug Fix: nothing was incorrect, an unbounded operation gained a bound and a knob. #40101 is filed as a **Bug Fix** rather than omitted under the internal-logging rule, because credentials reaching the logs is a security defect with a user-visible remedy; only credentials a user inlined rather than wrapped in a `SECRET` were affected. Snapshot drafts (with PR links) in mz-skills: - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 12 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md#borderline), rc.2: 1 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md#borderline), rc.3: 0, final: 0 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]>

Dates in RC snapshots are provisional until the release ships. One commit is added per snapshot; the final snapshot sets the real dates. ## ⚠️ Needs the release owner before this can merge **1. This PR conflicts with `main`, and the conflict is substantive.** While this PR sat open, [#40201](#40201) ("docs: describe which SQL Server constraint Materialize records as a key") merged to `main` on 2026-11-05 and independently added its own `## v26.49.0` section — an `### Advisory {#v26.49-advisory}` plus an `### Improvements {#v26.49-improvements}` bullet. Both sides edit the same region of `_index.md`, so `mergeable` is `CONFLICTING`. Two problems ride along with the conflict: - **The same PR is described twice.** [#40201](#40201) documents the behavior of [#40131](#40131), which `rc.1` already published here as a Bug Fix ("Reduced the risk of new SQL Server source tables returning incorrect results after a unique constraint or primary key is dropped upstream…"). *Recommendation: keep `#40201`'s Advisory and Improvement and drop the `rc.1` bug-fix bullet* — the hand-written pair is more complete, carries the upgrade-impact guidance, and links `/ingest-data/sql-server/#unique-keys`. - **Anchor collision.** `main`'s `## v26.49.0` carries `{#v26.49-advisory}` and `{#v26.49-improvements}`; this branch's combined section carries `{#v26.49-improvements}`, `{#v26.49-agent-skills}`, and `{#v26.49-bug-fixes}`. Merging both defines `#v26.49-improvements` twice on one page. *Recommendation: rename this branch's three anchors to `{#v26.49.1-improvements}`, `{#v26.49.1-agent-skills}`, `{#v26.49.1-bug-fixes}`* — which is what the `v26.49.1` snapshot's review notes intended — *and leave `{#v26.49-advisory}` and `{#v26.49-improvements}` to `#40201`'s section.* `#40201`'s review links already point at `/releases/#v26.49-advisory`, and the compatibility YAML's `/releases/#v26491` target is unaffected. **2. Wording drift against the merged `v26.49.1` snapshot.** Commit `39e6b854a` landed here on 2026-11-05, before [MaterializeInc/mz-skills#355](MaterializeInc/mz-skills#355) merged on 2026-11-06. Two entries read differently on the branch than in the merged snapshot — the hydration improvement is "Investigate hydration on the cluster details page" here versus "See how a cluster hydrated, and why it did not" there, and the Freshness and Disk-column bullets differ likewise. Resolving the conflict above is the natural moment to decide which wording ships. ## Note on the section heading The `v26.49.1` patch train reused this branch (commit `39e6b854a`) rather than opening its own, so this PR now publishes a single combined `## v26.49.1` section — this train's `rc.1`–`rc.3` entries plus the three `v26.49.1` entries, with the Cloud version in parentheses. That is the house convention for a patch that ships in its parent's rollout week; compare the published `## v26.48.2` (`*Released to Materialize Cloud: 2026-10-29 (v26.48.0)*`). The compatibility YAML accordingly carries one `v26.49.1` row dated 2026-11-06, which is what actually shipped to Self-Managed that day. The `final` commit for this train is therefore **empty**: its increment carries no entries ([MaterializeInc/release#635](MaterializeInc/release#635) states "No PRs in `v26.49.0` require review", `v26.49.0-rc.3...v26.49.0` is the lone version-bump commit, and the agent-skills window is empty), and the real dates — Cloud **2026-11-05**, Self-Managed **2026-11-06** — were already set by `39e6b854a`. It exists so the branch's commit-subject ledger records that the snapshot was applied. ## Decisions required (carried over from `rc.1`) - **`DROP INDEX`** ([#39098](#39098)): it now rejects drops that used to succeed. Call it out as a breaking change rather than an Improvement? - **Console on large clusters** ([#40306](#40306)): the Cloud Console deploys separately, so this may already be live. Keep it in v26.49? - **Maintained Objects page** ([#40105](#40105)): omitted because it is behind a LaunchDarkly flag. Is it generally available? - **`mz profile init` OAuth** ([#40319](#40319)): omitted because the PR lists open items. Which `mz` release carries it? - **Upsert v2** ([#40189](#40189)): still omitted because `enable_upsert_v2` is `false`. Publish it? ## Snapshot drafts (with PR links) in mz-skills - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.1/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.1/omitted.md), [MaterializeInc/mz-skills#351](MaterializeInc/mz-skills#351) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.2/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.2/omitted.md), [MaterializeInc/mz-skills#352](MaterializeInc/mz-skills#352) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.3/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.3/omitted.md), [MaterializeInc/mz-skills#353](MaterializeInc/mz-skills#353) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/final/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 10 — [rc.1/omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.1/omitted.md#borderline), rc.2: 0, rc.3: 0, final: 0 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]>

DAlperin requested review from a team as code owners October 26, 2026 14:36

DAlperin force-pushed the dov/upsert-v2-stash-buckets branch from 9a40b7c to 4eb7b3e Compare October 26, 2026 14:44

DAlperin mentioned this pull request Oct 26, 2026

storage: split upsert_continual_feedback_v2 into submodules #40190

Merged

DAlperin added this pull request to stack #40191 October 26, 2026 14:52

DAlperin marked this pull request as draft October 26, 2026 14:54

DAlperin force-pushed the dov/upsert-v2-stash-buckets branch 2 times, most recently from 5e71a9e to c5b9002 Compare October 26, 2026 15:23

DAlperin marked this pull request as ready for review October 26, 2026 15:23

DAlperin requested a review from antiguru October 26, 2026 15:23

devin-ai-integration Bot reviewed Oct 26, 2026

View reviewed changes

DAlperin force-pushed the dov/upsert-v2-stash-buckets branch from c5b9002 to dca974f Compare October 26, 2026 16:13

DAlperin requested a review from a team as a code owner October 26, 2026 16:13

This comment was marked as resolved.

Sign in to view

DAlperin force-pushed the dov/upsert-v2-stash-buckets branch 2 times, most recently from 7c14dec to fcbf650 Compare October 26, 2026 16:59

DAlperin added 2 commits October 26, 2026 13:57

DAlperin force-pushed the dov/upsert-v2-stash-buckets branch from fcbf650 to af4127b Compare October 26, 2026 18:52

antiguru approved these changes Oct 26, 2026

View reviewed changes

DAlperin changed the title ~~storage: bucket the upsert v2 stash by time~~ storage: make the chunked upsert v2 stash time-major Oct 26, 2026

antiguru approved these changes Oct 26, 2026

View reviewed changes

DAlperin force-pushed the dov/upsert-v2-stash-buckets branch from 20ba367 to b3c5127 Compare October 26, 2026 20:53

DAlperin enabled stacked auto-merge (squash) October 26, 2026 20:58

DAlperin merged commit fc2b1d1 into main Oct 26, 2026
87 checks passed

DAlperin deleted the dov/upsert-v2-stash-buckets branch October 26, 2026 21:27

bosconi mentioned this pull request Oct 27, 2026

v26.48.0 release notes #40113

Merged

bosconi mentioned this pull request Nov 4, 2026

v26.49.0 release notes #40469

Merged

	impl<TOuter: BucketTimestamp, TInner: Timestamp> BucketTimestamp for (TOuter, TInner) {
	impl<TOuter: BucketTimestamp, TInner: Timestamp+TotalOrder> BucketTimestamp for (TOuter, TInner) {

	/// A product timestamp buckets by its outer coordinate: advancing moves the
	/// A lexicographic pair timestamp buckets by its outer coordinate: advancing moves the

Conversation

DAlperin commented Oct 26, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Performance

Tests

Release notes

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

This comment was marked as resolved.

Uh oh!

def- commented Oct 26, 2026

QA LLM Review

1. MEDIUM -- Paged stash re-copies each tick's data about 11 times in steady state

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Uh oh!

antiguru Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

DAlperin commented Oct 26, 2026

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DAlperin commented Oct 26, 2026 •

edited by github-actions Bot

Loading

devin-ai-integration Bot left a comment •

edited

Loading