timely-util: fund upsert spine exertion from inserted updates by DAlperin · Pull Request #39044 · MaterializeInc/materialize

DAlperin · 2026-09-22T16:25:11Z

Motivation

Spine::exert is called once per arrange_core activation. When the exertion policy returns effort it grants a fixed allowance, either to an active merge or, with no merge in progress, as a virtual introduction that rolls up the layers below it. Nothing ties the number of grants to the amount of input, so the optional consolidation a spine performs scales with how often its operator is scheduled. Enough idle turns between two published batches lift each batch into the largest layer on its own, and work per row approaches the size of the largest batch divided by the published batch size.

On a skewed 105-source upsert hydration (400cc, 8 workers) the unmodified synchronous spine visited 16 to 24 rows per published row and its two trials differed by 47%, which is what cadence-bounded work looks like. With this change, on a 300 GB incompressible seed, two trials landed 2.4% apart at 127 to 130 policy grants per published batch (unmodified: 372), 12% to 15% behind the RocksDB-backed v1 on wall clock at CPU parity, where the synchronous v2 path at the same policy was 2.25x behind. Steady state at 50cc: snapshot 5m22s against 9m38s for plain sync, p95 ingest lag 56 s against 93 s.

Description

Two commits, meant to be read separately.

Vendor the fueled spine. src/timely-util/src/funded_spine/spine_fueled.rs is trace/implementations/spine_fueled.rs from differential-dataflow 0.25.1, byte for byte apart from crate:: paths rewritten to differential_dataflow:: and a license header. The header follows the convention for derived files (Materialize's header plus a note naming the MIT-licensed source, whose license text is in the root LICENSE), so the file needs no copyright-lint exclusion. Nothing uses it in this commit. Two clippy lints the upstream file trips are allowed at the module declaration rather than by editing the file.
Fund optional exertion from input. Confined to Spine::exert, Spine::insert, the constructor, two funding fields with their three constants, and two small helpers (grant, spend_funding) that exert calls:
- insert credits eight fuel units per inserted update, the multiple introduce_batch already funds for active merges, and banks eight policy allowances per inserted batch, capped at 64. arrange_core seals one batch per input frontier advance, so the bank is the progress signal.
- What this bounds is the number of granted policy requests: one per effort / 8 inserted updates from credit, about 125 at the cluster policy's effort of 1000, plus the banked allowances, which are the bound for small batches. It does not bound the work a grant causes. A grant with no merge in progress introduces a virtual batch that can start a merge up into the largest layer, and that merge finishes unfunded. What changes is that new merges are paid for by input rather than by scheduling turns. The measured result is 125 to 164 grants per published batch, against 372 unfunded.
- exert spends credit first, then a banked allowance, and otherwise declines a request that would start new work and stays quiet. A request that advances a merge already in progress proceeds unfunded: that merge must complete before anything else lands at its level, so finishing it early adds no work, while leaving it stalled holds both inputs and its partial output. Funding is still spent first when available, so a funded spine does the same work it did in the measurements above. Once the input has closed (the spine's upper is the empty frontier) it exerts without bound, so the trace still reaches the policy's reduced form.
- The Trace interface is unchanged, so the spine drops into arrange_core with no operator changes. Both upsert v2 feedback arrangements use it: the chunked flavor via FeedbackSpine, the paged flavor via a FundedValRowSpine alias added to mz-row-spine. Compute arrangements keep the stock spine.

Consequence: an upsert feedback spine whose input is open but not advancing no longer spins to reduced form on its own, though it still finishes any merge in progress. Reclocked sources keep ticking, so this has not mattered in practice.

The vendored copy does not track upstream. A differential-dataflow bump leaves it unchanged, and the module documentation says upstream spine changes need a manual review on each bump. As this copy diverges, not every upstream change will apply.

Verification

New unit tests in funded_spine/tests.rs: funded turns never lift small batches into the largest one and go quiet once funding is spent, unfunded turns still finish a merge in progress, a closed input reduces to one batch, and frontier advances alone fund consolidation of a quiet input. Existing upsert v2 operator tests exercise both flavors on the new spine.

Release note: None.

antiguru · 2026-09-30T09:16:59Z

+            if !funded && !self.merging.iter().any(|b| b.is_double()) {
+                // Unfunded: stay quiet until an insert pays.
+                return;
            }
+            self.grant(effort);


Can we write this as:

if funded || self.merging.iter().any(|b| b.is_double()) { self.grant(effort) }

DAlperin · 2026-09-30T14:24:58Z

Benchmark results

Replica: 400cc (8 workers, 65 GiB, swap-backed), 20% pool budget, four spill workers, arrangement_exert_proportionality = 16, all async and payload flags off. The measurement for the funding rule itself is the first table, which isolates it on the synchronous spine. The others are measured on the whole stack, #39042 through this PR.

The funding rule in isolation: 192M-row incompressible seed (about 407 GB, state about 6x memory):

Spine	Result	CPU seconds	Extents created	Visits per row
stock, synchronous	completed, 106m35s	41,719	5.26 TB	16.4
funded, synchronous	on a 54-minute pace, killed at 97.7%	21,089 at the kill	2.54 TB	not captured
funded, async merges	completed, 58m24s	25,222	2.64 TB	9.0
v1 (RocksDB)	completed, 47m24s	21,517

Funding halves CPU and extent bytes, and doubles pace. Up to the kill, the funded synchronous run matched the funded async run on pace, CPU and extents, so the rule accounts for the whole gain. The funded runs were killed near the end at the replica's 466 GB memory-plus-swap ceiling. Every non-payload variant on this seed peaks within 5 to 15% of that ceiling, v1 included (455 GB). So pass or fail here comes down to a few percent of headroom, not the rule, and pace and CPU to the kill point are the comparison. The 300 GB seed below has headroom.

Hydration: 300 GB incompressible seed (142.3M rows, state about 4.6x memory):

Arm	Node	Hydration	CPU seconds	Extents	Peak swap	Grants per batch
landing stack, trial 1	r7gd.8xlarge	35m46s	14,896	1.76 TB	289 GB	138
landing stack, trial 2	r7gd.8xlarge	32m44s	14,922	1.72 TB	281 GB	129
v1	r7gd.8xlarge	28m49s	13,962		315 GB
landing stack	r7gd.4xlarge	56m26s	17,215	1.83 TB	328 GB	164
v1	r7gd.4xlarge	31m54s	14,347		337 GB

On the 8xlarge the stack is 14 to 24% behind v1 at CPU parity. On the 4xlarge it is 77% behind, and the swap device's write bandwidth is the limit. Chunked writes 1.95 TB to that device against v1's 0.72 TB, and holds the device's 605 MB/s ceiling. That gap is bytes per row, which payload separation addresses. It is out of scope for this PR.

Hydration: 192M-row compressible seed (1,900-byte values with a repeated 475-byte pattern):

Arm	Hydration	CPU seconds	Extents	Peak swap
funded, synchronous	29m50s	14,336	0.86 TB	139 GB
stock, synchronous (earlier seed of the same shape)	44m11s
v1 (earlier seed of the same shape)	45 to 46 min

Steady state: 20M keys, ten minutes of uniform overwrites:

Size	Arm	Snapshot	Lag p50	p95	max
50cc	landing stack	6m08s	16 s	66 s	92 s
50cc	stock, synchronous	9m38s	20 s	93 s	118 s
50cc	v1	7m37s	2 s	17 s	22 s
25cc	landing stack	11m09s	15 s	40 s	56 s
25cc	stock, synchronous	16m08s	13 s	106 s	136 s
25cc	v1	18m07s	7 s	29 s	33 s

The snapshot is 31 to 36% faster than the stock spine, and p95 lag is 29 to 62% lower. p95 lag is still 1.4 to 4x v1's. That is the cost of synchronous probes against cold state, and this PR doesn't claim to fix it.

Vendored copy against the fork. The interim image used the same rule on a differential-dataflow fork branch. On the 300 GB seed it ran 32m22s and 33m08s at 127 and 130 grants per batch. This PR's vendored spine reproduces that within noise (the 8xlarge trials above).

Projection for the production case. This is an estimate, not a measurement. The production replica has 31 workers, 252 GB of memory and one NVMe swap device, and holds 1.56 TB of state. v1 hydrates it in 44 minutes, CPU-bound. Stock v2 did not finish: its worst sources reported 19.7 and 36.4 hours. The same per-worker memory, with four times the workers sharing one device, puts the stack at about 55 to 90 minutes if production values compress 1.8x to 2.7x. That is the range v1's footprint suggests. If they don't compress, it is about 2.4 hours. That turns a hydration that never finished into one about 1.2 to 2x slower than v1. Closing the rest of the gap needs fewer bytes per row.

Written by Claude (Claude Code) from the benchmark notes.

antiguru

Posted by Claude Code on behalf of @antiguru.

I confirmed the vendored spine matches differential 0.25.1 apart from the header and path rewrite. The 8 << level claim about introduce_batch also holds.

Main open question: what the credit actually bounds (inline). A funded grant can start a large merge that then finishes unfunded, so the real bound is grants per batch, not fuel per row. The miri ignores and a guard against a differential version bump are cheap to add.

antiguru · 2026-09-30T14:56:18Z

+        else {
+            // Introduce an empty batch with roughly *effort number of virtual updates.
+            let level = effort.next_power_of_two().trailing_zeros() as usize;
+            self.introduce_batch(None, level);


A funded grant debits effort (1000) from the credit, and this line introduces a virtual batch at level 10. That can start a merge up into the largest layer, and the merge then runs to completion unfunded through the is_double() path. So the work is bounded by the number of funded grants, not by fuel per inserted row. A 16-row insert funds almost no credit, but its 8 banked grants can each start a large merge.

The measurements show the bound holds in practice. Still, CONSOLIDATION_CREDIT_PER_UPDATE and the module doc read as a proportional fuel bound. Either charge the credit by the size of the merge a grant starts, or describe the actual bound as grants per inserted batch.

Agreed, the bound is on grants, not fuel. I've changed the module doc, the constant's doc and the PR description to say that: up to eight grants per inserted update from credit plus the banked allowances, and a merge a grant starts may finish unfunded. I kept the charging as it is, because charging by merge size would change the behavior the benchmarks measured. The grants-per-batch numbers (125 to 164, against 372 unfunded) are the bound actually observed.

antiguru · 2026-09-30T14:56:21Z

+}
+
+#[mz_ore::test]
+fn funded_exertion_stops_when_inserted_updates_are_spent() {


ci/test/cargo-test-miri.sh (nightly miri-test) runs the whole workspace, timely-util included. This test, closed_input_reduces_without_bound, and frontier_advances_fund_consolidation_of_a_quiet_input each build 2^18 rows and run about 170k exert calls, which will be very slow under miri. They need the same #[cfg_attr(miri, ignore)] as unfunded_turns_finish_merges_in_progress.

Done, all three.

antiguru · 2026-09-30T14:56:24Z

+
+#[rustfmt::skip]
+#[allow(clippy::as_conversions, clippy::needless_pass_by_ref_mut)]
+pub mod spine_fueled;


I checked the vendored file: it matches differential-dataflow-0.25.1/src/trace/implementations/spine_fueled.rs exactly, apart from the header and the path rewrite.

Once differential is bumped, upstream spine fixes won't reach this copy, and nothing will flag it. A TODO: pointing to a tracking issue for the fork-branch replacement would help. Better still would be a test that asserts the pinned differential version, so a bump forces someone to look at this file.

Added a note at the top of the module: the vendored spine is a copy, not a dependency, a differential-dataflow bump doesn't update it, and upstream changes need a manual review on each bump. Since this copy will diverge further, it may not make sense to port upstream one to one, so the note asks for a review rather than a forced sync.

antiguru · 2026-09-30T14:56:27Z

+            let funded = self.upper.borrow().is_empty() || self.spend_funding(effort);
+            if !funded && !self.merging.iter().any(|b| b.is_double()) {
+                // Unfunded: stay quiet until an insert pays.
+                return;


nit: unlike upstream, this path skips activator.activate(). That's correct, since the next insert reschedules the operator, but worth a comment because it's a difference from upstream.

A separate question: with an open frontier and no inserts, stale upsert retractions now stay unmerged. That costs memory and adds batches for every lookup cursor. Has memory on a quiet source been measured, beyond the steady-state ingest numbers?

Added the comment. On quiet sources: arrange_core inserts a batch on every frontier advance, empty or not, and each insert banks grants, so a reclocked source that stops receiving data still consolidates. frontier_advances_fund_consolidation_of_a_quiet_input covers it. Only a frontier that stops advancing entirely keeps its batches. Memory on an idle source hasn't been measured on AWS, and I'm adding that to the next benchmark run.

antiguru · 2026-09-30T14:56:29Z

+        assert_eq!(batch.lower(), &self.upper);
+
+        self.upper.clone_from(batch.upper());
+        self.consolidation_credit = self.consolidation_credit.saturating_add(


nit: consolidation_credit never decays. After hydrating N rows it holds about 8·N·effort fuel, which later idle turns spend down. The total stays proportional to input, so this is fine, but one sentence on the field doc would help.

DAlperin · 2026-09-30T15:15:14Z

Estimated impact on the production hydration that started this work

This is an estimate, not a measurement. The stack hasn't run against the production case yet, and that rerun is the next step.

The case. One upsert replica with 31 workers, a 252 GB memory limit and one NVMe swap device, hydrating 105 sources. That is 756M records and 1.56 TB of logical state, about 2 KB per record, with three sources holding 90% of it.

v1 (RocksDB): the slowest source finished in 38 to 44 minutes across five restarts. v1 was CPU-bound, averaging 23 cores with heavy throttling.
v2 before this stack: the replica never finished. Indexing fell to a quarter of v1's rate by minute 13. The two largest sources that reported a time took 19.7 and 36.4 hours. The pool ledger reached 7.3x its budget.

What the stack changes for that case.

The collapse. At policy 1337, exertion was bounded by scheduling cadence, so every published batch could be lifted into the largest layer on its own. controller: use the arrangement maintenance policy for storage replicas #39041 (policy 16, merged) and the funding rule in this PR remove that. On the bench, the stock synchronous spine took 106m35s on a 407 GB incompressible seed. The funded spine ran at a 54-minute pace on half the CPU and extent bytes.
Memory. timely-util: bound serialized column chunks and merge allocations #39042 and pool: bound insertion debt during concurrent enforcement #39374 bound chunk sizes and insertion debt. On the bench, the pool stayed within 1.13x of its budget.
What's left. The remaining cost is bytes written to the swap device. Chunked v2 writes about 6 times its state to swap during hydration. v1 writes about 2.5 times, most likely because most of RocksDB's compaction output is freed before the kernel ever pages it out.

Scaling the bench to production. Per worker the two are close:

Memory per worker: 8.1 GB on both.
State per worker: 1.33x larger in production, 50 GB against 37.5 GB.
Swap bandwidth per worker: about four times lower in production. The bench 8xlarge gave 8 workers one device with a p90 of about 1.2 GB/s. Production puts 31 workers on the same class of device.
Compression: production values compress and the bench seed doesn't. v1's footprint suggests a ratio of 1.8x to 2.7x.

The model. Hydration time is whichever limit is slower:

CPU: bench CPU seconds per GB, calibrated against v1's production run (a factor of 0.8), times 1.56 TB, divided by the cores production gets.
Device: bytes written to swap, divided by the device's write bandwidth.

The same model predicts the bench's device-bound 4xlarge run to within a minute: 57 minutes predicted, 58 measured.

Configuration	CPU floor	Device floor	Estimate	Against v1 (44 min)
v2 before this stack, observed			19.7 to 36.4 h, never finished	27 to 50x
this stack, values compress 1.8x to 2.7x	about 42 min	53 to 87 min	about 55 to 90 min	1.2 to 2x
this stack, values incompressible	about 42 min	about 140 min	about 2.4 h	3.2x

Reading. The stack should turn a hydration that never finished into one that completes, about 1.2 to 2x slower than v1 at the likely compression ratio. Closing the rest of the gap takes fewer bytes per row. A prototype that stores values out of line, so merges move only keys and handles, hydrated the bench's 300 GB seed in 13 minutes against v1's 33, writing 0.4x v1's swap bytes. It is not part of this stack.

What would move the estimate.

The compression ratio. It decides between about 55 minutes and about 2.4 hours. It can be read from the pool's compressed-to-logical extent ratio on the production run.
Heap outside the pool. On the bench this was mostly the Kafka consumer's prefetch, which scales with source and partition count. The model doesn't include it.
Steady state after hydration. None of the large-replica bench runs stayed below saturation, so the production replica's sustained ingest rate under this stack is unknown.

Written by Claude (Claude Code) from the benchmark notes.

antiguru

Posted by Claude Code on behalf of @antiguru.

I re-checked the fixes on the current head: the funded || merging rewrite, the comment on the skipped reactivation, the miri ignores, the note on the vendored copy, and the docs on the bound being grants rather than work. All look good.

One correction, inline: the new docs say credit funds eight grants per inserted update. At the policy's effort of 1000 it's about one grant per 125 updates, so the bank of eight grants per batch is what bounds small batches.

antiguru · 2026-10-01T10:40:54Z

+
+/// Fuel each inserted update credits for policy-requested effort, scaled by
+/// the spine's effort multiplier. A granted request spends one `effort`, so
+/// each inserted update funds up to eight grants. The multiple matches the


"Funds up to eight grants" only holds if a grant costs 1. The credit per update is 8 * self.effort, and the multiplier defaults to 1 (with_effort(1, …) in new). A grant spends the policy's return value, which is 1000 for the cluster policy (merge_effort = Some(1000) in src/cluster/src/client.rs) and for the test policy. So an inserted update funds about 1/125 of a grant, and a 16-row batch's credit funds none. For small batches, the bound actually comes from the eight banked allowances per batch. Could this doc, and the module doc at funded_spine.rs:30, say that? For example: "one grant per effort / 8 inserted updates, about 125 at the cluster policy's effort of 1000".

Right, the multiplier is 1 and a grant spends the policy's 1000. So credit funds about one grant per 125 inserted updates, and the eight banked allowances per batch are what bound small batches. I fixed the constant's doc, the module doc and the PR description. My earlier reply on the bound had the same mistake.

antiguru · 2026-10-01T10:40:56Z

+//! quiet until the next insert. Once the input closes, exertion is unbounded
+//! again so the trace reaches the policy's reduced form.
+//!
+//! What funding bounds is the number of granted requests: up to eight per


Same issue as on CONSOLIDATION_CREDIT_PER_UPDATE: at the policy's effort of 1000, credit funds about one grant per 125 inserted updates, not eight per update.

A verbatim copy of trace/implementations/spine_fueled.rs from the 0.25.1 release, with only its crate-internal paths rewritten to the differential_dataflow crate and a license header added. The header follows the convention for derived files, naming the MIT-licensed source whose license text is in the root LICENSE. Nothing uses it yet.

arrange_core calls Trace::exert once per operator activation, and each call the exertion policy answers grants a fixed allowance: fuel for an active merge or a virtual introduction that rolls up the layers below it. Nothing ties the number of grants to the amount of input, so optional consolidation scales with how often the operator is scheduled. Enough idle turns between two published batches lift each batch into the largest layer on its own, and merge work per row approaches the largest batch size over the published batch size. On a skewed 105-source upsert hydration the synchronous spine visited 16 to 24 rows per published row and its trials differed by 47%. The vendored spine now pays for policy-requested effort from credit that inserted updates accrue, eight fuel per update as introduce_batch already funds for active merges, and from a bank of allowances that each inserted batch tops up, eight per batch and at most 64 held, since arrange_core inserts batches as its input frontier advances. An unfunded request that would start new work is declined and the spine stays quiet until the next insert. A request that advances a merge already in progress proceeds unfunded: that merge must complete before anything else lands at its level, so finishing it early adds no work, while leaving it stalled holds both of its inputs and its partial output. Funding is still spent first when available, so a funded spine does the same work as before. A closed input, whose upper is the empty frontier, exerts without bound so the trace still reaches the reduced form. The stock Trace interface is unchanged, so the spine drops into arrange_core as is. Both upsert v2 feedback arrangements use it: the chunked flavor through FeedbackSpine and the paged flavor through a FundedValRowSpine alias in mz-row-spine. Compute arrangements keep the stock spine. On a 300 GB incompressible hydration seed at 400cc this rule took the synchronous v2 path from 2.25x behind the RocksDB-backed v1 to 12% to 15% behind at CPU parity, at 127 to 130 policy grants per published batch against 372 unmodified, with two trials 2.4% apart. Tests cover that funded turns never lift small batches into the largest one and go quiet once funding is spent, that unfunded turns still finish a merge in progress, that a closed input reduces to one batch, and that frontier advances alone fund consolidation of a quiet input.

Release notes for the **v26.48.0** train. One commit is added per snapshot; the `final` snapshot sets the real dates. **Applied: `rc.1`, `rc.2`, `rc.3`, `final` — the train is complete.** - **Content**: 3 Feature sections (upsert hydration; Kafka sink headers GA; MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS`), 9 Improvements, 2 Guides, 4 Bug Fixes. No **Agent Skills** section — every agent-skills PR in all four windows was omitted. - **Scope**: 123 PRs across the four snapshots — rc.1 114 (14 included), rc.2 5 (0 included), rc.3 1 (1 included), final 3 (0 included). - **Dates — now real, not provisional**: Cloud **2026-10-29**, Self-Managed **2026-10-30**. The `rc.1`-anchored provisional estimate was right, so `final` left both files unchanged. The `v26.48.0` tag's commit date is 2026-10-28T18:35:05Z, a Wednesday; the first Thursday on or after it is 2026-10-29, with Self-Managed the day after. - **Operator compatibility**: row for `v26.48`, release date `2026-10-30`. Unchanged by rc.2, rc.3, and final. - **Branch base**: `main`, whose newest published section was `## v26.46.1` when this branch was cut; v26.47.0 and v26.47.1 have since merged in. `v26.48.0` stays at the top of the list. ### final increment — empty, and verified empty `final`'s commit is empty. The tracking issue ([release#627](MaterializeInc/release#627)) lists exactly one item, the unassociated `4edd60a8d` "release: bump to version v26.48.0", which the release-process-automation rule omits. The three agent-skills PRs in the window ([#118](MaterializeInc/agent-skills#118), [#119](MaterializeInc/agent-skills#119), [#120](MaterializeInc/agent-skills#120)) were all omitted. This was **verified rather than assumed**, because a zero-PR snapshot and a failed fetch produce identical output: `compare/v26.48.0-rc.3...v26.48.0` returns `total_commits: 1`, and that commit is the version bump. Nothing merged onto the release branch in the 42 h 49 m between the `rc.3` tag and the ship tag. Across the train, every agent-skills PR from [#111](MaterializeInc/agent-skills#111) to [#120](MaterializeInc/agent-skills#120) falls in exactly one window, with no gaps or overlaps. ⚠️ **One decision is still open for the release owner — see "Upsert v2" below.** Every flag this train held open was re-read in source at the `v26.48.0` ship tag rather than inherited from the RC snapshots: | Flag | Default at `v26.48.0` | Source | Held since | Disposition | | -- | -- | -- | -- | -- | | `enable_upsert_v2` | **`false`** | `src/storage-types/src/dyncfgs.rs:505-509` | rc.2 | ⚠️ premise contradicted — see below | | `enable_upsert_chunked_stash` | **`false`** | `src/storage-types/src/dyncfgs.rs:469-475` | rc.2 | same | | `enable_upsert_paged_spill` | **`false`** | `src/storage-types/src/dyncfgs.rs:450-455` | rc.2 | same | | `enable_mcp_protocol_2026_07_28` | **`false`** | `src/adapter-dyncfgs/src/dyncfgs.rs:295-299` | rc.1 | hold stands — still omitted | | `cluster-details-page-cns-173` | n/a (LaunchDarkly) | no source default | rc.1 | hold stands — still omitted | **The rc.2 hold on [#40189](#40189) rests on a premise this branch now contradicts.** rc.2 omitted it at Borderline on exactly one ground — every measurement is taken on a path `enable_upsert_v2` keeps off by default — and asked whether any environment actually runs upsert v2. The branch answers that without anyone feeding the answer back: the human-added **"Improved hydration times and memory efficiency for upsert sources"** section states *"This change has been rolled out to all Materialize Cloud environments and applies to all existing and new clusters."* The three flags still read `false` at the ship tag, which is exactly the source-default-versus-LaunchDarkly divergence the rollout-lever rule covers: the published docs call wins, and the sole stated reason for the omit has fallen away. - **Not already covered.** The published section is about **hydration** — building a source from scratch, 206 min → 108–113 min, 542 GiB → 183–208 GiB. #40189 is about **catch-up after a backlog**, a different operation with its own motivating incident ([CPU-320](https://linear.app/materializeinc/issue/CPU-320): a source that *"could not catch up on any replica size after a producer rate increase"*). A customer who read the hydration section would learn nothing about catch-up. - **Suggested Improvements bullet, to add only if you confirm** — it was *not* applied in the `final` commit: > **Faster catch-up for upsert sources with a large backlog**: An upsert source working through a backlog now drains one timestamp per round instead of merging and re-pushing the whole backlog each round, so catch-up time grows linearly rather than quadratically with the backlog; draining a 100-timestamp tail fell from 6.5 to 0.15 seconds, and a 400-timestamp tail drains in 0.67 seconds. - **Caveat that must travel with the decision**: the baseline in those timings is the *paged* flavor on `main`, not the configuration Cloud ran before this rollout — so they are not a clean before/after for a given customer. Dropping the two timings and keeping the linear-versus-quadratic claim holds regardless of baseline. - **Two weaker links, stated rather than buried**: that the rolled-out configuration is the one #40189 was measured on is *inferred from the published section's prose* (its "buffer pool … serve sequential work from disk" reads as chunked stash plus paged spill), not confirmed by anyone. And the author declared "Release notes: None", naming `enable_upsert_v2` — so publishing this would be an **override** of an author signal, justified by your own later rollout statement. - **Third consecutive train** to hold this flag: `v26.44.0-rc.1` ([#39041](#39041)), `v26.45.0-rc.1` ([#39044](#39044)), `v26.48.0-rc.2` (#40189). If the rollout statement is accurate, the earlier two are worth revisiting rather than left to age out. **The other two holds stand, corroborated.** `enable_mcp_protocol_2026_07_28` still reads `false` at the ship tag and the five MCP PRs rc.1 held appear nowhere in the section; neither does the Console cluster-details page stack. In both cases you had the whole train in view and did not promote them — the confirmation rc.1 asked for, arriving in the negative. **No duplicates, re-checked at ship.** The seven PRs rc.1 flagged as already covered by v26.47.0 are still absent, the `## v26.47.1` section is untouched, and rc.3's entry ([#40065](#40065)) appears exactly once. **Two notes on human edits to this branch**, both recorded as fact rather than objection: - `524adb8` removed the rc.1 feature section **"Native Google Cloud Storage for persist"** ([#39983](#39983)) outright. Flagging it only because a published entry disappearing is easy to miss in a long branch — if that was deliberate, nothing to do. - Two rc.1 Improvements were promoted to their own feature sections with docs links and a code sample (Kafka sink headers [#39908](#39908); MySQL `EXCLUDE ALL NOT NULL CONSTRAINTS` [#39873](#39873)), and a **Guides** section was added that is not in the skill's template. The lesson for next time: an option whose flag flipped to `true` in the same release more likely warrants a feature section than the Improvements bullet the rubric suggests by default. **Carried forward from rc.3 and still open**: the `K8s recovery: envd on failing node` nightly — the job that sank [#40029](#40029), the first attempt at what #40065 re-lands — **was not checked**; it needs Buildkite, outside this run. The entry itself was verified correct in source at both the rc.3 and ship tags, so this is a CI-confidence question, not a notes-accuracy one. ### rc.3 increment One Improvement, for #40065: a read-only `environmentd` that reboots to pick up DDL changes during zero-downtime upgrade catch-up now restarts in place rather than exiting its container, so Kubernetes no longer applies crash-restart backoff. Not inherited from rc.1 — rc.1 correctly omitted the first attempt (#40029) together with its revert (#40061) as a no-op on `main`; rc.3 re-lands the behavior in narrowed form, verified as a real change by comparing `entrypoint.sh` at the `v26.47.0` tag (no retry loop) against `v26.48.0-rc.3` (re-exec on exit 75). The draft's closing "so zero-downtime upgrades catch up faster" was cut: no cited PR measures an upgrade duration, so the entry claims only the mechanism. - **Not a Bug Fix.** 0dt upgrades completed on v26.47.0; the catch-up reboots merely collected backoff delay. Existing behavior improved, so Improvements. - **Residual, deliberately unpublished.** A stale read-only process in a replaced pod still restarts in place on these halts until the kubelet kills its container, competing with its replacement for the generation's replicas. The `MZ_RESTARTED_IN_PLACE` marker check keeps it from booting writable, so it cannot fence the leader — the failure #40061 reverted. Worth watching if 0dt reports look odd after this ships. ### rc.2 increment — empty, by design rc.2's commit is empty: no entry was published and the provisional dates did not move. It exists so later runs can see from the commit subject that rc.2 was applied. Its single materialize PR, #40189, is the open item discussed under the `final` increment above. ### rc.1 — standing warnings ⚠️ **Seven PRs in rc.1's tracking issue were already covered by the v26.47.0 train, and none is republished here.** v26.47.0 took five RCs whose later ones were cherry-picks onto the release branch; the same PRs appear in this issue because they are also what landed on `main` for v26.48.0. Four of the seven **shipped to Cloud in v26.47.0 on 2026-10-22** — #37653, #39910, #39927, #40036 — so republishing any of them would put duplicate text in front of customers eight days later. The other three (#39903, #39906, #40046) were held at Borderline in v26.47.0 and are held again here, each reason re-checked rather than inherited. The "snapshots are disjoint by construction" guarantee holds *within* a train, not across trains, so this was caught mechanically by grepping every PR number in the issue against the v26.47.0 snapshot directories. All seven are listed under their own heading in `rc.1/omitted.md`. **Open questions for the release owner** - **Upsert v2** (#40189, rc.2) — see the `final` increment above. The rollout statement on this branch has changed the picture since rc.2; this is the one item that needs a decision before merge. - **MCP protocol revision 2026-07-28** (#39698, #39702, #39703, #39759, #39761) — the stack is complete and `enable_mcp_protocol_2026_07_28` still defaults to **`false`** at the ship tag. Omitted, with the absence of any MCP text on this branch taken as confirmation. Drafted wording remains in `rc.1/omitted.md` if the flag flips. - **Redesigned cluster details page in the Console** (#39446, #39507, #39508, #39510, #39607) — omitted for the same reason. Note that the Improvements section does carry **"New cluster details page on Self-Managed"**, added by hand, which links the v26.47 section rather than re-describing the page. - **Schema projection on the Console's Freshness page** (#40031) — a genuine new way to diagnose lag, but the author states the whole Freshness page "is currently behind a feature flag that is only enabled for certain customers in impersonation mode". **Is the page generally reachable yet?** This is the same page v26.47.0 held for the same reason. - **Time zone data 2026e timing** (#40045) — the author flags the Canadian zone changes as effective **2026-11-01**, three days after the Cloud date. Tight, but it holds. *Resolved since rc.1: the Kafka sink headers framing question — the GA section on this branch answers it.* **Verification notes** — rc.1's Step 5 pass cut one drafted entry and rewrote four: - **Cut**: a Bug Fix drafted for #40032 ("halt with exit code 168 when a live reader loses its lease"). The PR's own Problem section is about alerting — `mz_persist_lease_timeout_read` cannot distinguish a dead reader from a stuck one — and the follow-up it names is a Cloud alerting change. Nothing a customer observes changes. - **tzdata** (#40045): the draft said the Canadian zones move to "permanent daylight saving time", the author's prose. The PR's data table shows a permanent **−05/EST** and **−06/CST** — summer *offsets* kept year-round, but with another zone's standard-time *abbreviations*, which is what a user sees from `AT TIME ZONE`. Reworded to what the table supports. - **Hydration memory** (#39923): scoped to the consolidation step and the measured outcome (~628 GiB peak, no restart, against every pre-change run exiting on the 849 GiB limit). The PR explicitly says the fix "covers this consolidation path, not the sink's whole footprint"; [CPU-317](https://linear.app/materializeinc/issue/CPU-317) tracks the rest. - **Kafka clients** (#40026): "55 to 24" is measured on a 4-worker, 11-source setup. The CPU throttling and degraded freshness are observed symptoms of the *old* behavior at a few hundred sources, not a measured post-change result at that scale, so the entry states the count as the outcome and the throttling as what the count drove. - **`environmentd` accept backlog** (#40035): the author's hedge is kept — "less likely to be restarted", not eliminated, because the PR says "This makes the queue fill later, not never." **Judgment calls a reviewer may want to revisit** — #39605 is filed as an **Improvement**, not a Bug Fix: nothing was incorrect, an unbounded operation gained a bound and a knob. #40101 is filed as a **Bug Fix** rather than omitted under the internal-logging rule, because credentials reaching the logs is a security defect with a user-visible remedy; only credentials a user inlined rather than wrapped in a `SECRET` were affected. Snapshot drafts (with PR links) in mz-skills: - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.3/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 12 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.1/omitted.md#borderline), rc.2: 1 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.48.0/rc.2/omitted.md#borderline), rc.3: 0, final: 0 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]>

DAlperin added this pull request to stack #39045 September 22, 2026 17:03

DAlperin force-pushed the dov/sync-spine-funding branch from c1e3189 to 497e085 Compare September 22, 2026 17:35

DAlperin changed the title ~~differential: bound synchronous spine exertion by input~~ timely-util: fund upsert spine exertion from inserted updates Sep 22, 2026

DAlperin force-pushed the dov/sync-spine-funding branch 3 times, most recently from 6bfbebe to bf2fea7 Compare September 22, 2026 22:00

DAlperin force-pushed the dov/sync-spine-funding branch 2 times, most recently from 4bd3a47 to 2d28c6b Compare September 23, 2026 15:35

DAlperin force-pushed the dov/sync-spine-funding branch 2 times, most recently from 8294574 to f9b7ce0 Compare September 28, 2026 18:21

DAlperin marked this pull request as ready for review September 29, 2026 19:16

DAlperin requested review from a team as code owners September 29, 2026 19:16

antiguru reviewed Sep 30, 2026

View reviewed changes

DAlperin mentioned this pull request Sep 30, 2026

timely-util: bound serialized column chunks and merge allocations #39042

Merged

DAlperin mentioned this pull request Sep 30, 2026

timely-util, cluster: count columnar merge work and exertion policy decisions #39043

Merged

antiguru approved these changes Sep 30, 2026

View reviewed changes

antiguru reviewed Sep 30, 2026

View reviewed changes

DAlperin force-pushed the dov/sync-spine-funding branch from f9b7ce0 to bd96fc5 Compare September 30, 2026 14:56

DAlperin removed this pull request from stack #39045 September 30, 2026 14:57

DAlperin added this pull request to stack #39375 September 30, 2026 14:57

DAlperin force-pushed the dov/sync-spine-funding branch from bd96fc5 to 81957de Compare September 30, 2026 16:07

DAlperin force-pushed the dov/sync-spine-funding branch from 81957de to 9f3032e Compare September 30, 2026 19:33

antiguru reviewed Oct 1, 2026

View reviewed changes

DAlperin force-pushed the dov/sync-spine-funding branch from 9f3032e to 97c3792 Compare October 1, 2026 14:26

DAlperin force-pushed the dov/sync-spine-funding branch from 97c3792 to 31ad14a Compare October 1, 2026 16:56

DAlperin added 2 commits October 1, 2026 14:58

DAlperin force-pushed the dov/sync-spine-funding branch from 31ad14a to 743a057 Compare October 1, 2026 19:07

DAlperin merged commit bb5c454 into main Oct 1, 2026
87 checks passed

DAlperin deleted the dov/sync-spine-funding branch October 1, 2026 19:41

bosconi mentioned this pull request Oct 27, 2026

v26.48.0 release notes #40113

Merged

Conversation

DAlperin commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Verification

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DAlperin commented Sep 30, 2026

Benchmark results

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DAlperin commented Sep 30, 2026

Estimated impact on the production hydration that started this work

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DAlperin commented Sep 22, 2026 •

edited

Loading